A method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform

Abstract

This invention discloses a method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform, relating to the field of computer performance and energy consumption collaborative optimization technology, to improve the performance of deep learning inference applications while reducing energy consumption. The key technical points of this invention include: first, extracting sub-networks with performance or energy consumption optimization potential; then, measuring and fusing the performance and energy consumption data of nodes and sub-networks; next, establishing an analytical performance and energy consumption prediction model to predict the runtime and energy consumption of the inference engine under different mapping configurations; then, using a mapping optimization algorithm based on a variable-length sliding window to search for a mapping configuration with better performance and energy consumption; finally, generating the inference engine and iterating through the search for the optimal number of CUDA streams. This invention requires no user intervention and can automatically find a mapping configuration with better performance and energy consumption and generate an inference engine. Compared with existing methods such as JEDI, this invention offers greater performance improvement, greater energy savings, and a shorter optimization time.

Description

Technical Field

[0001] This invention relates to the field of computer performance-energy consumption co-optimization technology, specifically to a method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform. Background Technology

[0002] Deep learning applications are now prevalent on heterogeneous SoC (System on a Chip) platforms. Many of these platforms are equipped with various accelerators, such as GPUs (Graph Processing Units) and NPUs (Neural Processing Units), to accelerate deep learning applications. For example, the widely used NVIDIA Jetson series platform includes a CPU, GPU, and DLA (deep learning accelerator). The Jetson series heterogeneous SoC platform is a computing platform released by NVIDIA specifically designed for robotics and embedded edge AI applications. DLA is a hardware accelerator developed by NVIDIA for deep learning operations. DLA aims to provide full hardware acceleration for convolutional neural networks. DLA supports neural network layers such as convolution, deconvolution, fully connected layers, activation layers, pooling layers, and batch normalization. NVIDIA provides the TensorRT development tool to help users deploy deep learning inference applications on the Jetson platform, which can generate inference engines that can run on the Jetson platform from deep learning models. TensorRT primarily focuses on optimizing the performance of deep learning applications on a single accelerator (GPU or DLA) and cannot automatically map different layers of a neural network to the most suitable accelerator. TensorRT only supports two mapping strategies by default: ① Use GPU only, mapping all deep learning network layers to the GPU; ② Use DLA as much as possible, mapping as many neural network layers as possible to the DLA. TensorRT provides Python and C++ APIs, allowing users to map deep learning network layers to GPUs and DLAs according to custom strategies to generate GPU-DLA hybrid engines.

[0003] Both GPUs and DLAs are suitable accelerators for handling deep learning workloads. Generally speaking, GPUs offer higher absolute performance, better versatility, and can run almost all types of neural network layers. For computationally intensive neural network layers, GPUs are generally more energy-efficient (e.g., lower performance-latency product, EDP = Energy·Time), and should be used for scenarios where low latency is critical. DLAs are designed to run specific types of neural network layers, offering better energy efficiency for some layers, but their absolute latency is not necessarily lower than GPUs, and they cannot run neural network layers outside their design scope. It's also important to consider the additional overhead that occurs when switching deep learning workloads between GPUs and DLAs, which cannot be ignored. Therefore, compared to a mapping strategy that uses only GPUs or prioritizes DLAs, properly mapping deep learning workloads between GPUs and DLAs can potentially improve performance (i.e., increase inferences per unit time or reduce latency) while reducing energy consumption.

[0004] In recent years, many studies have optimized the mapping of deep learning workloads to multiple accelerators to improve performance and reduce energy consumption. Early methods manually traversed the mapping search space, which could not be optimized automatically, and the optimization process was tedious and extremely time-consuming. Some semi-automatic methods manually built empirical models to guide the mapping optimization process. These methods require users to have extensive expertise in heterogeneous computing and deep learning, and the construction of empirical models also requires a lot of time and data. Moreover, these empirical models are only applicable to specific applications or platforms and have poor generality.

[0005] Current advanced methods use evolutionary or heuristic algorithms to find the optimal layer mapping configuration. Due to the nature of these algorithms and the lack of performance and energy consumption models to guide the optimization process, these automated methods must repeatedly generate deep learning inference engines under different layer mapping configurations and measure performance and energy consumption. Therefore, these automated methods typically take tens of hours or even several days to find the optimal configuration. For example, the JEDI method, which currently has relatively good optimization results... [1] It is based on genetic algorithms and heuristic algorithms.

[0006] In summary, existing deep learning load mapping optimization methods are inefficient. Furthermore, existing methods do not fully consider the topology of deep neural networks, potentially missing many optimization opportunities. Summary of the Invention

[0007] Therefore, this invention proposes a method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform, which is used to realize an automatic optimization system for deep learning load mapping, so as to improve the performance of deep learning inference applications while reducing energy consumption.

[0008] According to one aspect of the present invention, a method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform is proposed, the method comprising: for a heterogeneous SoC platform equipped with GPU and DLA accelerator:

[0009] Extract candidate subnetworks with performance or energy consumption optimization potential from deep learning model networks;

[0010] The deep learning model network is split into multiple fusion nodes, and the runtime and energy consumption of each fusion node on the GPU and DLA accelerator are measured separately. Then, single-input single-output subnetworks are split from the deep learning model network, and the runtime and energy consumption of each single-input single-output subnetwork are measured separately. The single-input single-output subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node.

[0011] Based on measurement data, an analytical performance and energy consumption prediction model is established. The runtime and energy consumption of the inference engine for different mapping configurations are derived from the data of fused nodes and sub-networks. A mapping optimization algorithm based on variable-length sliding windows is used to find the mapping configuration with the best performance and energy consumption.

[0012] Generate the optimal inference engine for deep learning applications based on the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

[0013] Furthermore, the extraction of candidate subnetworks with performance or energy consumption optimization potential from the deep learning model network includes:

[0014] A pure GPU inference engine is generated from a deep learning model, and layer fusion information is recorded. The layer fusion information from the pure GPU engine is used to construct layer fusion nodes, with each group of fused layers constituting a fusion node. i ;

[0015] Initialize the data structure of the entire deep learning model network: Net = {Node i} Perform topological sorting on all merged nodes to ensure that the merged node sequence number id i Satisfy topological order; initialize candidate subnetwork sets setSeqNet, setSglIONet and the processed node set OldNodes; all entry nodes of the deep learning model Node. i The set of intermediate nodes that make up the breadth-first search (BFSNodes) is formed.

[0016] If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes; skip if the current node has already been processed; use breadth-first search algorithm and intersection operation to match the unique entry point Node.i SglIONet type candidate subNet l The results are then saved to `setSglIONet`. If no SglIONet type subnetwork can be matched, a depth-first search algorithm and intersection operation are used to match the unique entry point as a Node. i SeqNet type subNet l The results are then stored in setSeqNet; if a subNet is successfully matched... l , then subNet l Export node l,out Store in BFSNodes new and subNet l Other nodes are stored in OldNodes; otherwise, Node is stored in OldNodes. i All child nodes are stored in BFSNodes new and Node i Store in OldNodes; then use BFSNodes new Update BFSNodes and proceed to the next iteration; return the result when BFSNodes is empty, obtaining the deep learning model network Net and two sets of candidate subnetworks; where SglIONet is a single-input single-output subnetwork and SeqNet is a sequential subnetwork. The sequential subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node.

[0017] Furthermore, the step of splitting the deep learning model network into multiple fusion nodes, measuring the runtime and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then splitting the deep learning model network into single-input single-output sub-networks and measuring the runtime and energy consumption data of each single-input single-output sub-network separately includes:

[0018] Measure the no-load static power pwr of the Jetson platform s For each Node in the Net i and each SglIONet in setSglIONet i Performance and power consumption data were measured and collected on both the GPU and DLA, including:

[0019] From Node i Or SglIONet i Generate an inference engine on the GPU or DLA and output the engine runtime data to a JSON file containing the runtime of each fusion layer; if generating the DLA engine fails, use isDLA. iSet to false and skip the current loop; otherwise, set isDLA. i Set to true; then extract performance data, starting with the average runtime from the JSON file. Run the current engine multiple times consecutively to measure the average power PWR of the heterogeneous computing platform. t Then, calculate the average dynamic power and dynamic energy consumption. If the accelerator type is DLA, also obtain the average time and average dynamic energy consumption of the input and output stages, using the average power of the DLA engine. To approximate the dynamic energy consumption of the input and output stages and Finally, we obtain a set of deep learning model fusion nodes Net containing performance and energy consumption data, as well as two sets of candidate sub-networks setSeqNet and setSglIONet.

[0020] Furthermore, the performance and energy consumption prediction model is established as follows:

[0021] min F = f obj (Eng map Time map )

[0022]

[0023] In the formula, min represents minimization; f obj (Eng map Time map ) = Eng map Time map f obj Eng represents the performance-energy consumption optimization objective function. map This represents the average energy consumption of a single run of the GPU-DLA hybrid engine under a deep learning workload mapping configuration; Time map This represents the average runtime of a single run of the GPU-DLA hybrid engine under a deep learning workload mapping configuration; Time gpu This represents the average time for a single run of the pure GPU inference engine; Map represents the deep learning workload mapping configuration.

[0024] Furthermore, the average runtime of the deep learning workload in the performance and energy consumption prediction model under the GPU-DLA hybrid engine in mapping configuration m is... The average runtime and average pre- and post-processing time of the multiple sub-networks under this mapping configuration m are calculated. Including GPUs Corresponding to DLA They are represented as follows:

[0025]

[0026] but:

[0027] In the formula, This represents the average runtime of the k-th subnetwork on the GPU. This represents the average runtime of the l-th subnetwork on the DLA. and These represent the preprocessing and postprocessing times of the l-th sub-network on the GPU, respectively. This represents the average runtime of the load mapped to the GPU under this mapping configuration m; This indicates the average runtime of the load mapped to DLA under this mapping configuration m.

[0028] Furthermore, the average energy consumption per run of the deep learning workload in the performance and energy consumption prediction model under the GPU-DLA hybrid engine in mapping configuration m is... The average energy consumption during operation and the average energy consumption before and after processing of multiple sub-networks under this mapping configuration m are calculated and expressed as follows:

[0029]

[0030] In the formula, pwr represents the average dynamic energy consumption of the k-th subnetwork on the GPU. s This represents the average static power of a heterogeneous SoC platform. This represents the average dynamic energy consumption of the l-th subnetwork on the DLA. and These represent the average dynamic energy consumption of the l-th sub-network during preprocessing and postprocessing on the GPU, respectively.

[0031] Furthermore, the step of finding the performance-energy-optimal mapping configuration using a mapping optimization algorithm based on a variable-length sliding window includes:

[0032] First, identify candidate subNets with potential for performance or energy efficiency optimization. m Then, a variable-length sliding window is used to determine the performance-optimal mapping configuration; specifically, this includes merging the two sets of candidate subnetworks, setSeqNet and setSglIONet, into a single set, setSubNet; for each candidate subnetwork... m Determine if a candidate subnetwork has potential for performance or energy consumption optimization, and add promising subnetworks to the candidate set: If a candidate subnetwork... m It is of type SgllONet, and the computation only considers the subNet. m MapInfo, a tuple of mapping configuration properties when mapping to DLAm If the average runtime of a single inference operation is no greater than the inference time using only the GPU, and the objective function value of a single inference operation is also less than the objective function value using only the GPU, then the subNet is considered to be... m It has the potential for performance or energy consumption optimization, and this subNet... m Add a candidate subnet to the candidate set candSubNet; if a candidate subnet is added to the candidate set candSubNet. m It is a SeqNet type, and the local objective function value F is first initialized with the objective function value when using only the GPU. lcl Calculate each subNet m MapInfo, a tuple of mapping configuration attributes for [j:k]. k Then, the subNet with better performance and energy consumption is selected. m [j:k] is saved as subNet lcl And update the local objective function value F lcl If subNet lcl If valid, then subNet lcl Add to the candidate set candSubNet;

[0033] In the fusion node set Net, each candidate subNet with performance or energy consumption optimization potential is selected. m The smallest fusion node number id contained in it i Sort candSubNets in ascending order; use a variable-length sliding window to create a mapping configuration Map of adjacent candidate subnetworks. j Then calculate the corresponding mapping configuration attribute tuple MapInfo. j And find the map j The number of connected subgraphs consisting of fused nodes, NumGraphs, is considered the current mapping configuration if the condition is met. j Better performance and energy efficiency, compared to Mqp j Save as a Map and update F lcl Ultimately, the optimal mapping configuration (Map) is obtained.

[0034] Furthermore, in the process of finding the performance-energy-optimal mapping configuration using a mapping optimization algorithm based on a variable-length sliding window, the inference time and energy consumption using only the GPU are calculated by the following formula:

[0035]

[0036] Furthermore, in finding the performance-energy-optimal mapping configuration using the mapping optimization algorithm based on a variable-length sliding window, the current mapping configuration (Map) is considered acceptable if the following three conditions are met. jBetter performance and energy efficiency: ① In Map mapping j The following conditions must be met: ① The time for one inference operation does not exceed the time for one inference operation when using only the GPU; ② The current objective function value is less than the historical best value; ③ The number of connected subnetworks mapped to the DLA does not exceed the maximum number of subnetworks that a DLA can support on average.

[0037] Furthermore, the step of generating the optimal inference engine for deep learning applications according to the optimal mapping configuration and searching for the optimal number of CUDA streams includes: using the TensorRT API to generate a TensorRT inference engine for the deep learning model according to the TensorRT API with the optimal performance ω energy consumption mapping configuration Map, then gradually increasing the number of CUDA streams, running the inference engine on these CUDA streams, and measuring the number of inference queries per second (qps). curr until QPS curr The process involves reducing the number of CUDA streams to find the optimal number; ultimately, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained.

[0038] According to another aspect of the present invention, a deep learning load mapping optimization system for a multi-accelerator heterogeneous SoC platform is proposed. For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes:

[0039] The subnetwork extraction module is configured to extract candidate subnetworks in the deep learning model network that have the potential for performance or energy consumption optimization.

[0040] The performance-energy consumption measurement module is configured to split the deep learning model network into multiple fusion nodes, measure the runtime and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split the deep learning model network into single-input single-output subnetworks, and measure the runtime and energy consumption data of each single-input single-output subnetwork separately; the single-input single-output subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0041] The mapping optimization module is configured to build an analytical performance and energy consumption prediction model based on measurement data, derive the runtime and energy consumption of the inference engine for different mapping configurations from the data of fused nodes and sub-networks, and use a mapping optimization algorithm based on a variable-length sliding window to find the performance-energy-optimal mapping configuration.

[0042] The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and to search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

[0043] The beneficial technical effects of the present invention are:

[0044] This invention proposes a method and system for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform. First, it extracts sub-networks with performance or energy consumption optimization potential. Next, it measures the performance and energy consumption data of the fused nodes and sub-networks. Then, it establishes an analytical performance and energy consumption prediction model to predict the runtime and energy consumption of the inference engine under different mapping configurations. Afterward, it uses a mapping optimization algorithm based on a variable-length sliding window to search for a mapping configuration with better performance and energy consumption. Finally, it generates the inference engine and iterates through the search for the optimal number of CUDA streams. This invention requires no user intervention and can automatically find the best performance-energy-consumption mapping configuration and generate the inference engine. Compared with existing methods such as JEDI, this invention offers greater performance improvements, greater energy savings, and a shorter optimization time.

[0045] This invention can automatically optimize the mapping of different layers in a deep learning network to various heterogeneous accelerators (GPUs and DLAs), automatically find mapping configurations that offer both performance improvements and energy savings, generate an inference engine, and automatically determine the optimal number of CUDA streams. Compared with existing methods, this invention offers greater performance improvements, greater energy savings, and a shorter optimization process. Attached Figure Description

[0046] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in and form part of this specification, and are used to further illustrate preferred embodiments of the invention and explain the principles and advantages of the invention.

[0047] Figure 1 This is a framework diagram of the deep learning load mapping optimization system proposed in this invention.

[0048] Figure 2 This is a schematic diagram of the sequential subnetwork in this invention.

[0049] Figure 3 This is a schematic diagram of a single-input, single-output subnetwork in this invention.

[0050] Figure 4 This is a diagram illustrating the possible execution timing and energy consumption of the inference task in this invention.

[0051] Figure 5 This is a diagram showing the performance and energy consumption optimization results in this invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present invention, exemplary embodiments or examples of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described embodiments or examples are merely some, not all, of the embodiments or examples of the present invention. All other embodiments or examples obtained by those skilled in the art based on the embodiments or examples of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0053] This invention proposes a method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform. The method includes: for a heterogeneous SoC platform equipped with GPUs and DLA accelerators:

[0054] S1. Extract candidate subnetworks with performance or energy consumption optimization potential from the deep learning model network;

[0055] S2. The deep learning model network is split into multiple fusion nodes, and the running time and energy consumption data of each fusion node on the GPU and DLA accelerator are measured separately. Then, the deep learning model network is split into single-input single-output sub-networks, and the running time and energy consumption data of each single-input single-output sub-network are measured separately. The single-input single-output sub-network is a connected sub-network with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node.

[0056] S3. Based on measurement data, establish an analytical performance and energy consumption prediction model, derive the runtime and energy consumption of the inference engine for different mapping configurations from the data of fused nodes and sub-networks, and use a mapping optimization algorithm based on variable-length sliding windows to find the mapping configuration with optimal performance and energy consumption.

[0057] S4. Generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

[0058] The present invention will now be described in detail.

[0059] 1. Deep Learning Load Mapping Optimization System Framework

[0060] The deep learning load mapping optimization system framework proposed in this invention is as follows: Figure 1As shown. First, extract sub-networks with performance or energy consumption optimization potential (①): Traverse and analyze the deep learning model network to extract sequential sub-networks and single-input single-output sub-networks, which are candidate sub-networks with performance or energy consumption optimization potential; then, measure and collect the performance and energy consumption data of each fusion node and sub-network on GPU and DLA (②): Split the deep learning model network into multiple fusion nodes, measure the running time and energy consumption data of each fusion node on different accelerators (GPU and DLA) separately, then split the single-input single-output sub-network from the deep learning model network, and measure the running time and energy consumption data of each single-input single-output sub-network separately; then, based on the measurement data, establish an analytical performance and energy consumption model. The energy consumption prediction model evaluates the performance and energy consumption of different mapping configurations and uses a mapping optimization algorithm based on a variable-length sliding window to find the performance-energy-optimal mapping configuration (③): For sequential sub-networks, a variable-length sliding window is used to determine the performance-energy-optimal sub-sequences. These optimal sub-sequences and all single-input single-output sub-networks form candidate sub-networks. The variable-length sliding window is then used to determine the candidate sub-network sub-sequences with the best performance-energy consumption, thus obtaining the optimized mapping configuration. Finally, the optimal inference engine is generated according to the optimal mapping configuration, and the optimal number of CUDA streams is searched (④): The deep learning application inference engine is generated according to the optimized mapping configuration. The number of CUDA streams is set incrementally from small to large. The optimal number of CUDA streams is searched through the traversal. CUDA is a parallel computing platform and programming model provided by NVIDIA for its GPUs, DLAs and other devices. CUDA streams are used to manage the characteristics of tasks on GPUs and DLAs. Tasks in the same CUDA stream are executed sequentially, and tasks in different CUDA streams can be executed overlappingly.

[0061] 2. Formalization of the Deep Learning Load Mapping Optimization Problem

[0062] This section formalizes the problem of mapping neural network layers to appropriate accelerators (GPUs and DLAs) to achieve optimal performance and energy consumption. Table 1 lists the relevant symbols and meanings for the formalization of the problem. The measured performance and energy consumption data of the entire neural network, Net, are provided. Net consists of many fused nodes. i Composition. A fusion node. i It contains one or more layers because TensorRT performs layer fusion optimizations when generating the inference engine. Naturally, Node... i This is the basic granularity for collecting performance and energy consumption data and mapping deep learning workloads. Additionally, some subNets with performance and energy consumption optimization potential can be selected. iAs a measure of performance and energy consumption data, and for mapping deep learning workloads, TensorRT may perform additional optimizations among the fusion nodes when processing neural networks with multiple fusion nodes. Therefore, measuring the entire subnetwork as a whole provides more direct and accurate performance-energy consumption data than indirect results derived from performance-energy consumption data from multiple fusion nodes.

[0063] Formula (1) defines a fusion node. i ,setPnt i and setChd i Save topology information. All types of neural network layers can run on a GPU, but due to limitations of DLA and TensorRT, some types of layers cannot run on DLA. Use isDLA to address this. i Mark Node i Can it run on DLA? (Remaining 10 variables) arrive This refers to the performance-energy consumption data that needs to be measured for a fusion node. Dynamic power refers to the power after deducting static power (pwr). s The same applies to the remaining power and dynamic energy consumption afterward. Due to the characteristics of the Jetson platform and TensorRT, additional GPU cores are used to copy and format input and output data before and after the DLA workload. These additional overheads are handled by... express.

[0064]

[0065] Formula (2) defines a subNet. i SubNet i Includes multiple merge nodes j These fused nodes form a subNet. i (Formula (3)). SubNet i The included symbols are also shown in Table 1.

[0066]

[0067] setNode i ={Node in , ..., Node j , ..., Node out} (3)

[0068] Table 1 Formal Symbol Table for Deep Learning Load Mapping Optimization Problem

[0069]

[0070]

[0071] Based on the above definition, a formal representation of the deep learning load mapping performance and energy consumption optimization problem is given:

[0072]

[0073] The goal of the mapping optimization problem is to find the optimal mapping configuration (Map) that optimizes the performance-energy consumption objective function (f). obj The minimum performance is achieved, while the inference engine performance is no less than that of using only the GPU ($Time). map ≤Time gpu This invention selects the energy consumption delay product (EDP) as the performance-energy consumption optimization objective function, as shown in formula (5). Using the energy consumption delay product (EDP) as the sole optimization objective simplifies the optimization process and allows for simultaneous consideration of performance and energy consumption optimization. Users can also specify other optimization objective functions as needed, such as total energy consumption, energy consumption delay square product (EDP), etc. 2 P, etc. Merging nodes in the Map. i It is mapped to DLA, while other fusion nodes of the deep learning model are mapped to GPU by default.

[0074] f obj (Eng map Time map ) = Eng map Time map (5)

[0075] 3. Performance and energy consumption data measurement

[0076] This section discusses the various fusion nodes. i and subNets with performance or energy consumption optimization potential i Perform performance and energy consumption data measurements. (Node fusion) i The included layers can be determined by parsing the output log file of the trtexec tool. However, extracting subnetworks with performance or energy consumption optimization potential is not easy. Therefore, this section first proposes a candidate subnetwork extraction algorithm, and then designs an algorithm to measure each fusion node. i and candidate subNet i .

[0077] 3.1 Candidate Subnetwork Extraction Algorithm

[0078] Because additional GPU cores are used to copy and format input and output data before and after the DLA workload, which is very costly, a connected subNet with as few ingress and egress nodes as possible and a sufficient number of internal nodes is preferable. i They are more likely to have performance or energy consumption optimization potential. We will focus on two common subnetwork types that conform to this principle.

[0079] Definition 1: A Sequential Subnet (SeqNet) is a connected subnet. i It has a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node, such as... Figure 2 As shown.

[0080] Definition 2: A single-input, single-output subnet (SglIONet) is a connected subnet. i It has a unique entry node and a unique exit node, and there are at least two distinct paths from the entry node to the exit node, such as... Figure 3 As shown.

[0081] Sequential subnetwork SeqNet and single-input single-output subnetwork SglIONet are considered as candidate subnetworks with potential for performance or energy consumption optimization. Both SeqNet and SglIONet can be expressed as Equation (2). The unique entry node and exit node are denoted as Node. in and Node out .

[0082] The following is a candidate subnetwork extraction algorithm (Algorithm 1) designed to extract candidate subnetworks with performance or energy consumption optimization potential. First, the trtexec tool is used to generate a pure GPU inference engine from the deep learning model and record layer fusion information (line 1). At this point, trtexec performs layer fusion optimization for the GPU; therefore, the layer fusion configuration is more GPU-friendly. Even with a GPU-friendly layer fusion configuration, it is still possible to extract subnetworks by merging nodes. i Mapping to DLA to find performance or energy consumption optimization potential indicates that an optimized GPU-DLA hybrid engine will outperform a pure GPU engine. Therefore, for conservative reasons, a pure GPU engine layer fusion configuration is used to build layer fusion nodes, with each group of fused layers constituting a fusion node. i Next, initialize the data structure of the entire deep learning model network: Net = {Node} i At this point, a topological sort will be performed on all merged nodes to ensure that the merged node sequence number id is correct. iThe topological order is satisfied (line 1). Next, the candidate subnetwork set (setSeqNet and setSglIONet) and the processed node set OldNodes are initialized (line 2). All entry nodes of the deep learning model form the intermediate node set for breadth-first search (line 3).

[0083] If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes (lines 4-27). Skip if the current node has already been processed (lines 7-9). Use breadth-first search algorithm and intersection operation to match the unique entry point as Node. i Candidate subNets of the SgllONet type l The results are then saved to `setSglIONet` (lines 10-12). If no SglIONet type subnetwork can be matched, a depth-first search algorithm and intersection operation are used to match the unique entry point as a Node. i SeqNet type subNet l And save the results in setSeqNet (lines 13-18). If a subNet is successfully matched... l , then subNet l Export node l,out Store in BFSNodes new and subNet l Other nodes are stored in OldNodes (lines 19-21). Otherwise, the Node is stored in OldNodes. i All child nodes are stored in BFSNodes new and Node i Store in OldNodes (lines 22-25). Then use BFSNodes. new Update BFSNodes and proceed to the next iteration of the while loop (line 27). Return the result when BFSNodes is empty (line 29).

[0084]

[0085]

[0086]

[0087] 3.2 Performance and Energy Consumption Data Measurement Algorithm

[0088] This section designs algorithms to accurately measure performance and energy consumption data, preparing foundational data for subsequent performance and energy consumption modeling. (Node fusion) iThis is the basic granularity for performance and energy efficiency modeling, and each requires measuring performance and energy consumption data. Based on experimental data and observational analysis, for the sequential subnetwork SeqNet, from its contained fusion nodes... i The data allows us to deduce the performance and energy consumption of the sequential subnetwork SeqNet. For the single-input, single-output subnetwork SglIONet, there are multiple paths between the input and output. TensorRT performs more optimizations during the generation engine, which leads to improvements in the performance of the fusion node. i The performance and energy consumption results of the single-input single-output (SGLIONet) subnetwork derived from data are inaccurate and often overestimate energy consumption and execution time. Therefore, it is necessary to perform overall measurements on the SGLIONet subnetwork to obtain sufficiently accurate performance and energy consumption data.

[0089] The following is an algorithm (Algorithm 2) designed to measure performance and energy consumption data for each fusion node. i and single-input single-output subnetwork SglIONet i Performance and energy consumption data were obtained. First, Algorithm 1 was used to obtain the deep learning model network Net and two sets of candidate subnetworks (row 1). Then, the idle static power pwr of the Jetson platform was measured. s This is for subsequent calculations of dynamic power and energy consumption (line 2). Then, for each Node in the Net... i and each SgllONet in setSglIONet i Measure and collect performance and power consumption data on both the GPU and DLA (lines 3-19). First, use the trtexec tool from Node.js. i Or SglIONet i Generate the inference engine on the corresponding device (GPU or DLA) and output the engine runtime data to a JSON file (line 5). This JSON file contains the runtime of each fusion layer, as shown in lines 3-5 of Table 2. Before and after running the workload on the DLA, the input and output data may need to be copied and formatted on the GPU; the input and output times are shown in lines 2 and 6 of Table 2, respectively. Note that when multiple Nodes... i These Nodes, when running continuously on DLA i There is no data copying or formatting between them. If generating the DLA engine fails, isDLA will be used. i Set to false and skip the current loop (lines 6-9); otherwise, set isDLA. i Set to true (line 10).

[0090]

[0091] Next, extract performance data, starting with the average runtime from the JSON file. This is the sum of the average times in rows 3-5 of Table 2 (row 11). Then, run the current engine multiple times consecutively to measure the average power PWR of the heterogeneous computing platform. t (Line 12). Then, calculate the average dynamic power and dynamic energy consumption (line 13). If the accelerator type is DLA, also obtain the average time and average dynamic energy consumption for the input and output stages (lines 14-17). Due to the limitations of the Jetson heterogeneous computing platform, input or output power cannot be measured separately. The average power of the DLA engine can be used. To approximate the dynamic energy consumption of the input and output stages and (Lines 15-16). Finally, we obtain the deep learning model fusion node set Net, which contains performance and energy consumption data, as well as two sets of candidate sub-networks setSeqNet and setSglIONet (line 20).

[0092] Table 2 JSON file examples

[0093]

[0094] 4. Performance and energy consumption modeling and deep learning load mapping

[0095] Based on the performance and energy consumption data of the fusion nodes and candidate subnetworks measured in the previous section, this section first designs a performance and energy consumption model to evaluate the performance and energy consumption of the TensorRT engine generated under a specific mapping configuration. Then, under the guidance of the performance and energy consumption model, an algorithm is designed to search for the performance-energy-optimal mapping configuration and the optimal number of matching CUDA streams.

[0096] 4.1 Performance and Energy Consumption Model Design

[0097] First, a performance model, i.e., a runtime model, is designed to evaluate the performance of the TensorRT engine generated under a specific mapping configuration. Without loss of generality, assume that m subnetworks are mapped to DLA, and these subnetworks constitute the mapping configuration denoted as Map. m As shown in Equation (6). The remaining fusion nodes in the deep learning model are mapped to the GPU by default.

[0098] Map m ={subNet l} (6)

[0099] All candidate subNets l All fusion nodes k Composed of a collection of MapNodes m As shown in formula (7).

[0100]

[0101] Map configuration m All ingress fusion nodes and egress fusion nodes of the neutron network form two sets, EnNodes. m and ExNodes m As shown in formula (8).

[0102]

[0103] If a subnet l If a subnet SglIONet is a measured single-input single-output subnet, then its runtime data has been recorded. For subnets that have not been measured as a whole, their runtime data can be calculated using formula (9). Wherein, subNet l DLA runtime It contains all the Nodes. k DLA runtime The sum; subNet l Input phase time It is all its entry fusion nodes enNode i Input phase time The sum; subNet l Output phase time It is all its export fusion nodes exNode i Output phase time The sum of .

[0104]

[0105] Based on each fusion node i and each subNet l The runtime data can be used to deduce the mapping configuration Map. m The running times of the GPU and DLA in a single inference are shown in equations (10) and (11). The GPU running time is equal to the sum of the running time of the fusion node mapped to the GPU and the input / output phase time of the fusion node mapped to the DLA; the DLA running time is equal to the sum of the DLA running times mapped to the fusion node of the DLA.

[0106]

[0107] To derive the average runtime and energy consumption of the inference task, we first analyze the possible execution sequence of multiple consecutive executions of this inference task, such as... Figure 4As shown. Without loss of generality, it can be assumed that a TensorRT inference engine comprises four stages: GPU load, DLA input stage, DLA load, and DLA output stage. The Jetson Orin NX 16GB platform has two DLAs (DLA0 and DLA1), supporting two DLA workloads running simultaneously, such as D1 and D2. When using multiple CUDA streams, the three GPU activity stages (GPU load, DLA input stage, and DLA output stage) and DLA load can also run overlappingly, such as D1, D2, G3, and I3. When running the TensorRT inference engine across multiple CUDA streams, the TensorRT runtime can automatically schedule and overlap these four stages.

[0108] Based on the above analysis, the running time of 2n inference tasks using the same mapping configuration inference engine can be derived, as shown in formula (12).

[0109]

[0110] in The total overlap time between the GPU and DLA in one inference is the minimum of twice the GPU runtime and the DLA runtime, as shown in formula (13).

[0111]

[0112] Substituting formula (13) into formula (12) yields the average running time $Time^{map_m}$ for one inference operation, as shown in formula (14).

[0113]

[0114] Based on the above runtime model, an energy consumption model is designed. If a subNet... l If a subnet SglIONet is a measured single-input single-output subnet, then its energy consumption data has been recorded. For subnets that have not been measured as a whole, their energy consumption data can be calculated using formula (15). l DLA dynamic energy consumption It contains all the Nodes. k DLA dynamic energy consumption The sum of subNet. l Dynamic energy consumption during the input phase It is all its entry fusion nodes enNode i Dynamic energy consumption during the input phase The sum of subNet. l Dynamic energy consumption during the output stage It is all its export fusion nodes exNodei Dynamic energy consumption during the output stage The sum of .

[0115]

[0116] Based on each fusion node i and each subNet l The energy consumption and runtime data can be used to deduce the mapping configuration Map. m The average energy consumption of a single inference and the performance-energy consumption optimization objective function are shown in formulas (16) and (17).

[0117]

[0118] This concludes the derivation of the deep learning model's mapping configuration Map. m The performance and energy consumption models of the generated inference engine are shown in equations (14), (16), and (17). For brevity, a mapping configuration Map is defined. m MapInfo attribute tuple m This includes the mapping configuration itself and the next inference time for that mapping configuration. Performance-Energy Consumption Objective Function Value As shown in formula (18).

[0119]

[0120] 4.2 Subnetwork Mapping Algorithm Oriented to Performance and Energy Consumption Optimization

[0121] Guided by the performance and energy consumption model designed in the previous section, this section designs a subnetwork mapping algorithm (Algorithm 3) for performance and energy consumption optimization to determine the optimal performance-energy consumption mapping configuration. First, candidate subnetworks with performance or energy consumption optimization potential are identified. m (Lines 1-21), and then use a variable-length sliding window to determine the performance-energy-optimal mapping configuration (Lines 22-34).

[0122] Merge the two sets of candidate subnetworks, setSeqNet and setSglIONet, into a single set, setSubNet (line 2). For each candidate subnetwork, subNet... m Determine if a candidate subnetwork has performance or energy consumption optimization potential, and add promising subnetworks to the candidate set (lines 3-25). m It is of type SglIONet, and the computation only considers the subNet. m MapInfo, a tuple of mapping configuration properties when mapping to DLA m(Lines 4-6); If the average runtime of a single inference is no greater than the inference time using only the GPU and the objective function value of a single inference is also less than the objective function value using only the GPU, then the subNet is considered to be... m It has the potential for performance or energy consumption optimization, and this subNet... m Add the candidate set candSubNet. The inference time and energy consumption using only the GPU are calculated by formula (19).

[0123]

[0124] If a candidate subNet m It is a SeqNet type, and the local objective function value F is first initialized with the objective function value when using only the GPU. lcl (Line 11). Calculate each subNet. m MapInfo, a tuple of mapping configuration attributes for [j:k]. k (Lines 12-15) Then, the subNet with better performance and energy consumption is selected. m [j:k] is saved as subNet lcl And update the local objective function value F lcl (Lines 16-18). If subNet lcl If valid, then subNet lcl Add the candidate set candSubNet (lines 21-23).

[0125]

[0126]

[0127]

[0128] In the set of merged nodes Net, all merged nodes Node i All follow topological sorting. Therefore, they are sorted according to each candidate subNet with performance or energy consumption optimization potential. m The smallest fusion node number id contained in it i Sort candSubNet (line 26) in ascending order, which will allow you to sort all subNets. m All follow topological sorting. At this point, the adjacent candidate subNets are sorted. m They are more likely to be topologically adjacent. Adjacent candidate subNets are considered. m Mapping the entirety to DLA is more likely to eliminate these candidate subnets. mThis reduces input and output overhead, thereby further improving performance and energy consumption. Therefore, a variable-length sliding window can be used to create a mapping configuration Map for adjacent candidate subnetworks. j (Lines 28-30). Then calculate the corresponding mapping configuration attribute tuple MapInfo. j And find the map j The number of connected subgraphs consisting of merged nodes is NumGraphs (lines 31-32). The current mapping configuration is considered to be Map if the following three conditions are met. j Better performance and energy efficiency, Map j Save as a Map and update F lcl (Lines 33-35). ① In the Map mapping j The following conditions must be met: ① The time for one inference operation does not exceed the time for one inference operation using only the GPU. ② The current objective function value is less than the historical best value. ③ The number of connected subnetworks mapped to the DLA does not exceed the maximum number of subnetworks that a single DLA can support on average (10 subnetworks on the Jetson Orin NX 16GB platform). This ensures that the inference task can utilize all DLAs simultaneously (2 DLAs on the Jetson Orin NX 16GB platform). Finally, the optimal mapping configuration (Map, line 38) can be obtained.

[0129] 4.3 Algorithm for Searching for the Optimal Number of CUDA Streams

[0130] Integrating the above algorithms, the JDIMO algorithm framework (Algorithm 4) is designed to generate a performance-optimal TensorRT inference engine and search for the optimal number of matching CUDA streams. First, Algorithms 2 and 3 are used to determine the performance-optimal mapping configuration (lines 1-2). Then, the TensorRT API is used to generate the TensorRT inference engine for the deep learning model according to the performance-optimal mapping configuration (line 3).

[0131] Then gradually increase the number of CUDA streams and run the inference engine on these CUDA streams, measuring the number of inference queries per second (QPS). curr until QPS curr The optimal number of CUDA streams is determined by reducing the number of streams (lines 4-12). These CUDA streams will cycle through all DLAs (interleaving DLA0 and DLA1 on the Jetson Orin NX 16GB platform) to fully utilize the computing power of the heterogeneous computing platform. Finally, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained (line 13).

[0132]

[0133] The technical effects of the present invention were further verified through experiments.

[0134] Optimization objective: Minimize the energy consumption-delay product, i.e., f obj (Eng map Time map ) = Eng map Time map (Formula (5));

[0135] Experimental hardware platform: Jetson Orin NX 16GB multi-accelerator heterogeneous SoC computing platform; Deep learning models under test: VGG-16, VGG-19, MobileNet-v2, RetinaNet, YOLOv4.

[0136] Comparison method:

[0137] ①GPU-only, TensorRT's default strategy is to use only the GPU;

[0138] ②DLA-possible, TensorRT's default strategy two, try to use DLA;

[0139] ③JEDI [1] A mapping method with good existing optimization effect, which uses both GPU and DLA, divides the deep learning model into multiple stages, generates multiple engines accordingly, and runs them in a pipeline.

[0140] ④ The method of this invention - JDIMO.

[0141] It should be noted that all deep learning models (ONNX files) are downloaded from the ONNX Model Zoo code repository (https: / / github.com / onnx / models#onnx-model-zoo).

[0142] Performance and energy consumption optimization results are as follows Figure 5 As shown in the figure, the normalized energy-latency product (EDP) for one inference (lower is better), energy consumption for one inference (lower is better), inference queries per second (QPS, which represents performance and is higher is better), and average power consumption (lower is better) are displayed based on GPU-only data.

[0143] As can be seen, the method of this invention exhibits the lowest EDP, the lowest energy consumption, and the highest QPS in JDIMO, while the average power consumption is basically the same as that of the GPU-only method.

[0144] This invention (JDIMO): Average normalized EDP = 0.5, average normalized power consumption = 0.72, average normalized QPS = 1.42, average normalized power = 1.0. JEDI: Average normalized EDP = 0.68, average normalized power consumption = 0.82, average normalized QPS = 1.26, average normalized power = 1.02. DLA-possible: Average normalized EDP = 2.12, average normalized power consumption = 0.96, average normalized QPS = 0.86, average normalized power = 0.72. GPU-only: Average normalized EDP = 1.0, average normalized power consumption = 1.0, average normalized QPS = 1.0, average normalized power = 1.0.

[0145] Compared to GPU-only methods, the JDIMO method of this invention reduces EDP by 50%, saves 28% of energy consumption, and increases QPS by 42%, with essentially the same power consumption. Compared to the DLA-possible method, the JDIMO method of this invention reduces EDP by 76.4%, saves 25% of energy consumption, increases QPS by 65.1%, and increases power consumption by 38.9%. Although the DLA-possible method has lower power consumption, its normalized QPS is only 0.2, resulting in significant performance loss, and its normalized energy consumption is 1.8, indicating a significant increase in energy consumption rather than a saving. Compared to the JEDI method, the JDIMO method of this invention reduces EDP by 26.5%, saves 12.1% of energy consumption, increases QPS by 12.7%, and reduces power consumption by 2.0%.

[0146] Table 3 shows the optimized mapping configurations and optimization times for the four methods on five deep learning models. It can be seen that, overall, JDIMO maps more neural network layers to DLA than the JEDI method. However, comparing the DLA-possible method alone, and considering the performance and energy consumption optimization results above, it demonstrates that JDIMO has found a more balanced mapping configuration between performance and energy consumption.

[0147] While the optimization process time of the JDIMO method is longer than that of the GPU-only and DLA-possible methods, it is shorter than that of the JEDI method. Compared with the JEDI method, the JDIMO method of this invention reduces the optimization process time by an average of 93.6%.

[0148] The above data and analysis demonstrate that the method of the present invention can automatically find a better mapping configuration for deep learning workloads on multiple heterogeneous accelerators and generate an inference engine, find the optimal number of CUDA streams, improve performance while reducing energy consumption, and significantly shorten the optimization process time.

[0149] Table 3 Optimized mapping configuration and optimization process time

[0150]

[0151] 1. "GPU:", "DLA0:", and "DLA1:" represent mappings to GPU, DLA0, and DLA1, respectively. "DLA:" indicates mapping to any one or both DLAs. "-" indicates that consecutive numbers are omitted. The circled numbers correspond to the various engines generated by JEDI.

[0152] 2. For brevity, only nodes mapped to DLA are shown. The trtexec output log does not show intermediate nodes; "..." indicates omitted intermediate nodes.

[0153] Another embodiment of the present invention proposes a deep learning load mapping optimization system for a multi-accelerator heterogeneous SoC platform. For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes:

[0154] The subnetwork extraction module is configured to extract candidate subnetworks in the deep learning model network that have the potential for performance or energy consumption optimization.

[0155] The performance-energy consumption measurement module is configured to split the deep learning model network into multiple fusion nodes, measure the runtime and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split the deep learning model network into single-input single-output subnetworks, and measure the runtime and energy consumption data of each single-input single-output subnetwork separately; the single-input single-output subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node;

[0156] The mapping optimization module is configured to build an analytical performance and energy consumption prediction model based on measurement data, derive the runtime and energy consumption of the inference engine for different mapping configurations from the data of fused nodes and sub-networks, and use a mapping optimization algorithm based on a variable-length sliding window to find the performance-energy-optimal mapping configuration.

[0157] The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and to search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

[0158] The functionality of the deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform in this embodiment of the invention can be described by the aforementioned deep learning load mapping optimization method on a multi-accelerator heterogeneous SoC platform. Therefore, for the parts not described in detail in the system embodiment, please refer to the above method embodiment, and they will not be repeated here.

[0159] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

[0160] The following documents are cited in this invention:

[0161] [1]Jeong EJ, Kim J, Ha S. Tensorrt-based framework and optimization methodology for deep learning inference on jetson boards [J]. ACM Transactionson Embedded Computing Systems (TECS), 2022, 21(5):1-26.

Claims

1. A method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform, characterized in that, include: For heterogeneous SoC platforms equipped with GPUs and DLA accelerators: Extract candidate subnetworks with performance or energy consumption optimization potential from deep learning model networks; The deep learning model network is split into multiple fusion nodes, and the runtime and energy consumption of each fusion node on the GPU and DLA accelerator are measured separately. Then, single-input single-output subnetworks are split from the deep learning model network, and the runtime and energy consumption of each single-input single-output subnetwork are measured separately. The single-input single-output subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node. Based on measurement data, an analytical performance and energy consumption prediction model is established, and the runtime and energy consumption of the inference engine with different mapping configurations are derived from the data of fused nodes and sub-networks. And a mapping optimization algorithm based on a variable-length sliding window is used to find the performance-energy-optimal mapping configuration, including: First, identify candidate subNets with potential for performance or energy efficiency optimization. m Then, a variable-length sliding window is used to determine the performance-optimal mapping configuration; specifically, this includes merging the two sets of candidate subnetworks, setSeqNet and setSglIONet, into a single set, setSubNet; for each candidate subnetwork... m Determine if a candidate subnetwork has potential for performance or energy consumption optimization, and add promising subnetworks to the candidate set: If a candidate subnetwork... m It is of type SglIONet, and the computation only considers the subNet. m MapInfo, a tuple of mapping configuration properties when mapping to DLA m If the average runtime of a single inference operation is no greater than the inference time using only the GPU, and the objective function value of a single inference operation is also less than the objective function value using only the GPU, then the subNet is considered to be... m It has the potential for performance or energy consumption optimization, and this subNet... m Add a candidate subnet to the candidate set candSubNet; if a candidate subnet is added to the candidate set candSubNet. m It is a SeqNet type, and the local objective function value F is first initialized with the objective function value when using only the GPU. lcl Calculate each subNet m MapInfo, a tuple of mapping configuration attributes for [j:k]. k Then, the subNet with better performance and energy consumption is selected. m [j:k] is saved as subNet lcl And update the local objective function value F lcl If subNet lcl If valid, then subNet lcl Add the candidate set candSubNet; the inference time and energy consumption using only the GPU are calculated by the following formula: In the fusion node set Net, each candidate subNet with performance or energy consumption optimization potential is selected. m The smallest fusion node number id contained in it i The ascending-order sorting of `candSubNet` uses a variable-length sliding window to create a mapping configuration Map of adjacent candidate subnetworks. j Then calculate the corresponding mapping configuration attribute tuple MapInfo. j And find the map j The number of connected subgraphs consisting of merged nodes included in the map (MumGraphs) is considered to be the current mapping configuration if the condition is met. j Better performance and energy efficiency, Map j Save as a Map and update F lcl Ultimately, the optimal mapping configuration (Map) is obtained. Generate the optimal inference engine for deep learning applications based on the optimal mapping configuration, and search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

2. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 1, characterized in that, The candidate subnetworks with performance or energy consumption optimization potential extracted from the deep learning model network include: A pure GPU inference engine is generated from a deep learning model, and layer fusion information is recorded. The layer fusion information from the pure GPU engine is used to construct layer fusion nodes, with each group of fused layers constituting a fusion node. i ; Initialize the data structure of the entire deep learning model network: Net = {Node i } Perform topological sorting on all merged nodes to ensure that the merged node sequence number id i Satisfy topological order; initialize candidate subnetwork sets setSeqNet, setSglIONet and the processed node set OldNodes; all entry nodes of the deep learning model Node. i The set of intermediate nodes that make up the breadth-first search (BFSNodes) is formed. If BFSNodes is not empty, process each unprocessed fusion node in BFSNodes; skip if the current node has already been processed; use breadth-first search algorithm and intersection operation to match the unique entry point Node. i SglIONet type candidate subNet l The results are then saved to `setSglIONet`. If no SglIONet type subnetwork can be matched, a depth-first search algorithm and intersection operation are used to match the unique entry point as a Node. i SeqNet type subNet l The results are then stored in setSeqNet; if a subNet is successfully matched... l , then subNet l Export node l，out Store in BFSNodes new and subNet l Other nodes are stored in OldNodes; otherwise, Node is stored in OldNodes. i All child nodes are stored in BFSNodes new and Node i Store in OldNodes; then use BFSNodes new Update BFSNodes and proceed to the next iteration; return the result when BFSNodes is empty, obtaining the deep learning model network Net and two sets of candidate subnetworks; where SglIONet is a single-input single-output subnetwork and SeqNet is a sequential subnetwork. The sequential subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there is only one path from the entry node to the exit node.

3. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 2, characterized in that, The process of splitting the deep learning model network into multiple fusion nodes, measuring the runtime and energy consumption of each fusion node on the GPU and DLA accelerator separately, and then splitting the deep learning model network into single-input single-output sub-networks and measuring the runtime and energy consumption of each single-input single-output sub-network separately includes: Measure the no-load static power pwr of the Jetson platform s For each Node in the Net i and each SglIONet in setSglIONet i Performance and power consumption data were measured and collected on both the GPU and DLA, including: From Node i Or SglIONet i Generate an inference engine on the GPU or DLA and output the engine runtime data to a JSON file containing the runtime of each fusion layer; if generating the DLA engine fails, use isDLA. i Set to false and skip the current loop; otherwise, set isDLA. i Set to true; then extract performance data, starting with the average runtime from the JSON file. Run the current engine multiple times consecutively to measure the average power PWR of the heterogeneous computing platform. t Then, calculate the average dynamic power and dynamic energy consumption. If the accelerator type is DLA, also obtain the average time and average dynamic energy consumption of the input and output stages, using the average power pwr of the DLA engine. i dla To approximate the dynamic energy consumption of the input and output stages and Finally, we obtain a set of deep learning model fusion nodes Net containing performance and energy consumption data, as well as two sets of candidate sub-networks setSeqNet and setSglIONet.

4. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 3, characterized in that, The performance and energy consumption prediction model is established as follows: In the formula, min represents minimization; f obj (Eng map Time map ) = Eng map Time map f obj Eng represents the performance-energy consumption optimization objective function. map This represents the average energy consumption of a single run of the GPU-DLA hybrid engine under a deep learning workload mapping configuration; Time map This represents the average runtime of a single run of the GPU-DLA hybrid engine under a deep learning workload mapping configuration; Time gpu This represents the average time for a single run of the pure GPU inference engine; Map represents the deep learning workload mapping configuration.

5. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 4, characterized in that, The average runtime of the deep learning workload in the GPU-DLA hybrid engine under mapping configuration m in the performance and energy consumption prediction model. The average runtime and average pre- and post-processing time of the multiple sub-networks under this mapping configuration m are calculated. Including GPUs Corresponding to DLA They are represented as follows: In the formula, This represents the average runtime of the k-th subnetwork on the GPU. This represents the average runtime of the l-th subnetwork on the DLA. and These represent the preprocessing and postprocessing times of the l-th sub-network on the GPU, respectively. This represents the average runtime of the load mapped to the GPU under this mapping configuration m; This indicates the average runtime of the load mapped to DLA under this mapping configuration m.

6. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 5, characterized in that, The average energy consumption of a single run of the deep learning workload in the GPU-DLA hybrid engine under mapping configuration m in the performance and energy consumption prediction model. The average energy consumption during operation and the average energy consumption before and after processing of multiple sub-networks under this mapping configuration m are calculated and expressed as follows: In the formula, pwr represents the average dynamic energy consumption of the k-th subnetwork on the GPU. s This represents the average static power of a heterogeneous SoC platform. This represents the average dynamic energy consumption of the l-th subnetwork on the DLA. and These represent the average dynamic energy consumption of the l-th sub-network during preprocessing and postprocessing on the GPU, respectively.

7. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 6, characterized in that, In the process of finding the performance-energy-optimal mapping configuration using the mapping optimization algorithm based on a variable-length sliding window, the current mapping configuration is considered optimal if the following three conditions are met. j Better performance and energy efficiency: ① In Map mapping j The following conditions must be met: ① The time for one inference operation does not exceed the time for one inference operation when using only the GPU; ② The current objective function value is less than the historical best value; ③ The number of connected subnetworks mapped to the DLA does not exceed the maximum number of subnetworks that a DLA can support on average.

8. The method for optimizing deep learning load mapping on a multi-accelerator heterogeneous SoC platform according to claim 7, characterized in that, The process of generating the optimal inference engine for deep learning applications according to the optimal mapping configuration and searching for the optimal number of CUDA streams includes: using the TensorRT API to generate a TensorRT inference engine for the deep learning model according to the performance-energy-efficient mapping configuration Map, then gradually increasing the number of CUDA streams, running the inference engine on these CUDA streams, and measuring the number of inference queries per second (qps). curr until QPS curr The process involves reducing the number of CUDA streams to find the optimal number; ultimately, the optimal TensorRT inference engine and the matching optimal number of CUDA streams are obtained.

9. A deep learning load mapping optimization system on a multi-accelerator heterogeneous SoC platform, the system being used to implement the deep learning load mapping optimization method on a multi-accelerator heterogeneous SoC platform according to any one of claims 1-8, characterized in that, For a heterogeneous SoC platform equipped with a GPU and a DLA accelerator, the system includes: The subnetwork extraction module is configured to extract candidate subnetworks in the deep learning model network that have the potential for performance or energy consumption optimization. The performance-energy consumption measurement module is configured to split the deep learning model network into multiple fusion nodes, measure the runtime and energy consumption data of each fusion node on the GPU and DLA accelerator separately, and then split the deep learning model network into single-input single-output subnetworks, and measure the runtime and energy consumption data of each single-input single-output subnetwork separately; the single-input single-output subnetwork is a connected subnetwork with a unique entry node and a unique exit node, and there are at least two different paths from the entry node to the exit node; The mapping optimization module is configured to build an analytical performance and energy consumption prediction model based on measurement data, derive the runtime and energy consumption of the inference engine for different mapping configurations from the data of fused nodes and sub-networks, and use a mapping optimization algorithm based on a variable-length sliding window to find the performance-energy-optimal mapping configuration. The engine generation module is configured to generate the optimal inference engine for deep learning applications according to the optimal mapping configuration, and to search for the optimal number of CUDA streams to complete the deep learning workload mapping optimization.

Images