Multi-model inference acceleration system and method for automatic driving full-scene perception

Abstract

The application provides a multi-model inference acceleration system and method for automatic driving full-scene perception, converts multiple DNN models in automatic driving full-scene perception into a DAG-based calculation graph, designs a scheduling framework POS, adopts a deep reinforcement learning DRL method to obtain an optimal POS scheduling strategy, constructs an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select an optimal parallel strategy, and completes automatic driving full-scene perception; compared with the most advanced deep learning inference framework and specific scheduling method, the POS can always achieve 1.2-3.9 times of inference acceleration and 28%-55% of GPU utilization rate improvement. In addition, the search overhead of optimal scheduling is shortened by 1.2-2.9 times compared with the baseline.

Description

Technical Field

[0001] This invention belongs to the field of full-scene perception for autonomous driving, specifically, it relates to a multi-model inference acceleration system and method for full-scene perception of autonomous driving. Background Technology

[0002] Autonomous driving requires inference of multiple deep learning perception models on computing devices with GPUs. During the GPU execution of deep neural network (DNN) model inference, traditional deep learning frameworks typically abstract the DNN model as a computational graph (usually a directed acyclic graph, DAG) constructed from the dependencies between operators.

[0003] The following process is required when running the GPU scheduler:

[0004] (1) The scheduler selects an operator from the queue of operators waiting to be scheduled and sends it to the appropriate worker thread in the GPU.

[0005] (2) The scheduler checks the type and shape of the input tensor and calculates the type and shape of the output tensor.

[0006] (3) The scheduler allocates idle GPU kernels to operators with different types and shapes of tensors, and also allocates GPU memory to them.

[0007] (4) The scheduler prepares to submit the function parameters required by the kernel to the GPU and schedules the operators to the hardware one by one in the topology order, or schedules multiple operators to different streams in the GPU for execution at the same time. Summary of the Invention

[0008] This invention proposes a multi-model inference acceleration system and method for full-scene perception in autonomous driving. It converts multiple DNN models in full-scene perception of autonomous driving into a computation graph based on a directed acyclic graph (DAG), designs a scheduling framework POS, and uses the deep reinforcement learning (DRL) method to obtain the optimal POS scheduling strategy. Finally, it constructs an operator scheduling algorithm based on maximum entropy deep reinforcement learning to complete the perception of full-scene autonomous driving.

[0009] This invention is achieved through the following technical solution:

[0010] A multi-model inference acceleration method for full-scene perception in autonomous driving:

[0011] The method specifically includes the following steps:

[0012] Step 1: Convert multiple DNN models for full-scenario perception in autonomous driving into computational graph representations based on directed acyclic graphs (DAGs).

[0013] Step 2: Based on the computation graph obtained in Step 1, design a scheduling framework POS to accelerate multi-task DNN inference in full-scene perception of autonomous driving.

[0014] Step 3: Combine the computation graph obtained in Step 1 with the POS scheduling framework designed in Step 2, and use the deep reinforcement learning (DRL) method to obtain the optimal POS scheduling strategy.

[0015] Step 4: Introduce entropy into the deep reinforcement learning (DRL) method in Step 3, and construct an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the best parallel strategy for operators in the computation graph stages;

[0016] Step 5: Update the network parameters from Steps 1 to 4 to complete the acceleration of multi-model inference for full-scene perception in autonomous driving.

[0017] Furthermore, step 1 specifically includes:

[0018] Step 1.1, represent the computation graph as follows: G=(V, E) , where V represents the set of operators, a vertex represents an operator, and E is the set of edges that establish dependencies between operators;

[0019] Step 1.2: Divide the computation graph into multiple stages.

[0020] Represented as G={ };

[0021] Each stage is executed in FIFO order in the computation graph, and operators in the same stage are executed in parallel according to different scheduling strategies.

[0022] Different parts within the same stage are called groups, denoted as stage={ },

[0023] Operators in the same group are executed sequentially, while operators in different groups in the same stage are executed in parallel.

[0024] Step 1.3: Based on the computation graph, the overall latency of minimizing multi-task DNN inference is transformed into fine-grained operator scheduling, as shown in the following formula:

[0025] (1)

[0026] in, Indicates the scheduling search strategy, This represents the i-th stage. It is the runtime delay of the i-th stage. This represents the corresponding parallel scheduling strategy in the i-th stage;

[0027] Step 1.4: Use a cost model to measure the inference latency of the entire computation graph. :

[0028] (2)

[0029] Where f is the cost model, used to directly measure the scheduling strategy on the hardware device. The runtime delay caused by running computation graph G This represents all possible search spaces.

[0030] Furthermore, in step 2,

[0031] The scheduling framework POS utilizes deep reinforcement learning to quickly find the optimal scheduling strategy in a complex search space, and selects appropriate operator parallelization strategies for operators at different stages of the computation graph.

[0032] The operator parallelization strategies in the POS include: inter-operator parallelism, intra-operator parallelism, operator fusion, and subgraph reuse.

[0033] The parallel execution of operators in the computation graph is to change the execution order of operators from sequential to parallel.

[0034] Intra-operator parallelism refers to the parallel execution of arithmetic operations within a single operator.

[0035] The operator fusion is the process of merging multiple identical or different types of operators into a single large operator;

[0036] The subgraph reuse refers to reusing identical subgraphs in the computation graph.

[0037] Furthermore, in step 2,

[0038] The operator fusion specifically involves fusing operators with data dependencies on the computation graph to improve the memory locality of intermediate tensor data; and only child nodes with the same parent node can be fused.

[0039] The subgraph reuse specifically refers to: accelerating model inference by reducing memory access and kernel scheduling overhead; and reusing redundant subgraphs in a computation graph composed of multiple models.

[0040] The parallelism between operators specifically refers to dividing the computation graph into multiple stages, and executing operators in the same stage in parallel.

[0041] The intra-operator parallelism specifically refers to:

[0042] In a deep learning framework, after the scheduler assigns operators to the hardware one by one, the hardware scheduler enables multithreading, dividing individual operators into finer-grained scheduling units and mapping them to multiple threads to fully leverage the parallel advantages of the hardware.

[0043] Furthermore, in step 3,

[0044] The optimization objective in formula (2) is transformed into the reward function in deep reinforcement learning (DRL) and modeled as a Markov decision process (MDP), described by a 4-tuple: ( , , , r ), defined as follows:

[0045] state: It is a state space; at each scheduling time t, the agent in the DRL will construct a state. It consists of the following two parts:

[0046] (1) Search space based on computation graph ;

[0047] (2) System status information collected periodically on the device, including the current available CPU, GPU, memory utilization, and power consumption, respectively denoted as , , and ;

[0048] action: It is the action space, used to select the optimal scheduling strategy for the operators contained in each stage of the computation graph; therefore, the action at scheduling time t can be represented as ;

[0049] in, This indicates the i-th stage of the computation graph at time t. and The scheduling strategy selected by the agent and the corresponding execution delay are respectively;

[0050] award: It is the immediate reward that the agent receives when it performs an action at time t;

[0051] Will Represented as at each scheduling time t The execution latency obtained by the agent after selecting an appropriate parallel strategy for each stage in the computation graph;

[0052] The agent's goal is to maximize the cumulative expected reward. Set the reward function to the following form:

[0053] (3)

[0054] Strategy: Strategy ( ) indicates at time t The intelligent agent adjusts according to the state of the environment. To decide the next action The function of ); the scheduling strategy of the operator. Set as the policy function of the agent in DRL, the optimal policy Defined as the form of formula (4):

[0055] (4)

[0056] in, It is a discount factor. It is a strategy The resulting trajectory distribution.

[0057] Furthermore, in step 4,

[0058] Introduce entropy in DRL to maximize the entropy of visited states while maximizing the reward;

[0059] The entropy is defined as the expected value of the information content. The greater the uncertainty of a random event, the greater the entropy; conversely, if the random event is deterministic, its entropy is zero. The formula for calculating entropy is as follows:

[0060] (5)

[0061] Since the actions of the agent are discrete values, we construct the operator scheduling algorithm MEOS based on maximum entropy reinforcement learning to select the best parallel strategy for the operators in the stages of the computation graph.

[0062] Incorporating maximum entropy into the reward, formula (4) becomes as follows:

[0063] (6)

[0064] in, It is a temperature parameter used to balance reward and entropy; Indicates the state Next strategy Entropy;

[0065] The operator scheduling algorithm MEOS for maximum entropy reinforcement learning is as follows:

[0066] First, a dictionary is constructed to record the execution latency generated at each stage of the scheduling process based on the operator parallelism strategy adopted by the agent, and it is initialized to... ;

[0067] Then initialize all neural networks and their corresponding parameters in DRL;

[0068] To address the overestimation of Q-values ​​and achieve faster and more stable training, two Q-networks are introduced, with the one having the smallest Q-value selected as the target Q-value during each learning process.

[0069] Therefore, there are a total of 5 networks in MEOS, including 1 policy network, 2 Q networks and 2 target Q networks;

[0070] Then initialize an empty experience replay pool to store historical experiences;

[0071] For each round, the state at the current moment The AI ​​learns to select a suitable parallel strategy for the operators in each stage of the computation graph based on the policy and performs latency evaluation, then puts the result of each scheduling into a dictionary. Finally, a parallel strategy with the lowest latency is returned for each stage;

[0072] At the same time, the agent receives an immediate reward and updates its state according to formula (3), and puts the information at the current moment into the experience revisit pool as historical data for the next learning process of the agent.

[0073] Soft policy iteration is used to maximize rewards, including policy evaluation and policy improvement, and the two are performed alternately during training;

[0074] Specifically, at each gradient step, the policy is first computed in the policy evaluation. Soft state value:

[0075] (7)

[0076] Calculate the soft Q function using the modified Bellman backup operator:

[0077] (8)

[0078] Furthermore, in step 5,

[0079] Update the parameters for all networks, including:

[0080] Updating the soft Q-function: The soft Q-function is trained by minimizing the soft Bellman residual; therefore, the loss function of the Q-network is defined as follows:

[0081] (9)

[0082] in, It is obtained by sampling from the experience revisit pool using Monte Carlo estimation and based on the target network of the soft Q function;

[0083] Policy Update: In policy improvement, the policy network is updated to maximize rewards. The loss network of the policy network is defined as follows:

[0084] (10)

[0085] Where D represents KL, For temperature parameters;

[0086] Update temperature parameters: The specific formula for automatically updating temperature parameters is as follows:

[0087] (11)

[0088] in, It is a constant vector, equal to the hyperparameter representing the target entropy.

[0089] Updating the target network: The target network is softly updated to stabilize the training process, and is defined as follows:

[0090] (12)

[0091] A multi-model inference acceleration system for full-scene perception in autonomous driving:

[0092] The system includes a DNN model conversion module, a POS module, a DRL module, an optimal parallel module, and a network parameter update module;

[0093] The DNN model conversion module is used to convert multiple DNN models for full-scene perception in autonomous driving into computational graph representations based on directed acyclic graphs (DAGs).

[0094] The POS module designs a scheduling framework POS based on the computation graph obtained by the DNN model conversion module to accelerate multi-task DNN inference in autonomous driving full-scene perception.

[0095] The DRL module will combine the obtained computation graph with the designed POS scheduling framework and use the deep reinforcement learning DRL method to obtain the optimal POS scheduling strategy.

[0096] The optimal parallel module introduces entropy into the deep reinforcement learning DRL method in the DRL module, and constructs an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the optimal parallel strategy for operators in the stages of the computation graph.

[0097] The network parameter update module is used to update network parameters and accelerate multi-model inference for full-scene perception in autonomous driving.

[0098] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0099] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0100] Beneficial effects of the invention

[0101] Compared to state-of-the-art deep learning inference frameworks and specific scheduling methods, this invention consistently achieves 1.2x to 3.9x inference speedup and 28% to 55% GPU utilization improvement. Furthermore, the search overhead for optimal scheduling is reduced by 1.2x to 2.9x compared to the baseline. Attached Figure Description

[0102] Figure 1 This is an overview of the POS framework of the present invention;

[0103] Figure 2 The different operator parallelization strategies of the present invention are (a) operator fusion, (b) subgraph reuse, (c) inter-operator parallelism, and (d) intra-operator parallelism.

[0104] Figure 3 This is a comparison chart of the inference performance of different deep learning frameworks in the isomorphic multi-task DNN model of the present invention.

[0105] Figure 4 This is a comparison chart of the inference performance of different deep learning frameworks in heterogeneous multi-task DNN models according to the present invention.

[0106] Figure 5 This is a comparison chart of the inference performance of different scheduling strategies of the present invention in a homogeneous multi-task DNN model;

[0107] Figure 6 This is a comparison chart of the inference performance of different scheduling strategies of the present invention in a heterogeneous multi-task DNN model;

[0108] Figure 7 This is a comparison diagram of the search overhead of different scheduling strategies of the present invention;

[0109] Figure 8 To demonstrate the inference performance of the POS in different devices according to the present invention;

[0110] Figure 9 The throughput performance of YOLOP with different batch sizes for POS inference in this invention;

[0111] Figure 10GPU utilization performance of YOLOP with different batch sizes for POS inference in this invention;

[0112] Figure 11 This is the POS prototype of the present invention. Detailed Implementation

[0113] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0114] Combination Figures 1 to 11 .

[0115] A multi-model inference acceleration method for full-scene perception in autonomous driving:

[0116] The method specifically includes the following steps:

[0117] Step 1: Convert (abstract) multiple DNN models for full-scenario perception in autonomous driving into a computational graph representation based on a directed acyclic graph (DAG);

[0118] Step 2: Based on the computation graph obtained in Step 1, a novel scheduling framework POS is designed to accelerate multi-task DNN inference in full-scene perception of autonomous driving.

[0119] Step 3: Combine the computation graph obtained in Step 1 with the POS scheduling framework designed in Step 2, and use the deep reinforcement learning (DRL) method to obtain the optimal POS scheduling strategy.

[0120] Step 4: Introduce entropy into the deep reinforcement learning (DRL) method in Step 3, and construct an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the best parallel strategy for operators in the computation graph stages;

[0121] Step 5: Update the network parameters from Steps 1 to 4 to complete the acceleration of multi-model inference for full-scene perception in autonomous driving.

[0122] Step 1 specifically includes:

[0123] Step 1.1, represent the computation graph as follows: G=(V, E) , where V represents the set of operators, a vertex represents an operator, and E is the set of edges that establish dependencies between operators;

[0124] Step 1.2: Divide the computation graph into multiple stages.

[0125] Represented as G={};

[0126] Each stage is executed in the computation graph in a "first-come, first-served (FIFO)" order, and operators in the same stage can be executed in parallel according to different scheduling strategies;

[0127] Different parts within the same stage are called groups, denoted as stage={ },

[0128] Operators in the same group are executed sequentially, while operators in different groups in the same stage can be executed in parallel.

[0129] Step 1.3: Based on the computation graph, the overall latency of minimizing multi-task DNN inference is transformed into fine-grained operator scheduling, as shown in the following formula:

[0130] (1)

[0131] in, Indicates the scheduling search strategy, This represents the i-th stage. It is the runtime delay of the i-th stage. This represents the corresponding parallel scheduling strategy in the i-th stage;

[0132] Step 1.4: To find the optimal scheduling strategy, this invention utilizes a cost model to measure the inference latency of the entire computation graph. :

[0133] (2)

[0134] Where f is the cost model, used to directly measure the scheduling strategy on the hardware device. The runtime delay caused by running computation graph G This represents all possible search spaces.

[0135] In step 2,

[0136] The scheduling framework POS utilizes deep reinforcement learning to quickly find the optimal scheduling strategy in a complex search space, and selects appropriate operator parallelization strategies for operators at different stages of the computation graph.

[0137] The operator parallelization (scheduling) strategies in the POS include: inter-operator parallelism, intra-operator parallelism, operator fusion, and subgraph reuse;

[0138] The scheduler is the core component of POS. To fully exploit the parallel computing capabilities of the GPU, this invention employs four operator parallelism strategies: inter-operator parallelism, intra-operator parallelism, operator fusion, and subgraph reuse.

[0139] The parallel execution of operators in the computation graph is to change the execution order of operators from sequential to parallel.

[0140] Intra-operator parallelism refers to the parallel execution of arithmetic operations within a single operator.

[0141] The operator fusion is the process of merging multiple identical or different types of operators into a single large operator;

[0142] The subgraph reuse refers to reusing identical subgraphs in the computation graph.

[0143] The operator fusion specifically involves fusing "producer-consumer" operators with data dependencies on the computation graph to improve the locality of memory access for intermediate tensor data; and only child nodes with the same parent node can be fused.

[0144] This method avoids writing data to global memory for subsequent reads, instead performing multiple computations at once to avoid additional memory accesses. For example... Figure 2 As shown in (a), by fusing the three operators (conv, bias, and relu), only one CBR (conv+bias+relu) block is needed to complete the original three operator calculations, thereby reducing two memory access operations.

[0145] The subgraph reuse specifically refers to: accelerating model inference by reducing memory access and kernel scheduling overhead; and reusing redundant subgraphs in a computation graph composed of multiple models.

[0146] like Figure 2 As shown in (b), since there are many redundant subgraphs in the computation graph composed of multiple models, these subgraphs can be reused.

[0147] Specifically, due to the limitations on tensor types and shapes between operators in the computation graph, this invention only reuses subgraphs at the beginning of the computation graph, because the first few layers of multiple model network structures are highly redundant.

[0148] In addition, subgraph reuse can accelerate the search time of scheduling strategies to some extent.

[0149] The parallelism between operators specifically refers to dividing the computation graph into multiple stages, and executing operators in the same stage in parallel.

[0150] Current deep learning frameworks only employ intra-operator parallelism while neglecting the benefits of inter-operator parallelism, resulting in underutilization of GPU resources. Considering that inter-operator and intra-operator parallelism are orthogonal, this invention introduces inter-operator parallelism into the scheduling process of the computation graph.

[0151] like Figure 2 As shown in (c), the left figure illustrates the operator execution mechanism adopted by the current mainstream deep learning frameworks, which completes the inference of the entire model by sequentially executing one operator on multiple stream multiprocessors of the GPU each time.

[0152] The right figure illustrates parallelism between operators. This invention divides the computation graph into multiple stages, allowing operators in the same stage to be executed in parallel, thereby improving resource utilization and reducing latency.

[0153] The intra-operator parallelism specifically refers to:

[0154] like Figure 2 As shown in (d), after the scheduler in the deep learning framework schedules the operators to the hardware one by one, the hardware scheduler enables multi-threading, divides the individual operators into finer-grained scheduling units and maps them to multi-threading to give full play to the parallel advantages of the hardware.

[0155] This invention utilizes the NVIDIA CUDA application programming interface (API) to obtain the maximum number of threads allowed by the device. Specifically, GPU threads are organized in a grid, with each grid containing several thread blocks, and each thread block can contain up to several thousand threads. Threads are organized at two levels: coarse-grained thread blocks and fine-grained threads. Communication at the fine-grained parallelism level is achieved through shared memory and fence synchronization.

[0156] During execution, the GPU's task allocation unit assigns the mesh to the GPU cores. When the CUDA kernel starts, mesh information needs to be transferred from the CPU to the GPU. The task allocation unit then uses this information to allocate blocks to the streaming multiprocessors.

[0157] In step 3,

[0158] Although traditional heuristics can be used for scheduling strategies, their computational complexity and convergence speed are severely affected when the problem size becomes large, making it difficult to achieve satisfactory results.

[0159] Conversely, Deep Reinforcement Learning (DRL) combines the efficient decision-making capabilities of reinforcement learning with the powerful representational abilities of deep learning, achieving revolutionary breakthroughs in fields such as game theory, Go, robot control, and protein prediction. Considering the complexity and high-dimensional information of the computational graph generated by abstracting multiple models, it is natural to view operator scheduling as a sequential decision and utilize Deep Reinforcement Learning (DRL) to quickly find the optimal scheduling strategy.

[0160] The optimization objective in formula (2) is transformed into the reward function in deep reinforcement learning (DRL) and modeled as a Markov decision process (MDP), described by a 4-tuple: ( , , , r ), defined as follows:

[0161] state: It is a state space; at each scheduling time t, the agent in the DRL will construct a state. It consists of the following two parts:

[0162] (1) Search space based on computation graph ;

[0163] (2) System status information collected periodically on the device, including the current available CPU, GPU, memory utilization, and power consumption, respectively denoted as , , and ;

[0164] action: It is the action space, used to select the optimal scheduling strategy for the operators contained in each stage of the computation graph; therefore, the action at scheduling time t can be represented as ;

[0165] in, This indicates the i-th stage of the computation graph at time t. and The scheduling strategy selected by the agent and the corresponding execution delay are respectively;

[0166] award: It is the immediate reward that the agent receives when it performs an action at time t;

[0167] In order to better integrate operator scheduling with DRL, this invention will... Represented as at each scheduling time t The execution latency obtained by the agent after selecting an appropriate parallel strategy for each stage in the computation graph;

[0168] However, the objective of this invention is to minimize the overall inference latency of multiple DNN models, while the agent's goal is to maximize the accumulated expected reward. Therefore, the present invention sets the reward function in the following form:

[0169] (3)

[0170] Strategy: Strategy ( ) indicates at time t The intelligent agent adjusts according to the state of the environment. To decide the next action The function of ); the scheduling strategy of the operator. Set as the policy function of the agent in DRL, the optimal policy Defined as the form of formula (4):

[0171] (4)

[0172] in, It is a discount factor. It is a strategy The resulting trajectory distribution.

[0173] In step 4,

[0174] Traditional DRL methods have the following drawbacks when applied to full-scene perception tasks in autonomous driving:

[0175] (1) Low sampling efficiency: This is mainly due to the shortcomings of the same-policy methods such as TRPO and PPO. Since each policy update requires resampling a sufficient number of samples under the current policy and completely discarding the previous sampled data, a large number of samples and sample diversity are required to ensure the convergence of the algorithm.

[0176] (2) Highly sensitive to hyperparameters: This is mainly manifested in the shortcomings of heterogeneous policy methods such as DDPG and D4PG. Although the challenge of low sample utilization efficiency is solved by using the experience replay pool, the policy and Q-value (used for policy evaluation) are coupled with each other, resulting in unstable algorithm performance and easy to be affected by hyperparameters.

[0177] To this end, the present invention introduces entropy into DRL, which maximizes the entropy of visited states while maximizing the reward;

[0178] Entropy is defined as the expected value of information content, and is a measure of the uncertainty of a random variable. Clearly, the greater the uncertainty of a random event, the greater the entropy; conversely, if a random event is deterministic, its entropy is zero. The formula for calculating entropy is as follows:

[0179] (5)

[0180] Introducing entropy into DRL brings many benefits:

[0181] (1) Accelerated convergence. Maximum entropy enables the agent to use the learned policy as the initialization for more complex and specific tasks. Because through maximum entropy, the policy learns not only one way to solve the task, but all of them, thus enabling the agent to learn more near-optimal actions, thereby accelerating the learning speed.

[0182] (2) Encourage exploration. This is obvious, because maximizing the entropy of visited states while maximizing the reward makes the distribution of actions more even, makes it easier to find better strategies, and avoids getting trapped in local optima.

[0183] (3) Stronger robustness. The agent explores various optimal possibilities with different strategies, so it can make adjustments more easily when faced with disturbances.

[0184] Because the actions of the agent are discrete values, this invention constructs an operator scheduling algorithm (MEOS) based on maximum entropy reinforcement learning on the basis of the discrete soft actor-critic framework, which selects the best parallel strategy for operators in the stages of the computation graph.

[0185] Incorporating maximum entropy into the reward, formula (4) becomes as follows:

[0186] (6)

[0187] in, It is a temperature parameter used to balance reward and entropy; Indicates the state Next strategy The entropy.

[0188] The operator scheduling algorithm MEOS for maximum entropy reinforcement learning is as follows:

[0189] First, a dictionary is constructed to record the execution latency generated at each stage of the scheduling process based on the operator parallelism strategy adopted by the agent, and it is initialized to... ;

[0190] Then initialize all neural networks and their corresponding parameters in DRL;

[0191] It should be noted that, in order to solve the problem of Q-value overestimation and achieve faster and more stable training, this invention introduces two Q-networks, and selects the one with the smallest Q-value as the target Q-value in each learning process;

[0192] Therefore, there are a total of 5 networks in MEOS, including 1 policy network as actors, 2 Q networks and 2 target Q networks as critics;

[0193] Then initialize an empty experience replay pool to store historical experiences;

[0194] For each round, the state at the current moment The AI ​​learns to select a suitable parallel strategy for the operators in each stage of the computation graph based on the policy and performs latency evaluation, then puts the result of each scheduling into a dictionary. Finally, a parallel strategy with the lowest latency is returned for each stage;

[0195] At the same time, the agent receives an immediate reward and updates its state according to formula (3), and puts the information at the current moment into the experience revisit pool as historical data for the next learning process of the agent.

[0196] Soft policy iteration is used to maximize rewards, including policy evaluation and policy improvement, and the two are performed alternately during training;

[0197] Specifically, at each gradient step, the present invention first calculates the policy in the policy evaluation. Soft state value:

[0198] (7)

[0199] Calculate the soft Q function using the modified Bellman backup operator:

[0200] (8)

[0201] In step 5,

[0202] Update the parameters for all networks, including:

[0203] Updating the soft Q-function (commentator network): The soft Q-function is trained by minimizing the soft Bellman residual; therefore, the loss function of the Q-network (commentator network) is defined as follows:

[0204] (9)

[0205] in, It is obtained by sampling from the experience revisit pool using Monte Carlo estimation and based on the target network of the soft Q function;

[0206] Update the policy (actor network): In policy improvement, update the policy network to maximize rewards. The loss network of the policy (actor) network is defined as follows:

[0207] (10)

[0208] Where D represents KL, For temperature parameters;

[0209] Update temperature parameters: The specific formula for automatically updating temperature parameters in this invention is as follows:

[0210] (11)

[0211] in, It is a constant vector, equal to the hyperparameter representing the target entropy.

[0212] Updating the target network: The target network is softly updated to stabilize the training process, and is defined as follows:

[0213] (12)

[0214] The operator scheduling algorithm MEOS based on maximum entropy deep reinforcement learning is shown in Algorithm 1:

[0215]

[0216] A multi-model inference acceleration system for full-scenario perception in autonomous driving.

[0217] The system includes a DNN model conversion module, a POS module, a DRL module, an optimal parallel module, and a network parameter update module;

[0218] The DNN model conversion module is used to convert (abstract) multiple DNN models for full-scene perception in autonomous driving into a computational graph representation based on a directed acyclic graph (DAG);

[0219] The POS module, based on the computation graph obtained by the DNN model conversion module, designs a novel scheduling framework POS to accelerate multi-task DNN inference in autonomous driving full-scene perception.

[0220] The DRL module will combine the obtained computation graph with the designed POS scheduling framework and use the deep reinforcement learning DRL method to obtain the optimal POS scheduling strategy.

[0221] The optimal parallel module introduces entropy into the deep reinforcement learning DRL method in the DRL module, and constructs an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the optimal parallel strategy for operators in the stages of the computation graph.

[0222] The network parameter update module is used to update network parameters and accelerate multi-model inference for full-scene perception in autonomous driving.

[0223] Experimental setup:

[0224] 1. Multi-task DNN model

[0225] This invention selects nine mainstream DNN models from three series: ResNet, Inception, and MobileNet, as the model library. As shown in Table 2, this invention constructs a multi-task DNN inference model by combining these models in both homogeneous and heterogeneous ways. It should be noted that these models have different numbers and heterogeneous operators, and therefore have different requirements for computational and memory resources.

[0226]

[0227] Table 2 Multi-task DNN Models

[0228] 2. Baseline method:

[0229] This invention compares POS with the current state-of-the-art deep learning inference frameworks:

[0230] TensorFlow: As a static graph deep learning framework, TensorFlow first constructs a tensor-based static graph when performing DNN model inference, and then schedules the operators in the computation graph one by one in a sequential manner.

[0231] PyTorch: Unlike TensorFlow, PyTorch is a deep learning framework with a dynamic graph mechanism, meaning that DNN models do not need to generate a computation graph at compile time before performing inference. However, the scheduler in PyTorch also schedules operators in a sequential manner.

[0232] TensorRT: As a mainstream deep learning inference engine, TensorRT accelerates inference through methods such as weight pruning, operator fusion, and precision quantization, and schedules the computation graph, which is automatically fine-tuned by the kernel, to be executed in parallel by the GPU's streaming multiprocessor.

[0233] TVM: An operator-level inference framework for model optimization. TVM abstracts DNN models into a unified intermediate representation based on a computation graph, and then uses a machine learning-based scheduler to generate optimal scheduling policies.

[0234] OS: As an improvement on TVM, IOS uses two methods to accelerate model inference: a dynamic programming-based scheduler to preferentially select operator fusion and inter-operator parallelism.

[0235] In addition, this invention also compares POS with three different scheduling strategies—sequential scheduling, dynamic programming (DP), and greedy search—within the same reasoning framework.

[0236] 3. Comparison with current state-of-the-art deep learning inference frameworks:

[0237] This invention first compares the inference performance of the baseline framework with that of POS. The performance of all methods is normalized by the optimal inference latency to show the relative speedup.

[0238] The results are as follows Figure 3 As shown, POS consistently outperforms all benchmark frameworks in terms of inference latency for the three isomorphic multi-task DNN models.

[0239] Specifically, compared to the two mainstream deep learning frameworks, TensorFlow and PyTorch, POS offers a significant performance improvement, with a 2.6x to 3.9x performance increase. Even compared to the most advanced libraries, TVM, TensorRT, and IOS, POS can achieve a 1.2x to 1.5x speedup.

[0240] Figure 4 The performance comparison of all methods in heterogeneous multi-task DNN inference scenarios is shown. Compared to homogeneous models, the performance improvement of POS in heterogeneous models is reduced, but it still has a relative speedup of 1.2x to 2.8x.

[0241] 4. Comparison with different scheduling strategies

[0242] This invention compares the inference performance of different schedulers with MEOS.

[0243] The three baseline methods include sequential scheduling, greedy scheduling, and dynamic programming scheduling. For a fair comparison, this invention performs all scheduling on the iOS inference engine.

[0244] Figure 5 The inference performance of different scheduling methods in a homogeneous multi-task DNN model is shown. It can be seen that POS consistently outperforms all other three scheduling methods, with a relative speedup of 1.2 to 2.8 times.

[0245] Figure 6 This is a comparison of inference performance of different scheduling methods on heterogeneous multi-task DNN models. POS also shows a performance improvement of 1.2 to 2.9 times.

[0246] 5. Performance evaluation for real-world autonomous driving scenarios

[0247] This invention selects YOLOP to evaluate the performance of POS in multi-task DNN inference in real-world autonomous driving full-scenario perception.

[0248] As a composite inference model for full-scene perception in autonomous driving, YOLOP comprises three distinct subtasks: vehicle detection, lane segmentation, and drivable area segmentation, making it a heterogeneous multi-task DNN model. Similarly, POS and all baseline scheduling strategies run on an iOS-based inference engine.

[0249] Table 3 shows the inference performance of different frameworks and scheduling strategies, and it can be seen that POS has a clear advantage.

[0250]

[0251] Table 3. Latency of YOLOP inference under different frameworks and scheduling algorithms on NVIDIA Xavier NX GPUs

[0252] 6. Search Cost Analysis

[0253] To evaluate the scheduling and search performance of POS, considering that sequential scheduling has no search overhead, MEOS was compared with two other methods: greedy strategy and dynamic programming. All algorithms were run on the IOS inference engine.

[0254] like Figure 7 As shown, this invention considers both homogeneous and heterogeneous multi-task DNN models, demonstrating that POS can generate near-optimal scheduling strategies with shorter search times. Specifically, POS reduces the average scheduling search overhead by 1.2 to 2.9 times compared to the baseline.

[0255] 7. Micro-benchmarking

[0256] (1) Different equipment

[0257] Figure 8 The performance of POS on devices with different GPUs in inferring three multi-task DNN models is shown. Compared with the baseline inference framework, POS has a significant overall performance gain (1.35x to 2.84x) on different GPU accelerators.

[0258] More importantly, POS also has a significant advantage in real-time performance. Even on the Jetson Nano with the lowest computing power, it can infer the complex hybrid model YOLOP at a speed of 15.6 KJ.

[0259] The results show that POS can find a better scheduling strategy than the baseline framework. Furthermore, it reveals the scalability advantage of POS, enabling it to accelerate inference for multiple models simultaneously on GPU accelerators with different architectures. Leveraging the learning-based search algorithm MEOS, POS can automatically find the optimal scheduling strategy for different multi-task models and different GPU accelerators, significantly reducing the need for manual tuning.

[0260] (2) Different batch sizes

[0261] To make full use of the accelerator's computing resources, batch sizes are typically increased to infer a batch of input data simultaneously.

[0262] However, increasing the batch size reduces latency to some extent, which is not applicable for real-time inference on resource-constrained edge devices.

[0263] Conversely, when there are abundant computing resources on the device, the batch size can be increased to improve throughput. In fact, this is more common in cloud computing servers.

[0264] Furthermore, the schedules generated are different for different batch sizes.

[0265] Therefore, this invention evaluates the throughput of all frames at different batch sizes.

[0266] Specifically, Figure 9 The results show that as batch size increases, POS consistently outperforms the baseline method across all batch sizes. This demonstrates that POS can generate optimal schedules for different batch sizes.

[0267] 8. GPU utilization

[0268] Figure 10 The data shows the GPU utilization on edge devices during the inference process of heterogeneous multi-DNN models, along with POS and three other baselines.

[0269] This invention uses the open-source resource monitoring tool jtop to collect data in real time at a sampling frequency of 1ms.

[0270] As can be seen, MEOS's GPU utilization is 28%-55% higher than the baseline. This is because MEOS employs a more advanced scheduling strategy, fully leveraging the computational advantages of the GPU by utilizing different operator parallelism methods to increase the number of active threads on the GPU. This also explains the overall performance improvement in POS.

[0271] Figure 11 This is an experimental prototype of POS. Specifically, the autonomous vehicle is equipped with a depth camera and edge computing devices.

[0272] Among them, the depth camera is used to capture video in real time during the driving process; the edge computing device processes the input video in real time.

[0273] The NVIDIA Xavier NX is an embedded edge computing platform specifically designed to provide DNN inference services, running on an Ubuntu 18.04 operating system with cuDNN 7.6.5 and the CUDA 10.2 deep learning acceleration library.

[0274] This embodiment selects the Intel D435 depth camera, which can capture 1080p resolution video at a frame rate of 30FPS. This embodiment deploys the POS onto an NVIDIA Xavier NX edge device for multi-task DNN inference.

[0275] Considering the limited computing power of edge devices, this embodiment downsamples the video frames to a resolution format of 224×224×3, and sets the batch size to 1 during inference.

[0276] In addition, the number of iterations for all methods was set to 1000, and the average inference delay of 5 experiments was taken.

[0277] Specifically, POS inference performance is 3.16 times that of TensorFlow and 3.34 times that of PyTorch, respectively. Even when compared to the three most advanced deep learning inference frameworks—TensorRT, TVM, and IOS—POS still offers a performance improvement of 1.24 to 1.38 times.

[0278] Furthermore, when comparing different scheduling methods, MEOS in POS also shows a relative speedup of 1.21x to 2.13x.

[0279] Table 3 shows the latency of YOLOP inference under different frameworks and scheduling algorithms on NVIDIA Xavier NX GPUs.

[0280] This invention implements Algorithm 1 based on the PyTorch deep learning framework. Specifically, all neural networks in MEOS are trained using the Adam optimizer and employ a three-layer ReLU activation function, with 128, 64, and 32 hidden units in each layer, respectively.

[0281] In addition, regarding the settings of hyperparameters, this embodiment uses a fixed learning rate of 1e-4, a cache size of 1e6, and a batch size of 512.

[0282] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0283] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0284] The memory in this application embodiment can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0285] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0286] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0287] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0288] The foregoing has provided a detailed description of the multi-model inference acceleration system and method for full-scene perception in autonomous driving proposed by the present invention, and has elucidated the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-model inference acceleration method for full-scene perception in autonomous driving, characterized in that: The method specifically includes the following steps: Step 1: Convert multiple DNN models for full-scenario perception in autonomous driving into computational graph representations based on directed acyclic graphs (DAGs). Step 2: Based on the computation graph obtained in Step 1, design a scheduling framework POS to accelerate multi-task DNN inference in full-scene perception of autonomous driving. Step 3: Combine the computation graph obtained in Step 1 with the POS scheduling framework designed in Step 2, and use the deep reinforcement learning (DRL) method to obtain the optimal POS scheduling strategy. Step 4: Introduce entropy into the deep reinforcement learning (DRL) method in Step 3, and construct an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the best parallel strategy for operators in the computation graph stages; Step 5: Update the network parameters from Steps 1 to 4 to complete the acceleration of multi-model inference for full-scene perception in autonomous driving.

2. The method according to claim 1, characterized in that: Step 1 specifically includes: Step 1.1, represent the computation graph as follows: G=(V, E) , where V represents the set of operators, a vertex represents an operator, and E is the set of edges that establish dependencies between operators; Step 1.2: Divide the computation graph into multiple stages. Represented as G={ }; Each stage is executed in FIFO order in the computation graph, and operators in the same stage are executed in parallel according to different scheduling strategies. Different parts within the same stage are called groups, denoted as stage={ }, Operators in the same group are executed sequentially, while operators in different groups in the same stage are executed in parallel. Step 1.3: Based on the computation graph, the overall latency of minimizing multi-task DNN inference is transformed into fine-grained operator scheduling, as shown in the following formula: （1） in, Indicates the scheduling search strategy, This represents the i-th stage. It is the runtime delay of the i-th stage. This represents the corresponding parallel scheduling strategy in the i-th stage; Step 1.4: Use a cost model to measure the inference latency of the entire computation graph. : （2） Where f is the cost model, used to directly measure the scheduling strategy on the hardware device. The runtime delay caused by running computation graph G This represents all possible search spaces.

3. The method according to claim 2, characterized in that: In step 2, The scheduling framework POS utilizes deep reinforcement learning to quickly find the optimal scheduling strategy in a complex search space, and selects appropriate operator parallelization strategies for operators at different stages of the computation graph. The operator parallelization strategies in the POS include: inter-operator parallelism, intra-operator parallelism, operator fusion, and subgraph reuse. The parallel execution of operators in the computation graph is to change the execution order of operators from sequential to parallel. Intra-operator parallelism refers to the parallel execution of arithmetic operations within a single operator. The operator fusion is the process of merging multiple identical or different types of operators into a single large operator; The subgraph reuse refers to reusing identical subgraphs in the computation graph.

4. The method according to claim 3, characterized in that: In step 2, The operator fusion specifically involves fusing operators with data dependencies on the computation graph to improve the memory locality of intermediate tensor data; and only child nodes with the same parent node can be fused. The subgraph reuse specifically refers to: accelerating model inference by reducing memory access and kernel scheduling overhead; and reusing redundant subgraphs in a computation graph composed of multiple models. The parallelism between operators specifically refers to dividing the computation graph into multiple stages, and executing operators in the same stage in parallel. The intra-operator parallelism specifically refers to: In a deep learning framework, after the scheduler assigns operators to the hardware one by one, the hardware scheduler enables multithreading, dividing individual operators into finer-grained scheduling units and mapping them to multiple threads to fully leverage the parallel advantages of the hardware.

5. The method according to claim 4, characterized in that: In step 3, The optimization objective in formula (2) is transformed into the reward function in deep reinforcement learning (DRL) and modeled as a Markov decision process (MDP), described by a 4-tuple: ( , , , r ), defined as follows: state: It is a state space; at each scheduling time t, the agent in the DRL will construct a state. It consists of the following two parts composition: (1) Search space based on computation graph ; (2) System status information collected periodically on the device, including the current available CPU, GPU, memory utilization, and power consumption, respectively denoted as , , and ; action: It is the action space, used to select the optimal scheduling strategy for the operators contained in each stage of the computation graph; therefore, the action at scheduling time t is represented as... ; in, This indicates the i-th stage of the computation graph at time t. and The scheduling strategy selected by the agent and the corresponding execution delay are respectively; award: It is the immediate reward that the agent receives when it performs an action at time t; Will Represented as at each scheduling time t The execution latency obtained by the agent after selecting an appropriate parallel strategy for each stage in the computation graph; The agent's goal is to maximize the cumulative expected reward. Set the reward function to the following form: (3) Strategy: Strategy ( ) indicates at time t The intelligent agent adjusts according to the state of the environment. To decide the next action The function of ); the scheduling strategy of the operator. Set as the policy function of the agent in DRL, the optimal policy Defined as the form of formula (4): (4) in, It is a discount factor. It is a strategy The resulting trajectory distribution.

6. The method according to claim 5, characterized in that: In step 4, Introduce entropy in DRL to maximize the entropy of visited states while maximizing the reward; The entropy is defined as the expected value of the information content. The greater the uncertainty of a random event, the greater the entropy; conversely, if the random event is deterministic, its entropy is zero. The formula for calculating entropy is as follows: (5) Since the actions of the agent are discrete values, we construct the operator scheduling algorithm MEOS based on maximum entropy reinforcement learning to select the best parallel strategy for the operators in the stages of the computation graph. Incorporating maximum entropy into the reward, formula (4) becomes as follows: （6） in, It is a temperature parameter used to balance reward and entropy; Indicates the state Next strategy Entropy; The operator scheduling algorithm MEOS for maximum entropy reinforcement learning is as follows: First, a dictionary is constructed to record the execution latency generated at each stage of the scheduling process based on the operator parallelism strategy adopted by the agent, and it is initialized to... ; Then initialize all neural networks and their corresponding parameters in DRL; To address the overestimation of Q-values ​​and achieve faster and more stable training, two Q-networks are introduced, with the one having the smallest Q-value selected as the target Q-value during each learning process. Therefore, there are a total of 5 networks in MEOS, including 1 policy network, 2 Q networks and 2 target Q networks; Then initialize an empty experience replay pool to store historical experiences; For each round, the state at the current moment The AI ​​learns to select a suitable parallel strategy for the operators in each stage of the computation graph based on the policy and performs latency evaluation, then puts the result of each scheduling into a dictionary. Finally, a parallel strategy with the lowest latency is returned for each stage; At the same time, the agent receives an immediate reward and updates its state according to formula (3), and puts the information at the current moment into the experience revisit pool as historical data for the next learning process of the agent. Soft policy iteration is used to maximize rewards, including policy evaluation and policy improvement, and the two are performed alternately during training; Specifically, at each gradient step, the policy is first computed in the policy evaluation. Soft state value: （7） Calculate the soft Q function using the modified Bellman backup operator: (8)。 7. The method according to claim 6, characterized in that: In step 5, Update the parameters for all networks, including: Updating the soft Q-function: The soft Q-function is trained by minimizing the soft Bellman residual; therefore, the loss function of the Q-network is defined as follows: （9） in, It is obtained by sampling from the experience revisit pool using Monte Carlo estimation and based on the target network using the soft Q function; Policy Update: In policy improvement, the policy network is updated to maximize rewards. The loss network of the policy network is defined as follows: （10） Where D represents KL, For temperature parameters; Update temperature parameters: The specific formula for automatically updating temperature parameters is as follows: （11） in, It is a constant vector, equal to the hyperparameter representing the target entropy; Updating the target network: The target network is softly updated to stabilize the training process, and is defined as follows: (12)。 8. A multi-model inference acceleration system for full-scene perception in autonomous driving, characterized in that: The system includes a DNN model conversion module, a POS module, a DRL module, an optimal parallel module, and a network parameter update module; The DNN model conversion module is used to convert multiple DNN models for full-scene perception in autonomous driving into computational graph representations based on directed acyclic graphs (DAGs). The POS module designs a scheduling framework POS based on the computation graph obtained by the DNN model conversion module to accelerate multi-task DNN inference in autonomous driving full-scene perception. The DRL module will combine the obtained computation graph with the designed POS scheduling framework and use the deep reinforcement learning DRL method to obtain the optimal POS scheduling strategy. The optimal parallel module introduces entropy into the deep reinforcement learning DRL method in the DRL module, and constructs an operator scheduling algorithm based on maximum entropy deep reinforcement learning to select the optimal parallel strategy for operators in the stages of the computation graph. The network parameter update module is used to update network parameters and accelerate multi-model inference for full-scene perception in autonomous driving.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images