Using a runtime engine to facilitate dynamic adaptation of deep neural networks for efficient processing

Abstract

The disclosed embodiments relate to a system that facilitates dynamic runtime execution of a deep neural network (DNN). During operation, the system receives a model, a set of weights and runtime metadata for the DNN. The system also obtains code to perform inference-processing operations for the DNN. Next, the system compiles code to implement a runtime engine that facilitates throttling operations during execution of the inference-processing operations, wherein the runtime engine conserves computing resources by selecting portions of the inference-processing operations to execute based on the runtime metadata.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Application Ser. No. 62 / 900,311, entitled “Dynamic Adaptation of Deep Neural Networks for Efficient Processing,” by inventors Sek Meng Chai and Jagadeesh Kandasamy, filed on 13 Sep. 2019, which is hereby incorporated by reference. This application also claims priority under 35 U.S.C. § 119(e) to U.S. Provisional application Ser. No. 63 / 018,236, entitled “Dynamic Adaptation of Deep Neural Networks for Efficient Processing,” by inventors Sek Meng Chai and Jagadeesh Kandasamy, filed on 30 Apr. 2020, which is hereby incorporated by reference. This application is also related to pending U.S. patent application Ser. No. 16 / ______, entitled “Optimizing Execution of a Neural Network based on Operational Performance Parameters,” by inventors Sek Meng Chai and Jagadeesh Kandasamy, filed on the same day as the instant application (Attorney Docket No. LATI20-1003), which is...

AI Technical Summary

Benefits of technology

The patent describes a system that uses operational performance parameters to improve the execution of a DNN based on the accuracy of results, resource utilization, and operating conditions. The system then updates the DNN model to optimize performance and efficiency during execution. The reward function balances the objectives of maximizing classification accuracy, minimizing computational operations, power consumption, and latency. The technical effect is an improved performance and efficiency of DNN during execution.

Problems solved by technology

Deep neural networks tend to be computationally intensive because computational operations need to be performed to generate successive outputs for a large number of layers.

However, it is more of a problem to use such deep neural networks in resource-constrained environments, such as in edge devices, autonomous vehicles or portable devices, which only provide limited amounts of processing power, memory capacity and battery life.

Images