Reducing training times of deep neural networks through efficient hybrid parallelism

Abstract

Presented are systems and methods to automatically find efficient parallelization strategies for deep neural networks (DNNs). A computation graph comprising an efficiently ordered sequence of vertices aids in computing the best parallelizing strategy in a relatively short time. Effectiveness of the parallelization strategies is evaluated on various DNNs, and the performance of the strategies proposed by various embodiments is compared against data parallelism, expert-designed strategies, and other state-of-the-art approaches. Experimental results demonstrate that the proposed strategies outperform a baseline data parallelism strategy and achieve better performance than expert-designed strategies and state-of-the-art approaches.

Description

[0001] Cross References to Related Applications [0002] This patent application relates to and claims co-pending and commonly owned U.S. Serial No. 62 / 930,518, filed November 4, 2019, entitled "REDUCING TRAINING TIMES OF DEEP NEURAL NETWORKS THROUGH EFFICIENT HYBRIDPARALLELISM," pursuant to 35 USC § 119(e). Priority interest in a patent application listing Venmugil Elango as inventor (Docket No. 28888-2363P), the entire contents of which are hereby incorporated by reference for all purposes. Background technique [0003] The present disclosure relates generally to systems and methods for computer learning that may provide improved computer performance, features and use. More specifically, the present disclosure relates to systems and methods for reducing the training time of deep neural networks (DNNs) by efficiently mixing parallel techniques. [0004] DNN has achieved great success in many fields such as computer vision, natural language processing, recommendation system a...

AI Technical Summary

Problems solved by technology

The strategies designed by these experts do not generalize well to other DNNs except for the DNN for which the strategy is designed.

Images