Reducing training times of deep neural networks through efficient hybrid parallelism

A deep neural network, training time technology, applied in biological neural network models, probabilistic networks, neural architectures, etc., can solve problems such as the inability to popularize DNN well

Pending Publication Date: 2021-05-25
BAIDU USA LLC
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The strategies designed by these experts do not generalize well to other DNNs except for the DNN for which the strategy is designed.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Reducing training times of deep neural networks through efficient hybrid parallelism
  • Reducing training times of deep neural networks through efficient hybrid parallelism
  • Reducing training times of deep neural networks through efficient hybrid parallelism

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] In the following description, for purposes of explanation, specific details are set forth in order to provide an understanding of the present invention. It will be apparent, however, to one skilled in the art that the present disclosure may be practiced without these details. Furthermore, those skilled in the art will appreciate that the embodiments of the present disclosure described below can be implemented in various ways, such as a process, apparatus, system, device or method on a tangible computer readable medium.

[0023] Components or modules shown in the drawings are examples of exemplary embodiments of the present disclosure and are intended to avoid obscuring the disclosure. It should also be understood that throughout this discussion, components may be described as separate functional units, which may include subunits, but those skilled in the art will recognize that various components, or portions thereof, may be divided into separate components, or may be ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Presented are systems and methods to automatically find efficient parallelization strategies for deep neural networks (DNNs). A computation graph comprising an efficiently ordered sequence of vertices aids in computing the best parallelizing strategy in a relatively short time. Effectiveness of the parallelization strategies is evaluated on various DNNs, and the performance of the strategies proposed by various embodiments is compared against data parallelism, expert-designed strategies, and other state-of-the-art approaches. Experimental results demonstrate that the proposed strategies outperform a baseline data parallelism strategy and achieve better performance than expert-designed strategies and state-of-the-art approaches.

Description

[0001] Cross References to Related Applications [0002] This patent application relates to and claims co-pending and commonly owned U.S. Serial No. 62 / 930,518, filed November 4, 2019, entitled "REDUCING TRAINING TIMES OF DEEP NEURAL NETWORKS THROUGH EFFICIENT HYBRIDPARALLELISM," pursuant to 35 USC § 119(e). Priority interest in a patent application listing Venmugil Elango as inventor (Docket No. 28888-2363P), the entire contents of which are hereby incorporated by reference for all purposes. Background technique [0003] The present disclosure relates generally to systems and methods for computer learning that may provide improved computer performance, features and use. More specifically, the present disclosure relates to systems and methods for reducing the training time of deep neural networks (DNNs) by efficiently mixing parallel techniques. [0004] DNN has achieved great success in many fields such as computer vision, natural language processing, recommendation system a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/04G06N3/063G06N3/08
CPCG06N3/063G06N3/08G06N3/045G06F8/00G06N3/084G06N7/01G06N3/044G06N3/10G06F9/3885G06N3/04
Inventor 文穆吉尔·伊兰戈
Owner BAIDU USA LLC
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap