A training method and system for a deep neural network in a mixed memory environment

A deep neural network and neural network technology, applied in the field of deep neural network training in a mixed memory environment, can solve problems such as complex communication mechanisms and training performance degradation, and achieve the effects of improving training speed, reducing data volume, and reducing impact.

Active Publication Date: 2022-02-15
HUAZHONG UNIV OF SCI & TECH
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This method distributes the data reading overhead during training to multiple nodes, which can improve the speed of training, but it also introduces a complex communication mechanism, which brings a non-negligible communication overhead
Each iteration of the training process requires multiple GPUs to communicate with each other, resulting in a decrease in overall training performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A training method and system for a deep neural network in a mixed memory environment
  • A training method and system for a deep neural network in a mixed memory environment
  • A training method and system for a deep neural network in a mixed memory environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0042] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

[0043] First, the meanings of the variables involved in the present invention are explained as follows:

[0044]

[0045] NVRAM (Non-Volatile Random Access sMemory) is a new type of non-volatile random access memory, which can still maintain data after power failure. The reading speed is not much different from that of volatile memory DRAM, but the writing speed is lower. The service life is...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a training method and system for a deep neural network in a mixed memory environment, belonging to the technical field of deep learning. The invention caches a large amount of training data in the NVRAM based on the read characteristic of the NVRAM, and improves the speed of the neural network to obtain data. In the present invention, the training data originally calculated by the GPU is divided into two parts, which are respectively calculated in parallel by the CPU and the GPU. Using the computing power of the GPU and the CPU, two neural networks are used for training, and the copying is reduced while using the computing power of the CPU. The amount of data to the GPU memory can improve the training speed of the neural network by increasing the parallelism of the calculation. The network parameter snapshot after weighted average of the present invention is saved in NVRAM, adopts the mode of asynchronous backup, in the process of data writing into NVRAM, does not affect the speed of neural network training data, reduces the impact of NVRAM writing speed on training.

Description

technical field [0001] The invention belongs to the technical field of deep learning, and more specifically relates to a training method and system for a deep neural network in a mixed memory environment. Background technique [0002] At present, artificial intelligence (AI) has made comprehensive breakthroughs in technologies in many fields such as image, voice, and natural language processing. The breakthroughs in AI technology in recent years mainly come from Deep Learning. Deep learning technology has made great progress in various artificial intelligence applications by building a complex deep neural network (Deep Neural Network, DNN) and massive training data samples, especially in the field of image and sound compared with traditional algorithms. Improved recognition rate. In the field of big data analysis and mining, deep neural networks have been widely used. [0003] The classic neural network model is mainly expanded in different degrees in terms of "width" and...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06N3/04G06N3/063G06N3/08
CPCG06N3/084G06N3/063G06N3/045
Inventor 蒋文斌金海刘湃彭晶马阳刘博
Owner HUAZHONG UNIV OF SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap