An asynchronous parallel optimization method for neural network training
A neural network training and optimization method technology, applied in the field of asynchronous parallel optimization, can solve the problems of resource waste, slow update, waiting for weak computing power, etc., and achieve the effect of improving the training speed and accelerating the training process
Inactive Publication Date: 2019-03-22
TSINGHUA UNIV
View PDF8 Cites 3 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
When the computing power of different computers is different, this method will make the computing power strong, and the fast-updating computer has to wait for the weak computing power and slow-updating computer, resulting in a waste of resources.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0035] The present invention proposes an asynchronous parallel optimization method for neural network training, which will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0036] The invention proposes an asynchronous parallel optimization method for neural network training, taking n computers to train a neural network as an example. Order (x i ,y i ) represents a set of input and output data pairs, where x i is the corresponding input feature vector, y i for x i The corresponding real output value. Let D={(x i ,y i ), i=1,..,N} represents a data set with N sets of data pairs. A neural network can be viewed as an input of x i function, the output is the real output y i estimated value of which is Where w is the parameter to be adjusted in the neural network. Functions corresponding to networks of different structures f(x i ,w) are also different. The purpose of training the neural network is to make ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides an asynchronous parallel optimization method for neural network training, belonging to the field of deep learning. Firstly, the data pairs in the dataset used to train the neural network are distributed to n computers. Determining a communication topology structure of n computers, and obtaining a computer set corresponding to the sending data and the receiving data of each computer; Each computer initializes its own neural network and related parameters, and then iteratively trains the neural network. After each iteration, the updated weighted parameters, weighted consistency variables and total step size variables are sent to all computers communicating with the neural network. When all the computers finish the iterative training, the final neural network parameterson any computer are the final trained parameters of the neural network, and the neural network optimization is completed. The invention has the advantages of simple implementation, fast network training speed and good expansibility for large-scale data sets and computer clusters.
Description
technical field [0001] The invention belongs to the field of deep learning, and in particular relates to an asynchronous parallel optimization method for neural network training. Background technique [0002] Artificial intelligence and its related computer learning, deep learning and other fields have received extensive attention in recent years. Its applications in face recognition, face detection, natural language processing, speech recognition and other fields have also shown strong capabilities. [0003] A very common and important technology in artificial intelligence and deep learning is artificial neural network (referred to as neural network). A neural network is composed of several neurons, and a neuron can be regarded as a function that receives several input signals and then outputs a signal according to certain rules. This function corresponds to some adjustable parameters. Neurons are connected in series and in parallel according to certain rules to form a ne...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06N3/063G06N3/08
CPCG06N3/063G06N3/08
Inventor 游科友张家绮宋士吉
Owner TSINGHUA UNIV