Model-parallel full-connected layer data exchange method and system for deep neural network

Abstract

The invention discloses a model-parallel full-connected layer data exchange method and system for a deep neural network. The model-parallel full-connected layer data exchange method for a deep neural network includes the steps: uniformly dividing the full-connected layer of the deep neural network to N training units according to the number of nerve cells, and forming a network model being parallel with the full-connected layer model in the deep neural network; during the forward propagation process of the full-connected layer, utilizing a half-stop waiting forward propagation method to employ the processing modes of partial arrival, partial calculation, overall output and overall propagation on the input data of the front layer; during the backward propagation process of the full-connected layer, utilizing a quantified half-stop waiting backward propagation method to employing the processing modes of quantified arrival, quantified calculation and quantified propagation on the residual error data of the back layer; and after completing the primary forward and backward propagation, according to the solved weight gradient and threshold gradient, parallelly updating the weight data and threshold data of each layer. The model-parallel full-connected layer data exchange method for a deep neural network can overlap data communication and data calculation of the full-connected layer, and can accelerate convergence of the acceleration model on the premise of guaranteeing the correct rate.

Description

technical field [0001] The invention belongs to the technical field of deep learning, and more specifically relates to a method and system for exchanging fully connected layer data parallel to a model in a deep neural network. Background technique [0002] Deep Neural Network (DNN) is an artificial neural network (Artificial Neural Network, ANN) composed of an input layer, multiple hidden layers, and an output layer. Each layer is composed of multiple neuron nodes. The neuron nodes of the front layer and the back layer are connected to each other, such as figure 1 as shown, figure 1 All layers in are on the same training unit, I represents the input layer, H represents the hidden layer (the hidden layer needs to have more than one), O represents the hidden layer, the thin line represents the connection between neurons and neurons, and the thick line represents the component Connect with a component (here a layer). In the neural network model, a fully-connected layer (Full...

AI Technical Summary

Problems solved by technology

This solves the technical problem of large communication overhead in the prior art

Images