Distributed parallel training method and system for neural network acoustic model

Abstract

The invention provides a distributed parallel training method and system for a neural network acoustic model. The distributed parallel training method comprises the steps that 101) each group of training data is inputted into a client respectively; 102) each of the client receives the inputted training data, and transmits parameters between a plurality of GPU and a first CPU, which are arranged in the client, by adopting a bus, wherein the parameters comprise the model weight and the gradient; each GPU calculates the gradient based on the inputted model weight parameter and inputs the calculated gradient into the first CPU; the first CPU updates a model copy in the corresponding client by using the gradient uploaded by the GPUs, transmits an updated weight parameter back to each GPU so as to be used for carry out gradient calculation again, and meanwhile, the first CPU accumulates the gradient inputted by each GPU and updates a model in a parameter server according to a accumulation result; and 103) latest gradient information acquired by accumulation carried out by the CPU of each client is inputted into the server by adopting network resources, and then a neural network acoustic model stored in the server is updated.

Description

technical field [0001] The invention belongs to the field of speech recognition, and is a method for using multiple computer nodes to train a neural network in parallel to improve the training speed of a neural network acoustic model, and specifically relates to a distributed parallel training method and system for a neural network acoustic model. Background technique [0002] At present, the method of using large data volume and deep neural network (Deep Neural Network, DNN) to build an acoustic model has achieved outstanding results in the field of speech recognition, which has improved the final recognition accuracy by 20% to 30%. [0003] DNN is a technology that simulates the work of neurons in the human brain by connecting weights and nodes. It can be regarded as a classifier. The DNN structure mainly includes an input layer, a hidden layer and an output layer. There are weighted line connections, the number of nodes in the output layer is determined by the number of t...

AI Technical Summary

Problems solved by technology

[0004] However, due to the frequent parameter transfer and extremely high requirements on network bandwidth in the process of distributed training of neural networks, most of the current parallel training systems for neural networks use one machine and insert multiple GPU cards into it. , but this method has its limitations, only four GPU cards can be used at most, and the acceleration effect that can be provided is limited

Images