Parallel training method and device of neural network model and electronic equipment

Abstract

The invention provides a parallel training method of a neural network model. The method comprises the following steps: segmenting a target neural network model to obtain a block structure of the target neural network model; when the target neural network model is trained, monitoring the state change of the tensor information; determining the state of a block structure of the target neural network model according to the state change of the tensor information; in response to the state of the block structure of the target neural network model, the storage positions of the parameters of the target neural network model corresponding to different block structures are adjusted through the expelling strategy of the block structure, and the invention further provides a parallel training device of the neural network model, electronic equipment and a storage medium. According to the invention, parallel training of the target neural network model can be realized.

Description

technical field [0001] The invention relates to a training technology of a neural network model, in particular to a parallel training method, device, electronic equipment and storage medium of a neural network model. Background technique [0002] A machine learning model usually includes multiple operators. If the calculation time of each operator is too long, the service response will be too slow. For example, for the BERT model (Bid directiona l Encode eRepresentation from Transformers, a general pre-trained language representation model) in the field of natural language, in actual application scenarios, rich online service scenarios can be deployed based on the BERT model. In this scenario, people often use Graphics Processing Unit (GPU) to parallelize the BERT service calculation process to improve online response speed and reduce service delay. Softmax (normalization) is an important operator in the BERT model. If this operator runs on the GPU for too long, it will lea...

AI Technical Summary

Problems solved by technology

Softmax (normalization) is an important operator in the BERT model. If this operator runs on the GPU for too long, it will lead to inefficiency in the overall reasoning task of BERT. Therefore, it is necessary to improve the neural network model through parallel processing. The training speed reduces the training time of the neural network model

Images