Deep learning model training method, device and system based on mixed precision

Abstract

The invention discloses a deep learning model training method, device and system based on mixed precision, and the method comprises the steps: carrying out the data processing of sample data in a sample data set based on a deep learning model obtained through the last training, and obtaining a set number of first weight gradient data with the data precision being the first data precision; determining the data precision as a scaling coefficient of the second data precision according to the set number of first weight gradient data and the second data precision, the first data precision being higher than the second data precision; based on the sample data set and the scaling coefficient, training the deep learning model to update the weight parameters of the deep learning model, obtaining thedeep learning model trained this time, and the scaling coefficient is used for amplifying the loss value with the data precision being the second data precision in the process of training the deep learning model, so that the training efficiency and the training precision are improved.

Description

technical field [0001] The present invention relates to the technical field of model training, in particular to a mixed precision-based deep learning model training method, device and system. Background technique [0002] Deep learning models have been widely used in various fields, such as robotics, speech recognition, image recognition, and natural language processing. In practical applications, before the deep learning model is put into application, it is necessary to use a large amount of sample data to train the deep learning model. How to improve the training efficiency of the deep learning model and reduce the training cost is particularly important. At present, major hardware chip manufacturers have launched AI acceleration chips with super computing power in the range of low-precision values. Taking the V100 chip as an example, the computing power of its half-precision data processing unit is 10 times that of a single-precision data processing unit. In order to tak...

AI Technical Summary

Problems solved by technology

[0004] Embodiments of the present invention provide a mixed precision-based deep learning model training method, device, electronic equipment, system, and storage medium to solve the problem of how to improve the efficiency of searching for scaling coefficients in the prior art

Images