Hybrid precision training method and device, electronic equipment and storage medium

Abstract

The invention relates to a hybrid precision training method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a representation factor corresponding toeach training sample identification in a training data set obtained through the previous round of hybrid precision training, the representation factor being used for representing the difficulty degreeof model fitting, the training data set being composed of a plurality of training samples generated based on multimedia data; dividing the training samples in the training data set into a plurality of batches according to the representation factors corresponding to the training sample identifiers; and according to the training samples of the multiple batches and the scaling factor corresponding to each batch, performing mixing precision training on a model in batches so as to perform attribute prediction on the multimedia data in the training samples of each batch through the model. Accordingto the invention, gradient overflow can be avoided, the problem of training precision loss caused by gradient overflow in a training process in related technologies is solved, and the accuracy of a model prediction result is improved.

Description

technical field [0001] The present disclosure relates to machine learning technology, and in particular to a mixed precision training method, device, electronic equipment and storage medium. Background technique [0002] The mixed-precision training method has been widely used in the training acceleration of deep learning networks. By using IEEE16-bit half-precision values ​​to calculate and store weights and gradients, it can make full use of the tensor core hardware unit in the GPU to accelerate mathematical operations. In the near future In a GPU, half-precision computing throughput can be 2-8 times that of single-precision. At the same time, it can also greatly reduce the memory usage of the model during training, thus making the training of larger models possible. [0003] Half-precision floating-point uses only two bytes, compared to four bytes for single-precision floating-point, making its natural storage space half that of single-precision floating-point. This als...

AI Technical Summary

Problems solved by technology

[0005] The present disclosure provides a mixed-precision training method, device, electronic equipment, and storage medium to at least solve the problem of loss of training accuracy caused by gradient overflow in the training process in the related art

Images