Model training method

Abstract

The invention provides a model training method. The method comprises the following steps: defining a new quantitative linear layer; all elements in the multi-dimensional input tensor of the quantization linear layer are quantized into a PINT format, all elements in the to-be-calculated tensor of the quantization linear layer are quantized into a PINT data format, and matrix multiplication calculation is performed on the quantized multi-dimensional input tensor and the to-be-calculated tensor to obtain a fixed point result; performing inverse quantization on a fixed point result into a floating point number and spreading the floating point number to a subsequent network layer; and replacing an original linear layer in the model with a quantitative linear layer, and training the model based on a floating-point number and a PINT data format. According to the method, the quantization linear layer based on the PINT data format is developed, the PINT data format with low bit and high representation capability is applied to model training, and the quantization linear layer is used for replacing the linear layer used in the model, so that the requirements in the aspects of data calculation, storage and the like are effectively reduced under the condition that the accuracy change of the trained model is very small.

Description

technical field [0001] The application relates to the technical field of natural language processing, in particular to a model training method. Background technology [0002] In recent years, Bert and other models based on transformer network have performed well in natural language processing and other fields. Transformer is a classic model applied to NLP (natural language processing) proposed by Google's team in 2017. Transformer model uses self attention mechanism and does not adopt the sequential structure of RNN (recurrent neural network), so that the model can be trained in parallel and have the global information of samples. The popular models such as Bert (bidirectional encoder representations from transformers) are also based on the transformer implementation. [0003] Take the Bert model in the model as an example. The full name of Bert is bidirectional encoder representations from transformers, which is a pre trained language representation model. Bert model uses the en...

AI Technical Summary

Problems solved by technology

[0005] This application provides a model training method to solve the problem of requiring huge computing and storage resources in the model training process

Images