Compression method of neural language model based on tensor decomposition technology

Abstract

The invention discloses a compression method of a neural language model based on a tensor decomposition technology. According to the method, starting from linear representation of an original attention function, it is proved that the attention function can be linearly represented by a set of standard orthogonal basis vectors, and then parameters are compressed by sharing the set of basis vectors under the condition that a multi-head mechanism is constructed; meanwhile, the neural network model can have stronger discrimination capability through modeling in a tensor slicing mode. A new idea isprovided for developing a neural network model with low parameters and high accuracy.

Description

technical field [0001] The present invention relates to the field of neural language model compression, in particular, relates to compressing the original attention function of the Transformer neural network model. Background technique [0002] With the development of artificial intelligence, in the field of natural language processing, the neural language pre-training (Pre-training) model has demonstrated its effectiveness for most tasks. The Transformer model is based on the attention mechanism, replacing the cyclic neural network and the convolutional neural network. Currently, this model has been extensively extended, playing a key role in many other pre-trained language models, for example, the BERT pre-trained model. However, a large number of these pre-trained models are difficult to deploy on limited resources due to the large number of parameters. Therefore, the compression of pre-trained language models is an important research problem. [0003] Several methods ...

AI Technical Summary

Problems solved by technology

[0004] Tensor technology has appeared as a model compression technology. In general, they are used alone in the input layer of the model and the fully connected layer. These compressions have solved the problem of excessive model parameters to a certain extent. However, The work of combining multiple tensor compression methods to compress the internal structure of the model is missing

Images