Language model compression method and system

Abstract

The invention discloses a language model compression method and system. The method comprises the steps of, in response to an acquired input parameter, searching for a binary vector corresponding to the input parameter; at least enabling the binary vector to pass through a binary cyclic network layer, so as to obtain a binary cyclic network layer output; and at least enabling the cyclic network layer output to pass through a binary linear layer and outputting a result. The binary vector is used for encoding parameters of word embedding and LSTM, so that relatively high memory compression rate is achieved. The application of the binary LSTM to a large word list language model is innovatively explored. Furthermore, an experiment shows that the proposed model achieves a lossless compression ratio of 11.3 in Chinese and English data sets, and can achieve a compression ratio of 31.6 under the condition of losing a small part of the performance.

Description

technical field [0001] The invention belongs to the technical field of language model compression, in particular to a language model compression method and system. Background technique [0002] Language model (LM, Language Mode) plays an important role in natural language processing (NLP, Natural Language Processing) tasks. The N-gram language model used to be the most popular language model. Considering the previous N-1 words, the N-gram language model predicts the next word. However, this leads to loss of long-term dependencies. As N grows, the sample space size grows exponentially, which leads to sparse data. [0003] The neural network (NN, Neural Networks) model was first introduced into language modeling in 2003. Given a context with a fixed size, the model computes a probability distribution over the next word. However, the problem of long-term dependencies still exists because the context window is fixed. Currently, models based on Recurrent Neural Networks (RN...

AI Technical Summary

Problems solved by technology

However, only a few works are related to recurrent neural networks, and there is no comprehensive study of binarized large-vocabulary LSTM language models.

Images