Language model compression method and system

A technology of language model and compression method, applied in the direction of biological neural network model, special data processing application, instrument, etc., can solve the problem of LSTM language model research without large vocabulary, and achieve the effect of high memory compression rate

Pending Publication Date: 2018-11-23
AISPEECH CO LTD
View PDF3 Cites 6 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, only a few works are related to recurrent neural networks, and there is no comprehensive study of binarized large-vocabulary LSTM language models.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Language model compression method and system
  • Language model compression method and system
  • Language model compression method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts all belong to the protection scope of the present invention.

[0021] Below, first introduce the implementation mode of the present application, then will use experimental data to prove that the scheme of the present application is different from the prior art, and what beneficial effects can be realized.

[0022] Please refer to figure 1 , which shows a flow chart of an embodiment of the language model co...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a language model compression method and system. The method comprises the steps of, in response to an acquired input parameter, searching for a binary vector corresponding to the input parameter; at least enabling the binary vector to pass through a binary cyclic network layer, so as to obtain a binary cyclic network layer output; and at least enabling the cyclic network layer output to pass through a binary linear layer and outputting a result. The binary vector is used for encoding parameters of word embedding and LSTM, so that relatively high memory compression rate is achieved. The application of the binary LSTM to a large word list language model is innovatively explored. Furthermore, an experiment shows that the proposed model achieves a lossless compression ratio of 11.3 in Chinese and English data sets, and can achieve a compression ratio of 31.6 under the condition of losing a small part of the performance.

Description

technical field [0001] The invention belongs to the technical field of language model compression, in particular to a language model compression method and system. Background technique [0002] Language model (LM, Language Mode) plays an important role in natural language processing (NLP, Natural Language Processing) tasks. The N-gram language model used to be the most popular language model. Considering the previous N-1 words, the N-gram language model predicts the next word. However, this leads to loss of long-term dependencies. As N grows, the sample space size grows exponentially, which leads to sparse data. [0003] The neural network (NN, Neural Networks) model was first introduced into language modeling in 2003. Given a context with a fixed size, the model computes a probability distribution over the next word. However, the problem of long-term dependencies still exists because the context window is fixed. Currently, models based on Recurrent Neural Networks (RN...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/22G06N3/02
CPCG06N3/02G06F40/12
Inventor 俞凯刘轩曹迪石开宇
Owner AISPEECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap