Multilayer neural network language model training method and device based on knowledge distillation

A multi-layer neural network and language model technology, applied in biological neural network models, neural learning methods, knowledge expression, etc., can solve problems such as large and complex network structures, slow training speed, etc., to achieve fast training speed and good coding ability. , the effect of improving the accuracy

CN111611377AActive Publication Date: 2020-09-01HUAIYIN INSTITUTE OF TECHNOLOGY
4 Cites 47 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2020-09-01

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a multilayer neural network language model training method and device based on knowledge distillation. The method comprises the steps that firstly, a BERT language model and amulti-layer BILSTM model are constructed to serve as a teacher model and a student model, the constructed BERT language model comprises six layers of transformers, and the multi-layer BILSTM model comprises three layers of BILSTM networks; then, after the text corpus set is preprocessed, the BERT language model is trained to obtain a trained teacher model; and the preprocessed text corpus set is input into a multilayer BILSTM model to train a student model based on a knowledge distillation technology, and different spatial representations are calculated through linear transformation when an embedding layer, a hiding layer and an output layer in a teacher model are learned. Based on the trained student model, the text can be subjected to vector conversion, and then a downstream network is trained to better classify the text. According to the method, the text pre-training efficiency and the accuracy of the text classification task can be effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the fields of unsupervised text pre-training and deep learning, in particular to a multi-layer neural network language model training method and device based on knowledge distillation. Background technique

[0002] With the rapid increase of online text information data on the Internet, the language model plays a vital role in information processing. It is a key technology for processing large-scale text information and promotes the development of information processing in the direction of automation. A language model is simply the probability distribution of a sequence of words. Building a reasonable pre-trained language model can solve many current text information problems, such as text classification, text similarity, reading comprehension, etc., and then can efficiently use a large amount of text corpus data on the Internet to better provide people with More convenient service. However, most of the current text pre-train...

Examples

Embodiment Construction

[0026] In order to clearly illustrate the technical solution of the present invention, the related technologies involved in the present invention will firstly be briefly described below.

[0027] BERT (Bidirectional Encoder Representation from Transformers, Transformer's bidirectional encoding representation) language model: BERT uses the masked model to realize the bidirectionality of the language model, which proves the importance of bidirectionality for language representation pre-training. The BERT model is a two-way language model in the true sense, and each word can use the context information of the word at the same time. BERT is the first fine-tuning model to achieve the best results in both sentence-level and token-level natural language tasks. It is proved that pre-trained representations can alleviate the design requirements of special model structures for different tasks. BERT achieves the best results on 11 natural language processing tasks. And in BERT's extens...