BERT-based sequence generation method and device

Abstract

The invention discloses a BERT-based sequence generation method and device, relates to the technical field of natural language processing, and mainly aims to process a sequence generation task by using a BERT model. According to the main technical scheme, the method comprises the steps of obtaining a sequence generation model constructed based on a BERT model; setting iteration parameters of the sequence generation model; inputting first sequence data to the sequence generation model; and generating second sequence data by the sequence generation model according to the first sequence data and iteration parameters.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to a BERT-based sequence generation method and device. Background technique [0002] The BERT model is a language model proposed by Google based on a bidirectional Transformer. The BERT model combines the pre-training model and the downstream task model, that is to say, the BERT model is still used when doing downstream tasks. The full name of BERT is Bidirectional Encoder Representations from Transformers, which is a two-way encoder representation based on Transformer. Among them, the two-way meaning means that when processing a word, it can consider the information of the words before and after the word, so as to obtain the semantics of the context . BERT pre-trains deep bidirectional representations by jointly conditioning the context in all layers, thus, the pre-trained BERT representations can be fine-tuned with an additional output layer to quickl...

AI Technical Summary

Problems solved by technology

[0003] However, according to the above structure of the BERT model, the natural language processing tasks that the model can currently handle are mainly concentrated on text classification tasks, such as sentiment classification; sequence tagging tasks, such as real-time word segmentation recognition, part-of-speech tagging, etc., cannot handle sequence generation tasks , such as sentence simplification, machine translation and other tasks

However, the existing sequence generation model uses a one-way left-to-right decoding method when processing sequence generation tasks, which makes the training objectives and generation methods of the BERT model significantly different from the existing sequence generation models, which cannot be well understood. Applying BERT models to sequence generation tasks

Images