Pre-training framework with two-stage decoder for language understanding and generation

Abstract

The embodiment of the invention provides a pre-training framework with a two-stage decoder for language understanding and generation, comprising: an encoder for receiving a noisy text in a conditional generation task and encoding to obtain a noisy text sequence, the noisy text comprising a text with covered segments and / or disordered sentences; a decoder used for receiving the noisy text sequence output by the encoder, reconstructing the noisy text sequence in a first decoding stage to obtain a reconstructed text, and generating a significance representation corresponding to the noisy text sequence. In a second decoding stage, subsequent text is generated based on the reconstructed text and the meaning representation. The embodiment of the invention also provides a two-stage decoder. According to the embodiment of the invention, reconstruction and understanding are carried out in the first decoding stage, the understanding of the text is ensured in the second decoding stage, and the subsequent text can be generated, so that the pre-training quality is better. Explicit understanding can be applied to downstream tasks, contextual information can be conveniently obtained, and the application task range is wider.

Description

technical field [0001] The invention relates to the field of intelligent speech, in particular to a pre-training framework for language understanding and generation with a two-stage decoder. Background technique [0002] Self-supervised pre-training improves the state of the art for natural language generation (NLG) tasks, various language generation (conditional generation) tasks aim to generate natural language sentences given a document (conditional), such as context in task-oriented dialogue To response (Context-to-Response), session response generation, etc. [0003] To achieve pre-training, there are various types of pre-trained models for conditional text generation tasks. Most of these models fall into two categories. The first type of model, such as MASS (Masked Sequence to Sequence, covered sequence to sequence) and BART (Bidirectional and Auto-Regressive Transformers, two-way auto-regressive Transformer), decodes the masked part given the damaged text or reply ...

AI Technical Summary

Problems solved by technology

[0006] In order to at least solve the problems in the prior art that there is no relevant pre-training for subsequent text generation, resulting in insufficient subsequent generation capabilities, and lack of explicit generation related understanding, resulting in insufficient and unsmooth subsequent text generation capabilities

Images