A text generation algorithm integrating multi-type knowledge base and reasoning technology

Abstract

The present invention discloses a text generation algorithm that integrates multiple types of knowledge bases and reasoning technology: training COMET to generate reasoned common sense knowledge and events; extracting event sets and keyword sets from the dialogue text in the dialogue generation dataset; using the event set and keyword set as the input of the COMET model to output the corresponding common sense knowledge and reasoning knowledge; fine-tuning the pre-trained model GPT-2 and applying it to downstream tasks; using the dialogue text and the internal knowledge of the extracted event set and keyword set, or the external knowledge generated by COMET as the input of the GPT-2 encoder, and combining it with the attention mechanism to generate output text. By combining the generative common sense model with the pre-trained model, the present invention completes the process of integrating two different types of knowledge bases and knowledge reasoning, introducing concise and clear internal knowledge and rich and diverse external knowledge to dialogue generation, which helps GPT-2 generate more coherent and diverse responses.

Description

Technical Field

[0001] The present invention relates to a text generation algorithm integrating multi-type knowledge bases and reasoning technology, and belongs to the field of natural language processing in the computer field. Background Art

[0002] In 2013, Thomas et al. proposed the word embedding model, ushering in the era of neural networks and deep learning for natural language generation. In 2017, the Google Machine Translation team proposed the attention model and a series of deep learning models based on the Transformer. In just five years, the Transformer has become a mainstream model in natural language processing. Pre-trained models based on the Transformer achieve state-of-the-art performance across various tasks, and attention mechanisms have become a research hotspot for neural network-based tasks in recent years.

[0003] Text generation has a wide range of practical applications, including dialogue generation, story completion, and machine translation. Over the years, researchers have made significant efforts to improve the effectiveness and performance of text generation, moving closer to the goal of enabling machines to express themselves as fluently as humans in real-world applications.

[0004] In dialogue systems or question-answering systems, generating human-satisfactory responses based solely on the input text is often extremely difficult. To generate meaningful and informative output text, incorporating knowledge beyond the input sequence has become a popular approach in text generation. This research area, which incorporates knowledge into text generation, is known as knowledge-enhanced text generation.

[0005] Humans are able to continuously acquire and store knowledge from a wide range of sources, building their own knowledge base in their brains. Furthermore, when reading text or engaging in conversation, humans often make common-sense inferences to form their own understanding. In question-answering and dialogue systems, generating better responses may require reasoning and inference based on facts and common sense. Knowledge reasoning for knowledge graphs aims to infer new knowledge from existing knowledge. It uses rule mining to reason about associations within knowledge graphs, inferring unknown facts or relationships based on existing facts or relationships within the graph. Currently, knowledge reasoning for knowledge graphs plays an important role in numerous applications.

[0006] In dialogue systems, knowledge-enhanced Seq2Seq methods enable models to better understand the semantic meaning of input sequences and generate more meaningful responses. Similarly, in summary generation, question answering, and story generation, the inclusion of knowledge graphs enables machines to generate more detailed language, improving text generation performance. Summary of the Invention

[0007] The goal of this invention is to design a text generation algorithm that integrates multi-type knowledge bases and reasoning techniques to improve the performance of existing text generation algorithms. This invention provides a knowledge enhancement method for various downstream tasks in natural language processing (such as dialogue generation and story generation), capable of generating informative, logical, and coherent responses on text generation datasets.

[0008] The present invention adopts the following technical solutions to solve the technical problems:

[0009] A text generation algorithm integrating multi-type knowledge base and reasoning technology includes the following steps:

[0010] (1) Training of the generative common sense model COMET:

[0011] (2) Extraction of event sets and keyword sets from conversation texts;

[0012] (3) Generation of external knowledge;

[0013] (4) Generation of knowledge-enhanced dialogue;

[0014] The conversation text and the internal knowledge of the extracted event set and keyword set in step (2), or the external knowledge generated by COMET in step (3) are used as the input of the GPT-2 encoder, and the output text is generated in combination with the attention mechanism.

[0015] Furthermore, the training of the COMET model in step (1) includes the following steps:

[0016] (14) ATOMIC knowledge base and ConceptNet knowledge base are used as knowledge seed sets to train COMET;

[0017] (15) COMET adopts the GPT language model architecture, using multiple Tansformer blocks with multi-head scaled dot product attention and fully connected layers to encode the input text;

[0018] (16) Before each word of the input text is passed into the neural network layer, COMET uses the self-attention mechanism to assign a relevance score to each word in the input text sequence, and then sums their vector representations to obtain the final output of the self-attention layer.

[0019] Furthermore, the event extraction and keyword extraction of the dialogue text in step (2) includes the following steps:

[0020] (21) Use the ASER event extraction tool to extract events from all sentences in the dialogue generation dataset EmpatheticDialogue to form an event set;

[0021] (22) Search the ConceptNet knowledge base for words that appear in the conversation text context of the dataset and extract a set of keywords.

[0022] Furthermore, the generation of external knowledge in step (3) includes the following steps:

[0023] (33) The extracted event set and keyword set are used as the input of the COMET model respectively; the corresponding common sense knowledge and reasoning knowledge are output;

[0024] (34) During the decoding process of step (31), a beam search strategy is used to select events or keywords generated by COMET as external knowledge for text generation.

[0025] Furthermore, in the attention mechanism described in step (4), the formula of the attention vector is:

[0026] m=MLP([MHA(x);MHA(r)])

[0027] Among them, MHA(x) is the output of the multi-head attention function of the original Transformer layer, and MHA(r) is the output of the event and keyword query layer.

[0028] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects:

[0029] This paper proposes a text generation algorithm that integrates multi-type knowledge bases and reasoning techniques to improve the consistency and diversity of open-ended text generation responses. This paper uses a generative commonsense model to automatically construct a commonsense knowledge base. It employs a COMET-based knowledge fusion and knowledge reasoning strategy, and trains COMET using the ATOMIC reasoning knowledge base and the ConceptNet commonsense knowledge base as knowledge seed sets, providing rich and diverse open-ended commonsense knowledge for downstream text generation applications. This paper combines the knowledge generated by COMET with a pre-trained language model and employs a beam search-based decoding algorithm to better help the model understand the input text and contextual information. The proposed method is evaluated on the open-ended text generation task of dialogue generation. Experimental results show that introducing commonsense events and keywords generated by COMET helps GPT-2 generate more coherent and diverse text. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the proposed algorithm framework. DETAILED DESCRIPTION

[0031] The present invention will be described in further detail below with reference to the accompanying drawings.

[0032] In order to improve the consistency and diversity of open text generation responses, the overall framework of the text generation algorithm proposed in this paper is as follows: Figure 1 As shown in the figure, the process is divided into four parts: 1. Training the generative common sense model COMET. 2. Event and keyword extraction from conversation text. 3. Generation of external knowledge. 4. Knowledge-enhanced conversation generation.

[0033] (1) Training of the Generative Common Sense Model COMET

[0034] The generative commonsense model COMET is tasked with generating attributes or values ​​from objects and relations in a given knowledge tuple. The model accepts a continuous sequence of words contained in a tuple as input to the encoder and outputs a continuous sequence of words for the attribute or value. The GPT model provided by HuggingFace serves as the basis for COMET. The initialization parameters are the weights of the initial language model proposed by Radford et al. Following the GPT model design of Radford et al., COMET is initialized with 12 layers, 768 hidden states, and 12 attention heads. A dropout rate of 0.1 is used, and GeLU is used as the activation function. During training, the batch size is set to 64. A maximum learning rate of 6.25e-5 is used for the ATOMIC dataset, and 1e-5 is used for the ConceptNet dataset.

[0035] (2) Event extraction and keyword extraction from conversation text

[0036] The ASER event extraction tool was used to extract events from all sentences in the dataset, ensuring that the extracted events had complete semantics and maintained a consistent format with the events in the ATOMIC dataset. All words appearing in the sentences in the dataset were searched in the ConceptNet knowledge base to extract a set of keywords.

[0037] (3) Generation of external knowledge

[0038] The task of the generative common sense model COMET is to generate attributes or values ​​o from the object s and relation r of the given knowledge tuple. The model accepts the continuous word sequence [X s , X r ] is taken as input to the encoder and outputs a continuous sequence of words representing the attribute or value o.

[0039] The extracted events and keywords are used as the input of the COMET model respectively, and the generated events or keywords are selected using the beam search strategy, which are used as the external knowledge for text generation.

[0040] The beam search algorithm finds the optimal solution in a relatively restricted search space at a low cost, resulting in a solution close to the optimal solution in the entire search space. The beam search algorithm selects multiple candidate sequences for the input sequence at each time step based on conditional probabilities. The number of candidate sequences depends on the beam width k. At each time step, the beam search selects the k sequences with the highest conditional probabilities as possible choices for that time step.

[0041] (4) Knowledge-enhanced dialogue generation

[0042] This method fine-tunes GPT-2 based on pre-training with a large number of unsupervised examples and a small number of supervised examples from dialogue generation tasks. First, keywords and events that embody the core content of the input text are extracted as internal knowledge input to GPT-2. These extracted events and keywords are then fed into the COMET generative commonsense model, which selects relational dimensions that are more helpful for dialogue generation and generates corresponding events and keywords. These are also added to the GPT-2 input encoder as external knowledge. Furthermore, an attention mechanism is used to filter out poor-quality and noisy knowledge. Finally, a beam search decoding strategy is used to generate responses to the input sequence.

[0043] The formula for the attention vector is:

[0044] m=MLP([MHA(x);MHA(r)])

[0045] Among them, MHA(x) is the output of the multi-head attention function of the original Transformer layer, and MHA(r) is the output of the event and keyword query layer.

[0046] In the dialogue generation task using the EmpatheticDialogues dataset, the order of sequence encoder input is the input text sequence, the [MASK] sequence, the event sequence extracted from the input text, the [MASK] sequence, and the new knowledge generated by COMET. The generated knowledge can be events or keywords from ATOMIC or ConceptNet.

[0047] GPT-2 outputs a sequence of words one at a time. After generating each word, it adds it to the input sequence, which becomes the input for the next step of the model. A trained GPT-2 model contains two weight matrices: an embedding matrix and a positional encoding matrix. GPT-2 searches the embedding matrix for the corresponding word. These embedding vectors represent the corresponding word. Their length is related to the size of the GPT-2 model. The GPT-2small version used in this experiment uses an embedding vector of length 768 to represent a single word. The positional encoding matrix indicates the order of words in the sequence when the input word enters the Transformer module. In each Transformer module, the resulting vector obtained after the input word is processed by the self-attention layer and the neural network layer is passed to the next Transformer module in sequence.

[0048] A small version of the GPT-2 model provided by HuggingFace was used as the basis for the dialogue generation experiments. This version uses a Transformer architecture with 12 layers, 768-dimensional hidden states, and 12 attention heads. The pre-trained GPT-2 Byte Pair Encoding (BPE) tokenizer with an expanded vocabulary of 50,282 words is used to tokenize the text.

Claims

1. A text generation algorithm that integrates multi-type knowledge base and reasoning technology, characterized by: The following steps are involved: (1) Training of the generative common sense model COMET: (2) Extraction of event sets and keyword sets from conversation texts; (3) Generation of external knowledge; (4) Generation of knowledge-enhanced dialogue; The internal knowledge of the conversation text and the extracted event set and keyword set in step (2), or the external knowledge generated by COMET in step (3) is used as the input of the GPT-2 encoder, and the output text is generated in combination with the attention mechanism; The generation of external knowledge in step (3) includes the following steps: The extracted event set and keyword set are used as the input of the COMET model respectively; the corresponding common sense knowledge and reasoning knowledge are output; In the decoding process of step (31), a beam search strategy is used to select events or keywords generated by COMET as external knowledge for text generation; In the attention mechanism described in step (4), the formula of the attention vector is: ； in, is the output of the multi-head attention function of the original Transformer layer, It is the output of the event and keyword query layer.

2. The text generation algorithm integrating multi-type knowledge base and reasoning technology as claimed in claim 1 is characterized in that: The training of the COMET model in step (1) includes the following steps: The ATOMIC knowledge base and ConceptNet knowledge base are used as knowledge seed sets to train COMET; COMET adopts the GPT language model architecture, using multiple Tansformer blocks with multi-head scaled dot-product attention and fully connected layers to encode the input text; Before passing each word of the input text into the neural network layer, COMET uses the self-attention mechanism to assign a relevance score to each word in the input text sequence, and then sums their vector representations to obtain the final output of the self-attention layer.

3. The text generation algorithm integrating multi-type knowledge base and reasoning technology as claimed in claim 1 is characterized in that: Step (2) extracting events and keywords from the conversation text includes the following steps: (21) Use the ASER event extraction tool to extract events from all sentences in the dialogue generation dataset EmpatheticDialogue to form an event set; (22) Search the ConceptNet knowledge base for words that appear in the conversation text context of the dataset and extract a set of keywords.

Images