A short text story generation method and system based on common sense knowledge base

Abstract

The present invention discloses a method and system for generating short text stories based on a common sense knowledge base. The method comprises: constructing a common sense knowledge base; constructing a first training sample based on the common sense knowledge base and training a knowledge construction module to obtain a trained knowledge construction module; collecting a second training sample and training a text generation module to obtain a trained text generation module; obtaining the first sentence of a story, and iteratively generating a short text story of a preset length based on the trained knowledge construction module and the trained text generation module. The system comprises a common sense knowledge base, a knowledge construction module, and a text generation module. By using the present invention, short text stories can be automatically generated, and the story content is logically coherent and consistent with common sense. As a method and system for generating short text stories based on a common sense knowledge base, the present invention can be widely applied in the field of natural language processing.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method and system for generating short text stories based on a common sense knowledge base. Background Art

[0002] Story generation is a crucial task in natural language processing and has become a crucial component of natural language processing. However, implementing a good story generation model has always been a challenging task. Story generation tasks involve humans providing a story title or the first few sentences as input, and the model automatically generates the story content based on this input. Story generation has a wide range of applications, such as intelligent writing and article polishing. One of the key metrics for evaluating the quality of generated story content is story coherence. Existing autoregressive generative models only reference the existing story context when generating the next part of a story, without considering whether the generated next part is logically coherent with the existing plot. This ultimately results in the generated text being grammatically correct but containing common sense logical errors. Summary of the Invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a short text story generation method and system based on a common sense knowledge base, which can automatically generate short text stories with logical coherence and common sense in the story content.

[0004] The first technical solution adopted by the present invention is: a method for generating short text stories based on a common sense knowledge base, comprising the following steps:

[0005] Build a common sense knowledge base;

[0006] constructing a first training sample based on the common sense knowledge base and training the knowledge construction module to obtain a trained knowledge construction module;

[0007] Collecting a second training sample and training the text generation module to obtain a trained text generation module;

[0008] Get the first sentence of the story, and iteratively generate a short text story of a preset length based on the trained knowledge construction module and the trained text generation module.

[0009] Furthermore, each piece of common sense in the common sense knowledge base includes context, events, relationships and consequences.

[0010] Furthermore, the step of constructing a first training sample based on the common sense knowledge base and training the knowledge construction module to obtain a trained knowledge construction module specifically includes:

[0011] Partially splicing the context, events, relations, and consequences in the common sense knowledge base to obtain the first training sample;

[0012] The context, events, and relationships in the first training sample are used as inputs, and the consequences in the first training sample are used as outputs, to train the knowledge construction module and obtain a trained knowledge construction module.

[0013] Furthermore, the step of collecting the second training sample and training the text generation module to obtain a trained text generation module specifically includes:

[0014] collecting a second training sample, wherein the second training sample includes supplementary information of the existing story content and the knowledge module;

[0015] The second training sample is used as input and the next sentence of the existing story content is used as a label to train the text generation module to obtain a trained text generation module.

[0016] Furthermore, the knowledge construction module and the text generation module both include a GPT-2 model and peripheral scripts.

[0017] Furthermore, the step of obtaining the first sentence of the story and iteratively generating a short text story of a preset length based on the trained knowledge construction module and the trained text generation module specifically includes:

[0018] Get the first sentence of the story and set the number of sentences in the final short text story;

[0019] Based on the common sense knowledge base, the first sentence of the story and the current story content are combined to perform text splicing to obtain the first input;

[0020] Send the first input to the trained knowledge building module to generate the corresponding consequence output;

[0021] Based on the common sense knowledge base, the first sentence of the story, the current story content and the corresponding consequence output are combined to perform text splicing to obtain the second input;

[0022] Feed the second input into the trained text generation module to update the story content;

[0023] The working steps of the knowledge construction module and the text generation module are looped until the number of sentences in the story content reaches a preset value.

[0024] The second technical solution adopted by the present invention is: a short text story generation system based on a common sense knowledge base, comprising:

[0025] Common sense knowledge base, used to store common sense knowledge;

[0026] The knowledge building module is used to infer and output the corresponding consequences based on the received story content;

[0027] The text generation module is used to generate subsequent text sentence by sentence based on the received story content and corresponding consequences, and continuously interacts with the knowledge construction module.

[0028] The beneficial effects of the method and system of the present invention are as follows: the present invention learns common sense knowledge through the knowledge construction module during training, and continuously provides knowledge to the text generation module during story generation, so that the generated story has the advantages of coherent expression and logically reasonable story content. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the steps of a method for generating short text stories based on a common sense knowledge base of the present invention;

[0030] Figure 2 This is a structural block diagram of a short text story generation system based on a common sense knowledge base of the present invention;

[0031] Figure 3 This is a flowchart of story generation according to a specific embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the architecture of the GPT-2 model in the knowledge building module of a specific embodiment of the present invention;

[0033] Figure 5 Schematic diagram of the architecture of the GPT-2 model in the text generation module of a specific embodiment of the present invention;

[0034] Figure 6 This is a mechanism diagram of a self-attention method (Enhanced Masked Self-Attention) that uses Attention Mask to avoid interference from irrelevant information in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are provided for ease of description only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted based on the understanding of those skilled in the art.

[0036] In the short text story generation, the goal of the present invention is to: suppose the final desired story is S = {s1, s2, ..., s n}, s i Represents the i-th sentence in S. Given the first sentence s1 of the existing story, it is necessary to generate the rest of the story content {s2,s3,…,s n}, and make the story content logically coherent.

[0037] Reference Figure 1 and Figure 3 The present invention provides a method for generating short text stories based on a common sense knowledge base, the method comprising the following steps:

[0038] S1. Build a common sense knowledge base;

[0039] Common sense knowledge base KG: contains many pieces of knowledge, KG = {k1, k2, ..., k l}, k i represents the i-th common sense knowledge in the common sense knowledge base KG, and there are k i ={context i ,event i ,relation i ,effect i}, represents common sense knowledge k i It consists of 4 parts (context i 、Event i , relation i and the consequences i ), and the specific meanings of these four parts are as follows:

[0040] Context: It describes the context in which the knowledge is applicable. It consists of zero, one, or several sentences. If it consists of zero sentences, it means that it is applicable in any scenario. Its text description starts with " <context> "Start with"< / context> "Finish.

[0041] Event: describes what happened in the context, consisting of a sentence, and its text description begins with " <event> "Start with"< / event> "Finish.

[0042] Relationship: Indicates the aspect from which the consequences of an event in the above context will be explained. It is divided into three aspects: "effect", "mental" and "intent". Its text description starts with " <relation> "Start with"< / relation> "Finish.

[0043] Consequences: This section describes the consequences of the event within the context of the preceding scenario. When the relationship is "effect," this section describes the direct consequences of the event. When the relationship is "mental," this section describes the consequences from a psychological perspective, such as the emotions and feelings of the participants. When the relationship is "intent," this section describes the intentions of the participants (or those affected by the event) after the event. For example, if the event is "PersonX feels sick" (PersonX represents a person) and the relationship is "intent," the consequence would be "PersonX wants to go to the hospital," indicating that PersonX feels sick and intends to go to the hospital.

[0044] S2. Construct a first training sample based on the common sense knowledge base and train the knowledge construction module to obtain a trained knowledge construction module;

[0045] Knowledge building module: It consists of a GPT-2 model that uses the Enhanced Masked Self-Attention mechanism and peripheral scripts (Note: The GPT-2 model in this module and the GPT-2 model in the text generation module do not share the same model, but each module has a GPT-2 model. The Enhanced Masked Self-Attention mechanism will be explained in detail below). During training, the knowledge k in the common sense knowledge base KG is used to generate the knowledge. i The context, events, relationships, and consequences of i ,event i ,relation i ,effect i ]) as the learning sample of the model. After the training is completed, it can be considered that the model has learned how to respond to the given context and event. i and specific relations i Under the conditions, infer the consequences of the event i In the story generation phase, the script receives the existing story content {s1, s2, s3, ..., s i-1}, and construct the three inputs of the model: and The context i =[" <context>",s1,s2,…,s i-2 ,"< / context> "], event i =[" <event> ",s i-1 ,"< / event> "]; and there is Input these three inputs into the GPT-2 model in sequence and get three text outputs in sequence as well as They represent the consequences of an event when the relationships are "effect", "mental" and "intent" respectively. It will be passed to the text generation module, and the model in the text generation module can understand the previous sentence of the story it generates based on this. i-1 What consequences and impacts have been caused to the development of the story, so as to better plan the next sentence to be generated i .

[0046] S2.1. Partially concatenate the context, events, relations, and consequences in the common sense knowledge base to obtain the first training sample;

[0047] S2.2. Taking the context, events, and relationships in the first training sample as input and the consequences in the first training sample as output, the knowledge construction module is trained to obtain a trained knowledge construction module.

[0048] S3. Collect the second training sample and train the text generation module to obtain a trained text generation module;

[0049] S3.1. Collect a second training sample, where the second training sample includes supplementary information of the existing story content and the knowledge module;

[0050] Specifically, stories are collected and input into the trained knowledge construction module, the consequence part of the knowledge construction module is obtained, and the story content and the consequence part are spliced ​​together to form the training samples of the text generation module.

[0051] S3.2. Using the second training sample as input and the next sentence of the existing story content as a label, train the text generation module to obtain a trained text generation module.

[0052] Text generation module: used to generate story content, consisting of a GPT-2 model and a peripheral script; the GPT-2 model is responsible for generating story content, and the peripheral script is responsible for interacting with the knowledge construction module. This module generates story content in an iterative manner, accepting the first sentence s1 of the story as the initial input, and generating the remaining stories {s2, s3, ..., s1} sentence by sentence. n}. In the process of sentence-by-sentence generation, this module will continuously interact with the knowledge construction module. For example, the text generation module has generated story content {s1, s2, s3, ..., s i-1}, now want to generate the next story text s i , the script passes the existing story content {s1,s2,s3,…,s i-1}, get knowledge related to the story content It represents the description of the subsequent plot development inferred by the knowledge building module based on the existing story content. These descriptions will be regarded as knowledge to guide the story sentence s i The generation of knowledge. After that, the script performs text splicing on the relevant content to generate the input of the model. i , the input of the model is Then there is Among them, supplement i Represents the supplementary knowledge received from the knowledge building module, which is a concatenated text with in" <supplement> "and"< / supplement> "Represents supplement i The beginning and end of <effect> "、"< / effect> "And so on. The model accepts As input, generate s i The model iterates this process until it generates the last sentence of the story s n .

[0053] Input recognition Figure 3 in It is mainly composed of the first i sentences of the story and the effect output by the knowledge construction module based on the first i sentences, forming a string of text. The output is the i+1th sentence of the story, and the label is the i+1th sentence in the story sample.

[0054] During model training, the training samples of the GPT-2 model in the text generation module are [s1,supplement2,s2,supplement3,s3,…,supplement n-1 ,s n-1 ,supplement n ,s n ], the training samples of the GPT-2 model in the knowledge construction module come from KG. After training, it can be considered that the model has learned the knowledge in KG and can infer the consequences of an event in a given context.

[0055] S4. Obtain the first sentence of the story, and iteratively generate a short text story of a preset length based on the trained knowledge construction module and the trained text generation module.

[0056] Existing: Common Sense Knowledge Base KG

[0057] Input: the first sentence of the story s1, the number of sentences in the final generated story n (n is an integer greater than 1)

[0058] Output: Generated story S = {s1,s2,…,s n}

[0059] S4.1 Record the current story content as X i , initialize X2 = [s1]; remember that the GPT-2 model in the text generation module generates s i The input is initialization Empty text; remember the GPT-2 model generation and s in the knowledge building module i The input of relevant knowledge is and Initialize i=2;

[0060] S4.2, let context i =[" <context>",s1,s2,…,s i-2 ,"< / context> "](If i is less than or equal to 2, context i for" <context>< / context> "), event i =[" <event> ",s i-1 ,"< / event> "],

[0061] S4.3. Construct the input of the GPT-2 model in the knowledge construction module by text splicing, that is, The three inputs are fed into the model in sequence, and three outputs are obtained respectively. as well as,

[0062] S4.4 constructs the input of the GPT-2 model in the text generation module by text splicing, Among them, supplement i It is composed of multiple texts. Feed the input into the model and get the model output s i .

[0063] S4.5. Update existing story content X i+1 =[X i ,s i ]

[0064] S4.6. If i is equal to n, then output {s1,s2,…,s n If not, then i:=i+1, and jump to step S4.2.

[0065] Further as a preferred embodiment of the present method, the knowledge construction module and the text generation module both include a GPT-2 model and a peripheral script.

[0066] Specifically, the GPT-2 model of the knowledge building module refers to Figure 4 , which is composed of 12 Decoder Blocks stacked together, each Decoder Block mainly consists of Layer Norm, Masked Self-Attention and Feed Forward. The structure of the GPT-2 model of the text generation module is referenced Figure 5 The difference between this model and the knowledge building module is that it adopts the Enhanced Masked Self-Attention mechanism. The knowledge building module and the text generation module interact through peripheral scripts.

[0067] As a further preferred embodiment of this method, it also includes:

[0068] In order to avoid irrelevant supplements in the input (i.e. {supplement2,supplement3,…,supplement i-1}) interferes with the model. The model uses an enhanced masked self-attention method (such as Attention Mask) to avoid interference from irrelevant information. Figure 6 shown), where represents the jth token in the i-th sentence of the story in the sample used to train the GPT-2 model in the text generation module. Indicates the supplement in the sample i The jth token in the graph is represented by the model input below and the model output above. The dotted line represents the model’s attention. When the model focuses on the valid information in the Input Invalid information is blocked This avoids the generation of Cause interference. It is necessary to use attention mask to shield irrelevant supplements when calculating self-attention. Specifically, there are:

[0069]

[0070]

[0071] H l =FeedForward(LayerNorm(R l ))+Rl ,l∈[1,L]

[0072] R l =O l +H l-1

[0073] O l =EnhancedMaskedSelfAttention(N l )

[0074] N l =LayerNorm(H l-1 )

[0075] H0=E+P

[0076] where u t Represents the token at position t in the output of GPT-2, u <t Represents all tokens at positions 1 to t-1 in the output of GPT-2; Represents the tensor at the t-th position in the output of the last layer (i.e., the L-th layer), which is a shape of [d k ] one-dimensional tensor; H l Represents the output of the lth layer (there are L layers), which is a [n s ,d k ] tensor, where n s is the length of the input sequence, d k Indicates the length of the feature latent vector. E is a tensor formed by concatenating the semantic embeddings corresponding to all tokens in the input text, which is also a [n s ,d k ] is a tensor that is a trainable parameter; P is a tensor formed by concatenating the positional embeddings corresponding to all tokens in the input sequence. Its shape is the same as E and it is also a trainable parameter. LayerNorm represents batch normalization, and FeedForward represents a two-layer fully connected feedforward network. l =EnhancedMaskedSelfAttention(N l ) is calculated as follows:

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] in, The shape is [d k ,d k ] are all trainable parameters; All are of shape [d k ] are all one-dimensional tensors, which are also trainable parameters; the obtained Q l , K l 、V l The shape is [n s ,d k ] tensor; the calculated α l is a shape of [n s ,n s ] represents the attention weight, but there is an attention interference problem at this time. Therefore, a matrix with the same shape of [n s ,n s ]’s Attention Mask matrix M mask , which is used to mask the supplement in the input text that is irrelevant to the sentence to be generated. It is defined as:

[0085]

[0086] like Figure 2 As shown in FIG, a short text story generation system based on a common sense knowledge base includes:

[0087] Common sense knowledge base, used to store common sense knowledge;

[0088] The knowledge building module is used to infer and output the corresponding consequences based on the received story content;

[0089] The text generation module is used to generate subsequent text sentence by sentence based on the received story content and corresponding consequences, and continuously interacts with the knowledge construction module.

[0090] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] A short text story generation device based on common sense knowledge base:

[0092] at least one processor;

[0093] at least one memory for storing at least one program;

[0094] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned method for generating short text stories based on a common sense knowledge base.

[0095] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0096] A storage medium storing processor-executable instructions, characterized in that the processor-executable instructions, when executed by the processor, are used to implement the above-mentioned method for generating short text stories based on a common sense knowledge base.

[0097] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0098] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A short text story generation method based on common sense knowledge base, characterized in that: The following steps are involved: Build a common sense knowledge base; constructing a first training sample based on the common sense knowledge base and training the knowledge construction module to obtain a trained knowledge construction module; Collecting a second training sample and training the text generation module to obtain a trained text generation module; Obtain the first sentence of the story and iteratively generate a short text story of a preset length based on the trained knowledge construction module and the trained text generation module; Each piece of common sense in the common sense knowledge base includes context, events, relationships and consequences; The step of constructing a first training sample based on the common sense knowledge base and training the knowledge construction module to obtain a trained knowledge construction module specifically includes: Partially splicing the context, events, relations, and consequences in the common sense knowledge base to obtain the first training sample; Taking the context, events, and relationships in the first training sample as input and the consequences in the first training sample as labels, the knowledge building module is trained to obtain a trained knowledge building module; The step of obtaining the first sentence of the story and iteratively generating a short text story of a preset length based on the trained knowledge construction module and the trained text generation module specifically includes: Get the first sentence of the story and set the number of sentences in the final short text story; Based on the common sense knowledge base, the first sentence of the story and the current story content are combined to perform text splicing to obtain the first input; Send the first input to the trained knowledge building module to generate the corresponding consequence output; Based on the common sense knowledge base, the first sentence of the story, the current story content and the corresponding consequence output are combined to perform text splicing to obtain the second input; Feed the second input into the trained text generation module to update the story content; The working steps of the knowledge construction module and the text generation module are repeated until the number of sentences in the story content reaches a preset value; Introduce the matrix, which is defined as:

2. The method for generating short text stories based on a common sense knowledge base according to claim 1, characterized in that: The step of collecting the second training sample and training the text generation module to obtain a trained text generation module specifically includes: collecting a second training sample, wherein the second training sample includes supplementary information of the existing story content and the knowledge module; The second training sample is used as input and the next sentence of the existing story content is used as a label to train the text generation module to obtain a trained text generation module.

3. The method for generating short text stories based on a common sense knowledge base according to claim 2, characterized in that: The knowledge construction module and the text generation module both include a GPT-2 model and peripheral scripts.

4. A short text story generation system based on common sense knowledge base, characterized by: The method for generating short text stories based on a common sense knowledge base according to claim 1 comprises: Common sense knowledge base, used to store common sense knowledge; The knowledge building module is used to infer and output the corresponding consequences based on the received story content; The text generation module is used to generate subsequent text sentence by sentence based on the received story content and corresponding consequences, and continuously interacts with the knowledge construction module.

Images