A story generation method and system based on external knowledge enhancement

Abstract

The application discloses a story generation method and system based on external knowledge enhancement. The method is as follows: a knowledge base is constructed, including a knowledge set in a target field and a general knowledge set, each piece of knowledge in the knowledge base is a real story text; each piece of knowledge is divided into two parts: an index part and an extension part; a knowledge retrieval module retrieves matched indexes from the knowledge base according to an input prompt x, takes the extension part corresponding to each matched index as a candidate knowledge, and selects several keywords as external knowledge from each candidate knowledge; a knowledge selection module predicts a score for each selected keyword according to the input x and the candidate knowledge, and then selects knowledge for story generation according to the predicted scores of the keywords; a story generation module generates a story text y according to the selected knowledge for story generation; and then x and y are spliced into a coherent story.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a story generation method and system based on external knowledge enhancement. This method, targeting the characteristics of generating stories from long texts, utilizes a large-scale pre-trained language model. Based on an external knowledge base constructed from the story text, it introduces coarse-grained knowledge retrieval and fine-grained knowledge selection to enhance the model's story creation capabilities, aiming to generate high-quality story content. Background Technology

[0002] In the field of natural language processing, text generation refers to the generation of text for specific purposes using specific algorithms. Current research subfields mainly include dialogue generation, text summarization, question-answering generation, and story creation. Research on text generation technology has significant practical and theoretical implications. In daily life and work, text generation technology can assist humans in completing related tasks and improve work efficiency, such as intelligent customer service, machine translation, and assisted writing. In research, text generation technology aims to achieve the level of human manipulation of language and is a key technology for realizing higher stages of artificial intelligence. Therefore, text generation has always been a popular research task. This invention focuses on the story creation task within the field of text generation. Addressing the characteristics and challenges of story creation, it proposes an improved method based on existing story creation methods to enhance the quality of created stories.

[0003] This invention addresses the problem of story creation based on short text prompts, a task belonging to the field of long text generation. Compared to other types of text generation tasks, such as dialogue generation from short input to short output and summary generation from long input to short output, story creation is a short input to long output task, requiring the story creation model to have the ability to "associate." Generating a relatively long story from a small number of prompts requires the model to connect the prompts to relevant content. Sequence-to-sequence models naturally have the ability to learn mappings, but due to limitations in data and model structure, their actual performance is poor. Existing knowledge-integrated story generation methods incorporate external knowledge into the model implicitly or explicitly. This invention addresses the problems of current external knowledge-based story generation methods by designing a three-stage external knowledge-enhanced story generation method that includes knowledge retrieval, knowledge selection, and text generation. Summary of the Invention

[0004] The purpose of this invention is to provide a story generation method and system based on external knowledge enhancement. By constructing a suitable external knowledge base, the story creation model can retrieve and select knowledge that is beneficial to creation from the external knowledge base, thereby improving the quality of story creation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A story generation method based on external knowledge enhancement includes the following steps:

[0007] Construct a knowledge base The knowledge base Includes knowledge sets within the target domain and a general knowledge set The knowledge base Each piece of knowledge is a real-life story text; each piece of knowledge is divided into two parts: an index and an extension.

[0008] The knowledge retrieval module retrieves information based on the input prompts x = (x1, x2, ..., x...). n From the knowledge base The system retrieves matching indexes, uses the extended portion corresponding to each matching index as candidate knowledge, and selects several keywords from these candidate knowledge sources as external knowledge; x n This represents the nth word in the prompt;

[0009] The knowledge selection module predicts a score for each selected keyword based on the input x and the candidate knowledge, and then selects the knowledge for story generation based on the predicted scores of each keyword;

[0010] The story generation module generates story text y = (y1, y2, ..., y3) based on the selected knowledge used for story generation. m );y m x represents the m-th word in the story text; then x and y are concatenated to form a coherent story.

[0011] Furthermore, the knowledge retrieval module employs a cascaded graph search method to retrieve information from the knowledge base based on the input prompt x. The system retrieves N matching indices; then, it merges and removes duplicates from the extended portions corresponding to these N indices to obtain a keyword set. As external knowledge.

[0012] Furthermore, the knowledge selection module selects each keyword in the prompt x and external knowledge. Each keyword in the dataset is encoded separately to obtain the embedding representation corresponding to each keyword. This embedding representation is then fed into a feedforward neural network to obtain external knowledge. The correlation coefficient between each keyword and the prompt x is used as external knowledge. The predicted score of the corresponding keyword.

[0013] Furthermore, the method for training the knowledge selection module is as follows: for a given prompt x', from the knowledge base... Retrieved external knowledge The corresponding standard story is y′; based on external knowledge Each keyword k i The correlation with the standard story y′, and the external knowledge Keywords are categorized and assigned different relevance values ​​as corresponding keyword k. i pseudo-tag q i Different weights are assigned to different types of keywords; the loss function for training the knowledge selection module is... Among them, c i For keyword k i The predicted relevance values ​​are given by u1, u2, and u3, which are the weights of keywords with strong relevance, moderate relevance, and no relevance, respectively. R1, R2, and R3 are the sets of keywords with strong relevance, moderate relevance, and no relevance, respectively.

[0014] Furthermore, the loss function used to train the story generation module is a negative log-likelihood function. Where p(y i ) represents the probability of word yi in the vocabulary probability distribution output in the current step, and l represents external knowledge. Total number of keywords.

[0015] Furthermore, the story generation module is a sequence-to-sequence BART model based on a multi-head self-attention mechanism; the story generation module combines the input cue x and external knowledge. Unified as w1, w2, ..., w t It is then input into a bidirectional encoder based on a multi-head self-attention mechanism for encoding; where t is the cue x and The total number of keywords in the sequence, w t Let y be the t-th keyword; during the decoding phase, when predicting the next word y... i At this time, the decoder uses the previously decoded word and the encoder's output to predict the next word y. i .

[0016] Furthermore, the story generation module is a sequence-to-sequence BART model based on a multi-head self-attention mechanism, and a correlation coefficient vector is introduced into the sequence-to-sequence BART model based on a multi-head self-attention mechanism. Where c1~c l Representing external knowledge k1~k l The relevant values; Add to The adjusted vector is obtained above. use Replace the result vector calculated by the attention mechanism in the encoder and decoder of the story generation module. The attention mechanism calculates the query vector v and each keyword w. i The embedding representation h i The inner product yields the result vector.

[0017] A story generation system based on external knowledge enhancement, characterized by including:

[0018] knowledge base The knowledge base Includes knowledge sets within the target domain and a general knowledge set The knowledge base Each piece of knowledge is a real-life story text; each piece of knowledge is divided into two parts: an index and an extension.

[0019] The knowledge retrieval module is used to retrieve information based on the input prompt x = (x1, x2, ..., x...). n From the knowledge base The system retrieves matching indexes, uses the extended portion corresponding to each matching index as candidate knowledge, and selects several keywords from these candidate knowledge sources as external knowledge; x n This represents the nth word in the prompt;

[0020] The knowledge selection module is used to predict a score for each selected keyword based on the input x and the candidate knowledge, and then select the knowledge for story generation based on the predicted scores of each keyword.

[0021] The story generation module is used to generate story text y = (y1, y2, ..., y3) based on the selected knowledge used for story generation. m );y m x represents the m-th word in the story text; then x and y are concatenated to form a coherent story.

[0022] The story generation method of the present invention includes the following:

[0023] An external knowledge base is constructed. Considering the knowledge-driven nature of story creation, suitable external knowledge is built. This knowledge is divided into two categories: domain-specific knowledge and general external knowledge. Each piece of knowledge is a narrative text, containing knowledge about human story writing. Furthermore, each piece of knowledge is divided into two parts: an index sentence for retrieval and a set of extended keywords serving as external knowledge. To address the utilization of this knowledge, this invention employs semantic retrieval to retrieve suitable knowledge for the story creation system.

[0024] The semantic retrieval system comprises an encoder for index sentences in the knowledge base and an encoder for the current input, which serves as a story creation prompt. It is responsible for mapping the text to a semantic space in the vector dimension. Using inner product similarity, a measure of vector similarity, it retrieves index sentences that are closest to the current input. Then, the expanded keyword portion of the retrieved knowledge is used as external knowledge for the story creation system to utilize. The encoder of the semantic retrieval system consists of a bidirectional encoder based on the Transformer model, which, after pre-training, possesses superior semantic encoding capabilities.

[0025] A knowledge selection module, also based on the Transformer model, is constructed to select and filter retrieved knowledge, providing more accurate knowledge for story creation. For each keyword in the retrieved knowledge, the knowledge selection module predicts a score based on the input prompts, indicating its importance. Keywords with lower importance are eliminated, thus achieving the purpose of knowledge selection.

[0026] The training knowledge selection module includes two training methods: direct training using pseudo-supervisory signals and joint training with the subsequent story generation model. First, pseudo-labels for knowledge are constructed. Since standard knowledge is generally difficult to obtain, this invention adopts a heuristic approach to automatically label the relevance of knowledge.

[0027] This invention categorizes keywords in knowledge into three types: strongly relevant, moderately relevant, and irrelevant. The degree of relevance is based on the semantic similarity between the knowledge and the text in the standard story. Keywords appearing in the standard story, or synonyms of a word in the standard story, are designated as strongly relevant knowledge; keywords that are near-terminal (hyper-terminal) words of a word in the standard story are considered moderately relevant knowledge; the remaining keywords are considered irrelevant knowledge. The pseudo-labels for strongly relevant, moderately relevant, and irrelevant knowledge are set to 1, 0.5, and 0, respectively. Based on the pseudo-labels and the keyword scores predicted by the knowledge selection module, a loss function for knowledge selection is calculated using mean squared error. Because the pseudo-labels of knowledge are imbalanced, different weights are assigned to the loss function according to the proportion of each knowledge category to suppress class imbalance during training.

[0028] To further utilize standard stories to train the knowledge selection model, this invention designs a joint training method, with another module for joint training being a story generation model. The input to the story generation model is the current prompt, keywords as external knowledge, and the relevance coefficients of the keywords. The story generation model consists of an encoder and a decoder, both based on the Transformer model. The encoder encodes the text concatenated from the prompt and external knowledge. In the self-attention operation within the Transformer model, the attention weights of the external knowledge are influenced by the relevance coefficients. Similarly, at each decoding step of the story generation model's decoder, an attention operation is performed on the encoder's output, and the attention weights of the knowledge are also influenced by the relevance coefficients. The loss function of the story generation model is obtained by calculating the cross-entropy function between the predicted story and the standard story. By training on this loss function, the parameters of the knowledge selection model are further updated.

[0029] The Transformer model incorporates an attention mechanism. The specific operation of influencing attention weights with the relevance coefficient is as follows: In traditional self-attention mechanisms, given a query vector, the attention mechanism calculates its inner product with the representation of each input word, resulting in a vector denoted as V. This vector undergoes a scaled softmax operation to obtain the attention weights. To introduce the relevance coefficient, this invention constructs a relevance coefficient vector. For the current input part serving as a prompt, the value is 0; for the knowledge part, the value is the logarithm of the relevance coefficient predicted by the knowledge selection module. The relevance coefficient vector is added bitwise to the aforementioned result vector to obtain an adjusted vector, which replaces the original result vector in all self-attention operations in the encoder and decoder of the story generation module.

[0030] To enhance the story generation model's ability to utilize common sense, this invention proposes a pre-training task incorporating common sense knowledge. By pre-training the story generation model on additionally constructed common sense text, common sense knowledge is implicitly injected into the model. This invention constructs two types of common sense text: one based on entity knowledge from a knowledge graph, and the other on semantic knowledge from WordNet. Through template construction, this knowledge is transformed into natural language text. Similar to BART's method of constructing pre-training samples, this invention uses a masking approach, replacing a word in the common sense text with a mask, allowing the story generation model to recover the original text from the masked text during the pre-training phase. For the story generation model, further fine-tuning training is performed on a story dataset using standard relevant knowledge.

[0031] Compared with the prior art, the positive effects of the present invention are as follows:

[0032] This invention addresses the knowledge-driven nature of story creation tasks by designing a story-based external knowledge framework suitable for story creation. It utilizes vector-based nearest neighbor retrieval to find semantically similar knowledge, enhancing the story generation model's performance. To eliminate the influence of irrelevant knowledge, a knowledge selection module is introduced. This module predicts the scores of each keyword within the current input and external knowledge, selecting highly relevant keywords based on their scores. The knowledge selection model is trained using pseudo-supervised signals and through joint training with the story generation model. Furthermore, the model is pre-trained using constructed common-sense text to enhance its ability to utilize common-sense knowledge. Experiments on publicly available story datasets validate the effectiveness of the proposed external knowledge-enhanced story creation method and system. This invention improves the readability, consistency, and diversity of created stories. Attached Figure Description

[0033] Figure 1 Flowchart for a story creation method based on external knowledge enhancement.

[0034] Figure 2 This is an example illustrating the association between hyponyms, hypernyms, and synonyms in Wordnet;

[0035] (a) is a hyponym of Wordnet, (b) is a hypernym of Wordnet, and (c) is a synonym of Wordnet. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the described embodiments are only intended to facilitate the understanding of the present invention and do not limit it in any way.

[0037] For ease of description, this application first defines the task of knowledge-based story generation. Given a cue representing the input x = (x1, x2, ..., x...), ... n A knowledge base The goal of the task is to generate subsequent story text y = (y1, y2, ..., y3) m (), where m and n represent the lengths of the prompts and story text, respectively. x and y are concatenated to form a coherent story. The model can... The method retrieves relevant knowledge to assist in story generation. In this invention, the method targets a knowledge base. The invention describes the construction of the knowledge base and how to utilize its knowledge. The proposed method was validated on the publicly available short story dataset ROCStories. This dataset contains five-sentence stories, with the first sentence as input and the following four sentences as the target story.

[0038] For example, Figure 1 This is a schematic diagram of a story creation method based on external knowledge enhancement. The model consists of three main parts, from left to right:

[0039] • Knowledge Retrieval: Based on the current prompt x, retrieve information from the knowledge base. The system retrieves matching indexes and uses the corresponding keywords as the retrieved knowledge.

[0040] • Knowledge Selection: This module takes as input the original input x and the knowledge retrieved during the knowledge retrieval stage. It is responsible for selecting knowledge by calculating a score for each keyword through the model. This score is used for knowledge selection and the weighting of knowledge utilization by the subsequent story generation module.

[0041] • Story Generation: This module takes the original input x and the knowledge retrieved during the knowledge retrieval stage as input, as well as the score predicted by the knowledge selection module. The output is the final story. The following sections detail the method, including the construction of the knowledge base, the various parts of the model, and training and testing.

[0042] Knowledge base construction:

[0043] Previous knowledge-based story generation methods used template-based knowledge constructed from knowledge graphs as an external knowledge base. This invention, however, utilizes real-world story text as the external knowledge base. This is because real-world story text is more relevant and therefore more suitable for the task of story generation. Specifically, two types of knowledge bases are constructed: and These represent knowledge within the domain and knowledge outside the domain, respectively. The training set portion, built upon the ROCStories dataset, can be considered as domain-specific knowledge. Built upon publicly available story datasets, it is considered external common-sense knowledge. and The collection is Each piece of knowledge in the process is essentially a short story before it is processed. For The knowledge here corresponds to a sample in the training set of the ROCStories dataset. For To construct short stories similar to those in ROCStories, this invention first selects narrative articles from the Children's Book Test dataset (http: / / fb.ai / babi / ), and then breaks down the articles into several blocks. Specifically, this invention divides the articles into blocks of 5 sentences each to match the 5-sentence stories in ROCStories. For each piece of knowledge, this invention uses its first sentence as the index part and subsequent sentences as the extended part. The index part will be used to calculate similarity with the prompt x in the subsequent knowledge retrieval stage, while the extended part will serve as candidate knowledge. For the extended part, this invention further extracts non-stop words (verbs and nouns) as keywords. After these processes, Each piece of knowledge in the database contains a sentence that serves as an index, denoted as s. index And a series of keywords as extensions. It should be noted that the number of keywords contained in each piece of knowledge is not necessarily the same.

[0044] Knowledge retrieval module:

[0045] Build a good knowledge base Subsequently, when generating stories, the model of this invention can select suitable stories as reference knowledge. Given an input prompt x, the model's knowledge retrieval module first... The system retrieves several stories with similar context (i.e., the index portion of the knowledge) as candidate knowledge. This invention uses a dual-encoder structure to compute the encoded representations of x and the index portion, mapping them to the same embedding representation space:

[0046] d(x) = BERT1(x)

[0047] d(z) = BERT2(z)

[0048] Where z represents the index portion s of a knowledge entry. index d(·) represents the embedding representation of ·, and BERT1 and BERT2 are two BERT-based encoders. The present invention then calculates the similarity between x and z:

[0049] s(x, z) = <d(x), d(z)>

[0050] <a, b> represents the inner product of vectors a and b. This invention selects the N most similar pieces of knowledge, uses the extended parts of these N pieces of knowledge, merges and removes duplicates, resulting in a keyword set. This will serve as external knowledge for subsequent models.

[0051] Calculating the top N similarity values ​​is a Maximum Inner Product Search (MIPS) problem with a time complexity of O(Mh). 2 +NlogM), where M is the number of knowledge items in the knowledge base, and h is the dimension of the embedding vector. Because the knowledge base Containing millions of pieces of knowledge, accurately retrieving the top N similarity items is extremely time-consuming. Therefore, to accelerate the retrieval process, this invention employs an approximate retrieval method. Specifically, this invention uses cascaded graph search from Facebook's Faiss vector retrieval library (see https: / / faiss.ai / cpp_api / struct / structfaiss_1_1IndexHNSWFlat.html). This approximate retrieval method reduces the retrieval time from linear to sublinear complexity. Although its retrieval strategy is approximate, it still achieves good retrieval results. In experiments, this invention uses a pre-trained dual encoder as the parameters of its dual encoder to improve retrieval performance.

[0052] Knowledge Selection Module:

[0053] After retrieving relevant knowledge, one direct approach is to concatenate it with the original clues as input to a sequence-to-sequence model. However, not all retrieved knowledge is relevant, and their importance in generating the story varies. While sequence-to-sequence models can implicitly learn the importance of different parts of knowledge, explicit model design and additional knowledge selection signals can significantly improve the generation results. Therefore, this invention designs a knowledge selection module whose function is to predict each keyword k. i Importance score. A diagram of the knowledge selection module is shown below. Figure 1 The middle part is shown.

[0054] The core structure of the knowledge selection module is a BERT-based encoder. The module uses prompts (x) and external knowledge as input. This is the input. To enable the model to distinguish between cues and knowledge, this invention adds a special [SEP] symbol at the end of the cues to separate them. Furthermore, different segmentation IDs and position IDs are used for them. For words in the cues, the segmentation ID is 0; for words in the knowledge, the segmentation ID is 1. The position ID is as shown in p in the model diagram. i As shown, p i=i. That is, the position numbers of words in the prompts start from 0 and increment, while the position numbers of words in the knowledge base are fixed to a large number N (in the experiments on the ROCStories dataset, this invention sets N to 400, which ensures that N does not conflict with the position numbers of words in the prompts). The reason for using fixed position numbers for words in the knowledge base is that there is no relative order relationship between them. After BERT calculation, an embedding representation is obtained for each keyword (since a single keyword may be segmented into multiple sub-words, the word embedding representations of the sub-words are averaged and pooled to obtain the embedding representation of the keyword during input. The same processing method is used in the story generation module). These embedding representations are then processed by a two-layer feedforward neural network. For the i-th keyword in the knowledge base, a number c can be obtained. i (Values ​​range from 0 to 1). This invention refers to these numbers as Relevance Coefficients, which serve as the basis for whether knowledge is selected, as well as for knowledge selection and the self-attention mechanism in the subsequent story generation module.

[0055] By using correlation coefficients, this invention connects the knowledge selection module and the story generation module. In the story generation module, the loss function is calculated between the predicted story and the standard story. When calculating the backpropagation gradient, the correlation coefficient propagates the gradient to the knowledge selection module, allowing it to receive supervision signals from the standard story and thus update its parameters. To obtain a better knowledge selection module, this invention proposes using more supervision signals to train it. Due to the lack of standard knowledge, i.e., the absence of standard knowledge labels, this invention constructs pseudo-labels for each keyword of external knowledge. Given a cue x, the standard story y = (y1, y2, ..., y...). m External knowledge retrieved This invention is based on each keyword k i Based on their relevance to the standard story y, these stories are categorized and assigned different relevance values ​​as corresponding keywords k. i pseudo-tag q i :

[0056] • Strongly correlated: The keyword is a certain y i Or it could be a synonym. The relevance value for this type of keyword is 1.

[0057] • Moderate relevance: Keywords and a certain y i They have similar meanings. Similarity is defined as follows: if the distance between two words in the hypernym-hyponym tree (see explanation below) is less than or equal to 2, then their meanings are similar. The relevance value for this type of keyword is 0.5.

[0058] • Irrelevant: Keywords that do not belong to the above two categories have a relevance value of 0.

[0059] To construct pseudo-tags, this invention requires information on synonyms, hypernyms, and hyponyms. This invention uses the interface provided by Wordnet (https: / / wordnet.princeton.edu / ) (the interface is provided by the NLTK toolkit, see http: / / www.nltk.org for details) to obtain this information. Figure 2 This section presents examples of hyponyms, hypernyms, and synonyms from WordNet. A hyponym of a word is a more concrete expression of that word. For example, *enjoy* means "to enjoy, to appreciate," while *devour* can mean "to devour something ravenously, to watch something with relish." Therefore, *devour* can be seen as a more concrete expression of *enjoy*, and thus can be considered a hyponym of *enjoy*. Conversely, *enjoy* is a hypernym of *devour*, a more abstract expression of *devour*. A word can have multiple hyponyms and multiple hypernyms. By understanding the hyponym-hypernym relationships between words, a hypernym-hypernym tree can be obtained.

[0060] After obtaining the pseudo-label, record k. i The pseudo tag is q i This invention can calculate c i and q i The mean squared error between the three categories is used as the loss function for the knowledge selection module. Because the proportions of keywords in the three categories are unbalanced, to prevent the model from favoring the category with the most predicted keywords, this invention assigns different weights to keywords in different categories:

[0061]

[0062] u1, u2, and u3 are the weight coefficients for the categories of strongly relevant, moderately relevant, and irrelevant, respectively, and R1, R2, and R3 are the sets of keywords for the categories of strongly relevant, moderately relevant, and irrelevant, respectively. Through... Training is performed on the platform, and the parameters of the knowledge selection module can be updated directly under the supervision signal of the pseudo-label.

[0063] Story generation module:

[0064] The story generation module includes parts such as Figure 1 As shown in the rightmost part, this invention uses a sequence-to-sequence BART model based on a multi-head self-attention mechanism as its basic framework. The input to this module is a cue x and external knowledge. They can be uniformly represented as:

[0065] w1, w2, ..., w t

[0066] Where t is the hint x and The number of words in the sequence, where the first n words are (x1, x2, ..., xn). n ), which is the word in the prompt, followed by {k1, k2, ..., k l The part containing the '}' represents the knowledge portion. The entire input is encoded by a bidirectional encoder based on a multi-head self-attention mechanism. During the decoding phase, when predicting the next word 'y'... i At this time, the decoder will use the previously decoded words and the encoder's output. Similar to the position numbering settings in the knowledge selection module, in the story generation module... The position number of each word in the Chinese text is M.

[0067] The structure of the story generation module is the same as that of a regular BART module. The only difference is that this invention introduces a correlation coefficient c into BART. i These coefficients are involved in the computation in both the encoder and decoder. In traditional self-attention mechanisms, given a query vector v, the attention mechanism computes v and each w. i The embedding representation h i The inner product of and is denoted as . This vector will undergo a scaling softmax operation to obtain the attention weights. To introduce c... i This invention constructs a correlation coefficient vector. Its content is as follows:

[0068]

[0069] Where c1~c l Representing external knowledge k1~k l The correlation coefficient. The present invention will then... Add to The above yields an adjusted vector.

[0070]

[0071] Here, + represents element-wise addition of vectors. It will replace all self-attention operations in the encoder and decoder of the story generation module.

[0072] For the loss function of the story generation module, this invention adopts a negative log-likelihood function:

[0073]

[0074] Where p(y i ) represents the probability distribution of word y in the vocabulary output at the current step. i The probability of.

[0075] To enhance the story generation model's ability to utilize knowledge, this invention additionally trains the story generation module using common sense and semantic knowledge, implicitly endowing the model with common sense reasoning capabilities. Therefore, this invention constructs a training dataset of common sense and semantic knowledge for pre-training the story generation model.

[0076] This invention uses two types of knowledge to pre-train the story generation model. First, it employs triples from the ConceptNet knowledge graph, using templates to transform these triples into text. For example, (eiffel tower, AtLocation, paris) becomes "eiffel tower is at paris," and (telephone, UsedFor, communication) becomes "telephone is used for communication." This knowledge serves as common sense. Second, it converts hypernyms, hyponyms, and synonyms from WordNet into text sentences. For instance, if "devour" is a hyponym of "enjoy," the invention constructs the sentence "enjoy contains devour." For "follow" and "adopt," it constructs the sentence "follow equals adopt." This invention targets all keywords in the ROCStories dataset, finding their hypernyms, hyponyms, and synonyms using WordNet, and constructs text sentences using the above method. This knowledge serves as semantic knowledge. There are 1,174,267 common sense text entries and 315,950 semantic knowledge text entries. To construct training data suitable for BART, this invention replaces words in the knowledge text with [MASK] to obtain masked sentences. The masked sentences are then used as input for the story generation model to predict the original sentences. Specifically, during masking, a single word in the text is randomly selected for masking. Notably, for common sense text, stop words and punctuation are not considered during masking.

[0077] Training and testing:

[0078] During training, this invention first initializes the parameters of the knowledge selection and story generation modules using pre-trained BERT and BART (using a pre-trained model provided by huggingface (https: / / huggingface.co / )). For the knowledge selection model, this invention first uses a loss function... The knowledge selection module is trained separately for several rounds until the loss function converges, thereby improving its knowledge selection capabilities. Subsequently, this invention uses a story generation model as an auxiliary training tool, considering... and The combined loss function is a weighted sum of the two loss functions, which is used to further train the knowledge selection model. For the story selection model (note that this model is not the same as the story generation model mentioned above in the knowledge selection model), this invention first pre-trains it using common sense and word meaning knowledge. Then, using external knowledge and prompts as input and stories as output, the story generation model is trained. Here, only strongly relevant and moderately relevant knowledge is used as external knowledge.

[0079] During testing, this invention first retrieves relevant knowledge based on prompts. The knowledge selection module then assesses the relevance of the knowledge. For keyword k... i The predicted correlation coefficient is c. i This invention defines the predicted correlation value p. i for:

[0080]

[0081] That is, according to c i The value assigned to the keyword k i The nearest label is selected. Knowledge with a predicted value of 0 is then removed, retaining only those with predicted values ​​of 0.5 and 1. The hints, retained knowledge, and relevant values ​​are then input into the story generation model, which uses this model to predict the story.

[0082] In its implementation, the basic models of this invention, including GPT-2, BART, and BERT, adopt the base versions defined in Huggingface (GPT2: https: / / huggingface.co / gpt2 / resolve / main / config.json, BART: https: / / huggingface.co / facebook / bart-base / resolve / main / config.json, BERT: https: / / huggingface.co / bert-base-uncased / blob / main / config.json). During the knowledge retrieval stage, this invention selects the five knowledge items with the highest similarity. To determine the knowledge selection loss function... The weights u1 to u3 for each category are calculated based on the proportions of each category in all retrieved knowledge. The proportions of strongly relevant, moderately relevant, and irrelevant knowledge are approximately 2%, 18%, and 80%, respectively. Since irrelevant knowledge accounts for a larger proportion and strongly relevant knowledge accounts for a smaller proportion, this invention sets u1 > u2 > u3. Specifically, in the experiment, the values ​​of u1 to u3 are 10, 5, and 1, respectively. This invention first trains the knowledge selection module, and then trains the entire model. The training stops when the loss function value converges on the validation set. The comprehensive loss function... The α value is set to 0.5. The batch size is 8. This invention uses the AdamW optimizer to update the parameters. For the learning rate and its update strategy, this invention uses a Warm-Up update strategy, where the learning rate linearly increases from 0 to 1.5e-4 in the first 2000 updates. Then, in the remaining training steps, it gradually decreases linearly to 0.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A story generation method based on external knowledge enhancement, comprising the following steps: Construct a knowledge base The knowledge base Includes knowledge sets within the target domain and a general knowledge set The knowledge base Each piece of knowledge is based on a real story text; Each piece of knowledge is divided into two parts: an index and an extension. The knowledge retrieval module uses input prompts... From the knowledge base The matching index is retrieved, and the extended part corresponding to each matching index is taken as candidate knowledge. Several keywords are selected from each candidate knowledge as external knowledge. This represents the nth word in the prompt; The knowledge selection module selects based on the input. The candidate knowledge is used to predict a score for each selected keyword, and then the knowledge used for story generation is selected based on the predicted scores of each keyword. The method for training the knowledge selection module is as follows: for a given prompt From the knowledge base Retrieved external knowledge The corresponding standard story is Based on external knowledge Each keyword and standard story The relevance of external knowledge Keywords are categorized and assigned different relevance values ​​as corresponding keywords. pseudo-tags And set different weights for different types of keywords; The loss function for training the knowledge selection module is: ;in, Keywords The predicted correlation value, , , These are the weights for keywords with a strongly relevant category, keywords with a moderately relevant category, and keywords with an irrelevant category, respectively. , , These are sets of keywords categorized as strongly relevant, moderately relevant, and irrelevant. The story generation module generates story text based on the selected knowledge used for story generation. ; Represents the m-th word in the story text; then and y Piecing them together into a coherent story.

2. The method according to claim 1, characterized in that, The knowledge retrieval module uses a cascaded graph search method based on the input prompts. From the knowledge base The search results found the matching results. N One index; then the index The extended parts corresponding to the index are merged and deduplicated to obtain a set of keywords. As external knowledge.

3. The method according to claim 2, characterized in that, The knowledge selection module provides prompts. Each keyword in the text, external knowledge Each keyword in the dataset is encoded separately to obtain the embedding representation corresponding to each keyword. This embedding representation is then fed into a feedforward neural network to obtain external knowledge. Each keyword and hint The correlation coefficient between them, as external knowledge The predicted score of the corresponding keyword.

4. The method according to claim 1, characterized in that, The loss function used to train the story generation module is a negative log-likelihood function. ;in, This represents the probability distribution of the vocabulary output in the current step. The probability, l For external knowledge Total number of keywords.

5. The method according to claim 1, characterized in that, The story generation module is a sequence-to-sequence BART model based on a multi-head self-attention mechanism; the story generation module will process the input prompts. and external knowledge Unified representation as And input it into a bidirectional encoder based on a multi-head self-attention mechanism for encoding; where It's a hint and The total number of keywords in the sequence. Let t be the keyword; during the decoding phase, when predicting the next word... At this point, the decoder uses the previously decoded word and the encoder's output to predict the next word. .

6. The method according to claim 5, characterized in that, The story generation module is a sequence-to-sequence BART model based on a multi-head self-attention mechanism, and a correlation coefficient vector is introduced into the sequence-to-sequence BART model based on the multi-head self-attention mechanism. ;in ~ Representing external knowledge ~ The relevant values; Add to The adjusted vector is obtained above. ;use Replace the result vector calculated by the attention mechanism in the encoder and decoder of the story generation module. The attention mechanism calculates the query vector. and each keyword Embedded representation The inner product yields the result vector. .

7. A story generation system based on external knowledge enhancement, characterized in that, include knowledge base The knowledge base Includes knowledge sets within the target domain and a general knowledge set The knowledge base Each piece of knowledge is a real-life story text; each piece of knowledge is divided into two parts: an index and an extension. The knowledge retrieval module is used to retrieve information based on input prompts. From the knowledge base The matching index is retrieved, and the extended part corresponding to each matching index is taken as candidate knowledge. Several keywords are selected from each candidate knowledge as external knowledge. This represents the nth word in the prompt; The knowledge selection module is used to select knowledge based on the input. The candidate knowledge is used to predict a score for each selected keyword, and then the knowledge used for story generation is selected based on the predicted scores of each keyword. The method for training the knowledge selection module is as follows: for a given prompt From the knowledge base Retrieved external knowledge The corresponding standard story is Based on external knowledge Each keyword and standard story The relevance of external knowledge Keywords are categorized and assigned different relevance values ​​as corresponding keywords. pseudo-tags And set different weights for different types of keywords; The loss function for training the knowledge selection module is: ;in, Keywords The predicted correlation value, , , These are the weights for keywords with a strongly relevant category, keywords with a moderately relevant category, and keywords with an irrelevant category, respectively. , , These are sets of keywords categorized as strongly relevant, moderately relevant, and irrelevant. The story generation module is used to generate story text based on the selected knowledge used for story generation. ; Represents the m-th word in the story text; then and Piecing them together into a coherent story.

8. The system according to claim 7, characterized in that, The knowledge selection module provides prompts. Each keyword in the text, external knowledge Each keyword in the dataset is encoded separately to obtain the embedding representation corresponding to each keyword. This embedding representation is then fed into a feedforward neural network to obtain external knowledge. Each keyword and hint The correlation coefficient between them, as external knowledge The predicted score of the corresponding keyword.

9. The system according to claim 8, characterized in that, The loss function used to train the story generation module is a negative log-likelihood function. ;in, This represents the probability distribution of the vocabulary output in the current step. The probability, l For external knowledge Total number of keywords.

Images