Text generation method, device, host device, system, program product and medium

Abstract

The present invention provides a text generation method, device, host device, system, program product and medium, which relate to the field of artificial intelligence. The method comprises: obtaining input text; the input text comprises a content summary text and a word reference text, and the word reference text comprises a specified word; using a pre-trained language model to perform a text generation operation according to the input text, and obtaining a logic matrix corresponding to a word in a generated text; the pre-trained language model has a word list comprising candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate word, and the selection probability is the probability that the candidate word is selected as a word in the generated text; using a neural network layer in a pre-trained probability transfer model to process the logic matrix to increase the selection probability of a specified candidate word among the candidate words, and obtaining an adjusted logic matrix; the specified word is composed of at least one specified candidate word; using the adjusted logic matrix to determine the generated text; and the text generation effect can be optimized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a text generation method, device, host device, system, program product and medium. Background Art

[0002] With the continuous development of artificial intelligence technology, pre-trained language models are gradually being applied in various fields. For example, in the fields of novels, advertising copywriting, and official document writing, pre-trained language models can continue the text according to the text content of historical chapter texts. However, in this process, the generated text output by the pre-trained language model may differ from the specified content. For example, the generated text may use less specified words that often appear in the historical chapter text, which leads to deviations between the generated text and the historical chapter text.

[0003] Therefore, how to reduce the difference between the generated text output by the pre-trained language model and the specified content is a technical problem that technicians in this field need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a text generation method, apparatus, host device, system, program product and medium, which can effectively reduce the difference between the generated text output by the pre-trained language model and the specified content.

[0005] In order to solve the above technical problems, the present invention provides a text generation method, comprising:

[0006] Obtaining input text; the input text includes a content summary text and a word reference text, and the word reference text includes a specified word;

[0007] Using a pre-trained language model to perform a text generation operation according to the input text, obtaining a logic matrix corresponding to words in the generated text; the pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate word, wherein the selection probability is the probability that the candidate word is selected as a word in the generated text;

[0008] The logic matrix is ​​processed by using a neural network layer in a pre-trained probability transfer model to increase the selection probability of a designated candidate word among the candidate words, thereby obtaining an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words;

[0009] The adjusted logic matrix is ​​used to determine words in the generated text to obtain the generated text.

[0010] Optionally, the processing of the logic matrix by using a neural network layer in a pre-trained probability transfer model includes:

[0011] The logic matrix is ​​processed using a fully connected layer in the pre-trained probability transfer model.

[0012] Optionally, the training process of the pre-trained probability transfer model includes:

[0013] Acquire training text, and use the pre-trained language model to perform text generation operation according to the training text to obtain a training logic matrix; the training text includes the content summary text and the word reference text;

[0014] The training logic matrix is ​​processed by using the fully connected layer in the initial probability transfer model to obtain a logic matrix to be evaluated;

[0015] Processing the logic matrix to be evaluated by using a normalization function to obtain basic selection probabilities of the candidate words, and forming a first probability distribution by using the basic selection probabilities;

[0016] The basic selection probability is processed by using a preset truncation function to obtain an adjusted selection probability of the candidate word, and the adjusted selection probability is used to form a second probability distribution; the preset truncation function is used to determine a target designated candidate word whose basic selection probability is not the maximum basic selection probability, and is used to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability;

[0017] The first probability distribution and the second probability distribution are used to calculate a loss value, and the loss value is used to update the parameters of the initial probability transfer model to obtain the pre-trained probability transfer model.

[0018] Optionally, when calling the preset truncation function, execute:

[0019] For each of the target designated candidate words, adjusting the basic selection probability of the target designated candidate word to a preset multiple of the maximum basic selection probability;

[0020] Or, construct an adjustment interval with 1 as the lower limit of the interval and a preset multiple of the maximum basic selection probability as the upper limit of the interval;

[0021] For each of the target designated candidate words, a random value is selected from the adjustment interval, and the product of the random value and the basic selection probability of the target designated candidate word is used as the adjusted selection probability of the target designated candidate word.

[0022] Optionally, calculating the loss value by using the first probability distribution and the second probability distribution includes:

[0023] A relative entropy between the first probability distribution and the second probability distribution is calculated, and the relative entropy is used as the loss value.

[0024] Optionally, obtaining the input text includes:

[0025] Acquire a historical chapter text, and determine the designated word from the historical chapter text;

[0026] The sentence or paragraph where the designated word is located in the historical chapter text is used as the word reference text.

[0027] Optionally, using the adjusted logic matrix to determine words in the generated text to obtain the generated text includes:

[0028] Determine the selection probability of the candidate words by using the adjusted logic matrix, and sample words by using different sampling parameters and / or different sampling strategies according to the selection probability to obtain multiple groups of generated texts;

[0029] An optimal generated text is determined among the multiple groups of generated texts.

[0030] Optionally, determining the best generated text among the multiple groups of generated texts includes:

[0031] Acquire historical chapter texts and corresponding chapter summary texts, and form a first text pair with the historical chapter texts and the chapter summary texts, and form a second text pair with each group of the generated texts and the content summary texts;

[0032] For each group of second text pairs, using the trained recurrent neural network model to determine the matching degree between the second text pair and the first text pair;

[0033] An optimal generated text is determined from the multiple groups of generated texts according to the matching degree.

[0034] Optionally, the determining the degree of match between the second text pair and the first text pair using the trained recurrent neural network model includes:

[0035] The first text pair and the second text pair are sequentially input into each recurrent neural network unit in the recurrent neural network model, and the matching degree output by the recurrent neural network unit corresponding to the second text pair is received.

[0036] Optionally, the training process of the recurrent neural network model includes:

[0037] Acquire training text; the training text comprises a plurality of text paragraphs with coherent contents and paragraph summary texts corresponding to each of the text paragraphs;

[0038] Combining each of the text paragraphs and the corresponding paragraph summary text into a third text pair, and combining the third text pairs into a sequence according to the order of the paragraphs to obtain a positive sample sequence;

[0039] Randomly forming fourth text pairs from each of the text paragraphs and each of the paragraph summary texts, and combining the fourth text pairs into sequences in a random order to obtain a negative sample sequence;

[0040] The recurrent neural network model is trained using the positive sample sequence and the negative sample sequence.

[0041] The present invention also provides a text generation device, comprising:

[0042] An acquisition module, used to acquire input text; the input text includes a content summary text and a word reference text, and the word reference text includes a specified word;

[0043] A first processing module is used to perform a text generation operation according to the input text using a pre-trained language model to determine a logic matrix corresponding to a word in the generated text; the pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine a selection probability of the candidate word, wherein the selection probability is a probability that the candidate word is selected as a word in the generated text;

[0044] A second processing module is used to process the logic matrix using a neural network layer in a pre-trained probability transfer model to increase the selection probability of a designated candidate word among the candidate words, thereby obtaining an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words;

[0045] The text generation module is used to determine the words in the generated text by using the adjusted logic matrix to obtain the generated text.

[0046] The present invention also provides a host device, comprising:

[0047] Memory for storing computer programs;

[0048] A processor is used to implement the text generation method as described above when executing the computer program.

[0049] The present invention also provides a text generation system, comprising:

[0050] A host device, configured to execute the text generation method as described above;

[0051] The user device is used to send input text to the host device and obtain generated text from the host device.

[0052] The present invention also provides a computer program product, comprising a computer program or instructions, which implement the steps of the text generation method described above when executed by a processor.

[0053] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the text generation method as described above is implemented.

[0054] The present invention provides a text generation method, comprising: obtaining an input text; the input text comprises a content summary text and a word reference text, the word reference text comprises a designated word; using a pre-trained language model to perform a text generation operation according to the input text, and obtaining a logic matrix corresponding to words in a generated text; the pre-trained language model has a vocabulary comprising candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate words, the selection probability being the probability that the candidate words are selected as words in the generated text; using a neural network layer in a pre-trained probability transfer model to process the logic matrix, so as to increase the selection probability of designated candidate words in the candidate words, and obtain an adjusted logic matrix; the designated word is composed of at least one designated candidate word; and using the adjusted logic matrix to determine the words in the generated text, and obtain the generated text.

[0055] The beneficial effect of the present invention is that the present invention can first obtain an input text, and use a pre-trained language model to perform a text generation operation according to the input text to obtain a logical matrix corresponding to the words in the generated text. Among them, the input text includes a content overview text and a word reference text, and the content overview text is used to provide a content framework of the generated text; the word reference text contains a specified word, which is used to prompt the pre-trained language model how to use the specified word. In addition, the pre-trained language model has a vocabulary containing candidate words, and the logical matrix is ​​used to determine the selection probability of the candidate word, and the selection probability is the probability that the candidate word is selected as a word in the generated text. In other words, the logical matrix generated by the pre-trained language model plays an important role in screening the words contained in the generated text. Subsequently, in order to increase the probability of the generated text containing the above-mentioned specified words, the present invention can use the neural network layer in the pre-trained probability transfer model to process the logical matrix to increase the selection probability of the specified candidate word in the candidate word, and obtain an adjusted logical matrix. Among them, the specified word is composed of at least one specified candidate word. Finally, the adjusted logical matrix can be used to determine the words in the generated text to obtain the generated text. In other words, the present invention can add a pre-trained probability transfer model after the pre-trained language model to directly adjust the logic matrix, which can significantly increase the probability of the specified words appearing in the generated text, thereby reducing the difference between the generated text output by the pre-trained language model and the specified content. The present invention also provides a text generation device, a host device, a system, a computer program product, and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A flowchart of a text generation method provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of training a probability transfer model provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of the structure of a recurrent neural network model provided by an embodiment of the present invention;

[0060] Figure 4 A flowchart of another text generation method provided by an embodiment of the present invention;

[0061] Figure 5 A structural block diagram of a text generation device provided by an embodiment of the present invention;

[0062] Figure 6 A structural block diagram of a host device provided by an embodiment of the present invention;

[0063] Figure 7 A structural block diagram of a text generation system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] A pre-trained language model is a machine learning model used to understand and generate natural language. Common pre-trained language models include large language models (LLM). The model usually has a vocabulary, which contains several candidate words, and each candidate word is encoded in the form of a token. A candidate word is the smallest text unit, which can be a word, a punctuation mark, a number, a character, a subword, etc. Token is the data form provided to the pre-trained language model for internal processing. Candidate words can be combined with each other to form new words. For example, at least one candidate word can form a specified word. When performing a text generation operation, the model can parse the input text, predict tokens one by one based on the parsing results, and then decode the obtained token sequence into natural language text. In other words, the pre-trained language model generates natural language text by selecting words one by one.

[0066] With the continuous development of artificial intelligence technology, pre-trained language models are gradually being applied in various fields. For example, in the fields of novels, advertising copywriting, and official document writing, pre-trained language models can continue to write texts based on the text content of historical chapter texts. However, in the text generation process, the generated text output by the pre-trained language model may differ from the specified content. For example, the generated text may use less specified words that appear in the historical chapter text, which leads to deviations between the generated text and the historical chapter text. The reason for this defect is that when predicting each word, the pre-trained language model usually first generates a logical matrix (Logits), and uses the normalization function (Softmax) to process the logical matrix to obtain the selection probability of multiple candidate words, and then samples the multiple candidate words according to the selection probability to select the appropriate word. Among them, the selection probability is the probability that the candidate word is selected. The higher the selection probability, the easier it is to be selected, and the lower the selection probability, the less likely it is to be selected. It can be seen that the logical matrix directly affects the selection probability of each candidate word. If the logic matrix generated by the pre-trained language model is not reliable enough, it will lead to a lower selection probability of the specified candidate words that form the specified words, which will further reduce the probability of the specified words appearing in the generated text.

[0067] In related technologies, in order to increase the probability of a specified word appearing in a generated text, a pre-trained language model is usually fine-tuned to optimize the model parameters of the model. However, a pre-trained language model usually contains a huge number of parameters, which makes it difficult to ensure both training efficiency and training effect.

[0068] In view of this, in order to solve the technical problem of how to reduce the difference between the generated text output by the pre-trained language model and the specified content, the present invention can provide a text generation method, which can add a pre-trained probability transfer model after the pre-trained language model, and use the pre-trained probability transfer model to directly adjust the above-mentioned logic matrix. Since the pre-trained probability transfer model has learned how to adjust the original logic matrix output by the pre-trained language model to increase the selection probability of the specified candidate words, it can significantly improve the probability of the specified words appearing in the generated text, thereby reducing the difference between the generated text output by the pre-trained language model and the specified content.

[0069] The text generation method provided by the embodiment of the present invention is introduced below. It should be noted that this embodiment does not limit the host device of this method, and can be set according to actual application requirements, for example, it can be a personal computer, a server, etc. It is worth pointing out that when this host device is deployed at the back end, in order to meet the needs of users, the user device can also interact with the host device, such as the user device can send input data to the host device, and the host device can send generated text to the user device. This embodiment also does not limit the above-mentioned user device, for example, it can be a personal computer, a mobile device, etc.

[0070] Please refer to Figure 1 , Figure 1 A flowchart of a text generation method provided by an embodiment of the present invention, the method may include:

[0071] S101, obtaining input text; the input text includes a content summary text and a word reference text, and the word reference text contains a specified word.

[0072] In this step, the input text can be obtained first. The input text can include two parts, namely, the content overview text and the word reference text. The content overview text is used to describe the general content of the text to be generated. For example, for a novel continuation task, the content overview text can be a chapter introduction. The word reference text contains specified words, which are used to prompt the pre-trained language model how to generate the specified words. In other words, the content overview text is used to constrain the text content, and the word reference text is used to constrain the use of words. The two together constrain the content generated by the pre-trained language model.

[0073] Furthermore, in order to reduce the content difference between the generated text output by the pre-trained language model and the historical chapter text, the above-mentioned word reference text can be extracted from the historical chapter text. That is, firstly, the designated word is determined from the historical chapter text, and then the sentence or paragraph where the designated word is located in the historical chapter text is used as the word reference text to prompt the pre-trained language model about the usage of the designated word in the historical chapter text.

[0074] Based on this, the obtaining of input text may include:

[0075] Step 11: Obtain the historical chapter text, and determine the designated word from the historical chapter text;

[0076] Step 12: Use the sentence or paragraph where the designated word is located in the historical chapter text as the word reference text.

[0077] It should be noted that this embodiment does not limit the method of determining the specified words from the historical chapter text. For example, it can be determined manually; the inverse document frequency method (TF-IDF, term frequency–inverse document frequency) can also be used to extract words whose inverse document frequency values ​​meet preset requirements from the historical chapter text as specified words; named entity recognition or language model extraction methods can also be used for extraction.

[0078] Furthermore, in order to reduce the content difference between the generated text output by the pre-trained language model and the historical chapter text, the content summary text may further include a historical chapter summary of the historical chapter text in addition to describing the general content of the text to be generated. The historical chapter summary may be obtained by extracting key points from the historical chapter text using the pre-trained language model.

[0079] S102, using a pre-trained language model to perform a text generation operation according to the input text to obtain a logic matrix corresponding to the words in the generated text; the pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate words, and the selection probability is the probability of the candidate word being selected as a word in the generated text.

[0080] In this step, the input text can be input into the pre-trained language model, so as to use the pre-trained language model to perform text generation operations according to the input text, and obtain a logical matrix corresponding to the words in the generative text. Among them, the pre-trained language model has a vocabulary containing candidate words, and the logical matrix is ​​used to determine the selection probability of the candidate word. Specifically, the logical matrix can be normalized (Softmax) to obtain the selection probability, which is the probability that the candidate word is selected as a word in the generative text. Since the pre-trained language model usually predicts words one by one, when generating a large section of generative text, it will predict multiple logical matrices one by one, corresponding to multiple words in the generative text. It should be pointed out that the logical matrix generated by the pre-trained language model in this embodiment will not be used directly to predict words, but needs to be processed by the pre-trained probability transfer model before being used to predict words.

[0081] It should also be noted that this embodiment does not limit the specific working mode of the pre-trained language model, and reference may be made to related technologies.

[0082] S103, using the neural network layer in the pre-trained probability transfer model to process the logic matrix to increase the selection probability of the designated candidate words among the candidate words, and obtain an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words.

[0083] In this step, the neural network layer in the pre-trained probability transfer model is used to process the logic matrix to increase the selection probability of the specified candidate words in the candidate words, and obtain an adjusted logic matrix. Among them, the pre-trained probability transfer model belongs to a neural network model, which is composed of multiple neural network layers, and each neural network layer contains a number of trainable model parameters. In this embodiment, the model parameters in the model can be trained and optimized in advance so that the model can directly process the logic matrix and increase the selection probability of the specified candidate words in the candidate words. Since the above-mentioned specified words can be composed of at least one specified candidate word, this embodiment can significantly improve the probability of the specified words appearing in the generated text, thereby narrowing the gap between the generated text and the specified content text (such as the history chapter text).

[0084] It should be noted that this embodiment does not limit the internal structure of the pre-trained probability transfer model, which can be set according to actual application requirements. For example, the pre-trained probability transfer model can be composed of at least one set of fully connected layers (Fully Connected Layer), which can perform more complex integration and transformation on the logic matrix, so as to better improve the probability of occurrence of the specified words in the generated text.

[0085] Based on this, the processing of the logic matrix by using the neural network layer in the pre-trained probability transfer model may include:

[0086] Step 21: Process the logic matrix using the fully connected layer in the pre-trained probability transfer model.

[0087] S104: Using the adjusted logic matrix to determine words in the generated text, to obtain the generated text.

[0088] In this step, after completing the adjustment of the logic matrix, this embodiment can use the adjusted logic matrix to determine each word in the generated text one by one to obtain the generated text. Specifically, each adjusted logic matrix can be normalized (Softmax) to obtain the selection probability of each candidate word. Subsequently, according to the selection probability, word sampling is performed using sampling parameters (such as temperature, Top-p parameters, Top-k parameters, etc.) and sampling strategies (such as beam search strategy, Top-p strategy, Top-k strategy, etc.) to select appropriate words and finally obtain the generated text.

[0089] Furthermore, considering that different sampling parameters and sampling strategies can generate generative texts with different text styles, the above text styles also affect the difference between the generative text and the specified content. For example, for news summary generation, it needs to objectively reflect the content of the original text, so it may be more inclined to a lower temperature and smaller top-k and top-p values ​​to ensure that the generated summary is highly consistent with the original text; for creative writing, it requires higher creativity and diversity, so it may be necessary to increase the temperature and increase the top-k and top-p values ​​to encourage the model to generate more novel and unique expressions. Therefore, this embodiment can also use different sampling parameters and / or different sampling strategies to sample words according to the selection probability of candidate words to obtain multiple groups of generative texts, and determine the best generative text among the multiple groups of generative texts to determine a higher quality generative text.

[0090] Based on this, using the adjusted logic matrix to determine the words in the generated text to obtain the generated text may include:

[0091] Step 31: using the adjusted logic matrix to determine the selection probability of the candidate words, and adopting different sampling parameters and / or different sampling strategies to perform word sampling according to the selection probability to obtain multiple groups of generated texts;

[0092] Step 32: Determine the best generated text among the multiple groups of generated texts.

[0093] It should be noted that this embodiment does not limit how to determine the best generative text among multiple groups of generative texts, and can be set according to actual application requirements. For example, it can be output to a user device and receive a selection instruction returned by the user device to determine the best generative text; other models can also be used to determine the best generative text.

[0094] Based on the above embodiment, the present invention can first obtain an input text, and use a pre-trained language model to perform a text generation operation according to the input text to obtain a logical matrix corresponding to the words in the generated text. Wherein, the input text includes a content overview text and a word reference text, and the content overview text is used to provide a content framework of the generated text; the word reference text contains a specified word, which is used to prompt the pre-trained language model how to use the specified word. In addition, the pre-trained language model has a vocabulary containing candidate words, and the logical matrix is ​​used to determine the selection probability of the candidate word, and the selection probability is the probability of being selected as a word in the generated text. In other words, the logical matrix generated by the pre-trained language model plays an important role in screening the words contained in the generated text. Subsequently, in order to increase the probability of containing the above-mentioned specified words in the generated text, the present invention can use the neural network layer in the pre-trained probability transfer model to process the logical matrix to increase the selection probability of the specified candidate words in the candidate words, and obtain an adjusted logical matrix. Wherein, the specified word is composed of at least one specified candidate word. Finally, the adjusted logical matrix can be used to determine the words in the generated text to obtain the generated text. In other words, the present invention can add a pre-trained probability transfer model after the pre-trained language model to directly adjust the logic matrix, which can significantly improve the probability of specified words appearing in the generated text, thereby reducing the difference between the generated text output by the pre-trained language model and the specified content.

[0095] Based on the above embodiment, the training process of the pre-trained language model is introduced below. Based on this, the training process of the pre-trained probability transfer model may include:

[0096] S201, obtaining training text, and using the pre-trained language model to perform a text generation operation according to the training text to obtain a training logic matrix; the training text includes the content summary text and the word reference text.

[0097] In this step, the training text is used to train the probability transfer model, which also needs to include two parts: content overview text and word reference text. The training logic matrix is ​​the logic matrix generated by the pre-trained language model based on the training text.

[0098] S202: Process the training logic matrix using the fully connected layer in the initial probability transfer model to obtain a logic matrix to be evaluated.

[0099] In this embodiment, the initial probability transfer model refers to an untrained probability transfer model. In this step, the training logic matrix is ​​processed using the fully connected layer in the initial probability transfer model to obtain the logic matrix to be evaluated, and the adjustment of the candidate word selection probability by the initial probability transfer model is determined based on the logic matrix to be evaluated.

[0100] It is worth noting that this embodiment does not require fine-tuning of the pre-trained language model, but only requires training of the initial probability transfer model, which can avoid long training time and unstable training results.

[0101] S203: Process the logic matrix to be evaluated by using a normalization function to obtain basic selection probabilities of the candidate words, and use the basic selection probabilities to form a first probability distribution.

[0102] In this step, the normalization function is used to process the logic matrix to be evaluated to obtain the basic selection probability of each candidate word, and all the basic selection probabilities are used to form the first probability distribution. At this time, the basic selection probability of the specified candidate word may not be improved, so the basic selection probability of the specified candidate word will be adjusted in the next step.

[0103] S204. Process the basic selection probability using a preset truncation function to obtain an adjusted selection probability of the candidate word, and use the adjusted selection probability to form a second probability distribution; the preset truncation function is used to determine a target designated candidate word whose basic selection probability is not the maximum basic selection probability, and is used to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability.

[0104] In this step, each basic selection probability is processed using a preset truncation function, and the basic selection probability of the designated candidate word is processed using the preset truncation function in particular, to obtain the adjusted selection probability of each candidate word, and the adjusted selection probability is used to form a second probability distribution. Among them, the preset truncation function is used to determine the target designated candidate word whose basic selection probability is not the maximum basic selection probability, and to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability. For other candidate words that do not belong to the designated candidate words, the preset truncation function only multiplies the basic selection probability of the candidate word by 1. It can be seen that in the second probability distribution, the adjusted selection probabilities of the designated candidate words are all maintained at a relatively high level.

[0105] It should be noted that this embodiment does not limit how to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability. For example, the basic selection probability of the target designated candidate word can be directly adjusted to a preset multiple of the maximum basic selection probability. In addition, this embodiment can also construct an adjustment interval, and select a random value from the adjustment interval as the adjustment multiple of the selection probability of the designated candidate word to improve the training effect. Specifically, the adjustment interval can be constructed with 1 as the lower limit of the interval and the preset multiple of the maximum basic selection probability as the upper limit of the interval, and for each target designated candidate word, a random value is selected from the adjustment interval to use the product of the random value and the basic selection probability of the target designated candidate word as the adjusted selection probability of the target designated candidate word. In this way, the training effect of the model can be effectively guaranteed.

[0106] Based on this, when calling the preset truncation function, execute:

[0107] Step 41: For each of the target designated candidate words, the basic selection probability of the target designated candidate word is adjusted to a preset multiple of the maximum basic selection probability.

[0108] Or, when calling the preset truncation function:

[0109] Step 51: constructing an adjustment interval with 1 as the interval lower limit and a preset multiple of the maximum basic selection probability as the interval upper limit;

[0110] Step 52: For each of the target designated candidate words, a random value is selected from the adjustment interval, and the product of the random value and the basic selection probability of the target designated candidate word is used as the adjusted selection probability of the target designated candidate word.

[0111] Specifically, the adjusted selection probability can be calculated by the following formula:

[0112] ;

[0113] Among them, target_probs i represents the adjusted selection probability of the i-th candidate word, exp() represents the natural exponential function, n_vocab represents the total number of candidate words; G() represents the truncation function, which is expressed as:

[0114] ;

[0115] Among them, new_probs llm represents the basic selection probability, k iis a random number between [1, 1.1×max(target_probs)], max() represents the maximum value function, index() function is used to determine the sequence number of the candidate word, and index_of_next_token represents the specified candidate word.

[0116] S205. Calculate a loss value using the first probability distribution and the second probability distribution, and use the loss value to update parameters of the initial probability transfer model to obtain the pre-trained probability transfer model.

[0117] In this step, the second probability distribution is the training target of the initial probability transfer model, so this embodiment can use the first probability distribution and the second probability distribution to calculate the loss value, and use the loss value to update the parameters of the initial probability transfer model. This embodiment can specifically calculate the relative entropy (KL divergence) between the first probability distribution and the second probability distribution, and use the relative entropy as the loss value, where the relative entropy is an evaluation indicator that describes the difference between the two probability distributions.

[0118] Based on this, the calculating of the loss value by using the first probability distribution and the second probability distribution includes:

[0119] Step 61: Calculate the relative entropy between the first probability distribution and the second probability distribution, and use the relative entropy as the loss value.

[0120] The relative entropy between the first probability distribution and the second probability distribution can be expressed as:

[0121] ;

[0122] in, represents the adjusted selection probability, Indicates the basic selection probability, loss KL Represents relative entropy.

[0123] It should be noted that steps S201 to S205 can be executed repeatedly to reduce the above loss value to a preset threshold. After several rounds of training, it can be ensured that the first probability distribution is substantially consistent with the second probability distribution, thereby achieving the effect of increasing the probability of occurrence of the specified candidate word.

[0124] In addition, it is worth pointing out that since the probability transfer model in the present invention does not simply increase the selection probability of the specified candidate word without adjusting the selection probability of the remaining candidate words, but can learn the overall probability distribution after the selection probability of the specified candidate word is increased, it can take into account the selection probability of the remaining candidate words and adjust the selection probability of all candidate words from a global perspective so that the larger selection probability is biased towards the specified candidate word. In this way, the model performance can be improved, so that the pre-trained language model can achieve better text generation effect.

[0125] Based on the above embodiment, the structure and training method of the above probability transfer model are fully introduced below based on a specific schematic diagram. Figure 2 , Figure 2 FIG. 1 is a schematic diagram of a training of a probability transfer model provided by an embodiment of the present invention. Figure 2 As shown, the probability transfer model used in the present invention has 2 fully connected layers, and their weights are:

[0126] ;

[0127] .

[0128] Among them, Matrix represents the weight matrix, hid_size represents the size of the fully connected layer, and n_vocab represents the total number of candidate words.

[0129] After freezing the parameters of the pre-trained language model, the logical matrix output by the content overview text and word reference text is:

[0130] ;

[0131] After the two fully connected layers are converted, the new logits distribution size is still [n_vocab, hid_size]:

[0132] ;

[0133] ;

[0134] Among them, f is the activation function and b is the bias.

[0135] After softmax, we get the basic probability distribution and adjusted probability distribution:

[0136] ;

[0137] ;

[0138] Among them, new_probs iIndicates the basic selection probability of the i-th candidate word, target_probs i represents the adjusted selection probability of the i-th candidate word, exp() represents the natural exponential function, n_vocab represents the total number of candidate words; G() represents the truncation function, which is expressed as:

[0139] ;

[0140] Among them, new_probs llm Represents the basic selection probability, which is composed of all new_probs i Composition; k i is a random number between [1, 1.1×max(target_probs)], max() represents the maximum value function, index() function is used to determine the sequence number of the candidate word, and index_of_next_token represents the specified candidate word.

[0141] The loss is designed as KL divergence, forcing the probability transfer model to make the logits output by LLM more inclined to the specified token, while ensuring the stability of the probability distribution of other tokens.

[0142] ;

[0143] in, represents the adjusted selection probability, Indicates the basic selection probability, loss KL Represents relative entropy.

[0144] By training the proprietary vocabulary, chapter overviews, and background overviews in the novel, the probability transfer model can be optimized. When actually generating text, the content output by the pre-trained language model is transferred through the probability transfer model and then sampled. This method can effectively improve the accuracy of the pre-trained language model in generating proprietary vocabulary and generating content according to specified backgrounds and topics. At the same time, it avoids fine-tuning the pre-trained language model and reduces the consumption of computing resources.

[0145] Based on the above embodiment, the specific process of determining the best generated text in multiple groups of generated texts is described below. Based on this, determining the best generated text in the multiple groups of generated texts may include:

[0146] S301, obtaining historical chapter texts and corresponding chapter summary texts, and combining the historical chapter texts and the chapter summary texts into a first text pair, and combining each group of the generated texts and the content summary text into a second text pair.

[0147] In order to reduce the difference between the generative text and the historical chapter text, especially the contextual logic difference between the two, this embodiment will obtain the historical chapter text, and calculate the matching degree between each group of generative text and the historical chapter text, so as to select the appropriate optimal generative text. To this end, this step can obtain the historical chapter text and the corresponding chapter summary text, and form the historical chapter text and the chapter summary text into a first text pair, and form each group of the generative text and the content summary text into a second text pair for subsequent processing.

[0148] S302: For each group of second text pairs, use the trained recurrent neural network model to determine the matching degree between the second text pairs and the first text pairs.

[0149] In this embodiment, in order to determine the matching degree between the generative text and the historical chapter text, a recurrent neural network model (RNN, Recurrent Neural Network) is specially introduced. The model can be composed of multiple recurrent neural network units in series. Each recurrent neural network unit receives two inputs: a text pair and the hidden state information output by the previous recurrent neural network model. The hidden state information received by the first recurrent neural network unit is issued by itself; each recurrent neural network unit generates two outputs: matching degree and hidden state information. Among them, the hidden state information is obtained by the recurrent neural network unit processing all its inputs. The reason for introducing the recurrent neural network model is that the model has a strong feature extraction and parsing ability for sequence-shaped data, and can capture sequence features and analyze the correlation between each data in the sequence. In this embodiment, a sequence can be formed between the historical chapter text and the generative text, and in an ideal state, the historical chapter text and the generative text should have a correlation. Therefore, the recurrent neural network model can be used to process the first text pair and the second text pair to determine the matching degree between the first text pair and the second text pair, so as to select the appropriate optimal generative text.

[0150] Based on this, the determining the matching degree between the second text pair and the first text pair using the trained recurrent neural network model may include:

[0151] Step 71: sequentially input the first text pair and the second text pair into each recurrent neural network unit in the recurrent neural network model, and receive the matching degree output by the recurrent neural network unit corresponding to the second text pair.

[0152] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of a recurrent neural network model provided by an embodiment of the present invention, which may be composed of multiple recurrent neural network units. iRepresents the hidden state information emitted by the i-th recurrent neural network unit, specifically:

[0153]

[0154] Among them, i is the maximum number of paragraphs, f is the activation function of the hidden state layer, and b_h is the bias of the hidden state layer. i It represents the matching degree issued by the i-th recurrent neural network unit, specifically:

[0155] ;

[0156] Among them, g is the activation function of the output layer, b_o is the bias; the shape of O_i is [hid_size, 2].

[0157] Each recurrent neural network unit receives a text pair, such as the first text pair can be input into recurrent neural network unit 1, and the second text pair can be input into recurrent neural network unit 2. When the historical chapter text has multiple paragraphs, the first text pairs corresponding to the historical chapter texts can also be sequentially input into each recurrent neural network unit, and the second text pair can be input into the adjacent next recurrent neural network unit.

[0158] S303: Determine the best generated text from the multiple groups of generated texts according to the matching degree.

[0159] In this step, the generated text with the greatest matching degree can be determined as the best generated text.

[0160] The following describes the training method of the recurrent neural network model. Based on this, the training process of the recurrent neural network model includes:

[0161] S401, obtaining a training text; the training text comprises a plurality of text paragraphs with coherent contents and a paragraph summary text corresponding to each of the text paragraphs.

[0162] S402: Each of the text paragraphs and the corresponding paragraph summary text forms a third text pair, and the third text pairs are combined into a sequence according to the paragraph order to obtain a positive sample sequence.

[0163] S403 , randomly forming fourth text pairs with the text paragraphs and the paragraph summary texts, and combining the fourth text pairs into sequences in a random order to obtain a negative sample sequence.

[0164] S404: Train the recurrent neural network model using the positive sample sequence and the negative sample sequence.

[0165] Specifically, under the premise of having a high-quality training text, this embodiment expects LLM to generate a generative text that is as contextually coherent and logically tight as the training text, and the content is closely linked to the chapter theme. For any chapter, each paragraph in the chapter should be a specific implementation of the chapter theme, and there should be consistency in the logical relationship between the paragraph and the theme, and between the paragraph and the paragraph. Therefore, based on the existing high-quality training text, each chapter can be used as a positive sample to construct a list of <Chapter Overview, Paragraph>, that is, to form a positive sample sequence. In addition, negative samples can also be constructed. There are two sources for the construction of negative samples. Similar to the positive sample, the paragraphs in the chapter are then disrupted to construct a negative sample of <Chapter Overview, Disordered Paragraphs>; in addition, the overview in the chapter can be combined with the paragraphs of other chapters to construct a negative sample of <Chapter Overview, Other Chapter Paragraphs>, that is, to form a negative sample sequence. Subsequently, the positive sample sequence and the negative sample sequence can be used to circulate the neural network model so that the model learns how to classify the positive sample sequence and the negative sample sequence. The loss function is designed as a binary cross entropy loss function, with the positive sample label being 1 and the negative sample label being 0.

[0166] Finally, for easier understanding, please refer to Figure 4 , Figure 4 This is a flowchart of another text generation method provided by an embodiment of the present invention. This embodiment provides a complete text generation process including:

[0167] 1. Input the input text containing the content summary text and the word reference text into the pre-trained language model for processing to obtain a logical matrix;

[0168] 2. Input the logic matrix into the probability transfer model for processing to increase the selection probability of the specified candidate word and obtain an adjusted logic matrix;

[0169] 3. Normalize the adjusted logic matrix to obtain a probability distribution, and use a mixed sampling method (i.e., different sampling parameters and different sampling strategies) to sample words based on the probability distribution to obtain a variety of generated texts;

[0170] 4. Input the historical chapter texts and each group of generative texts into the recurrent neural network model for processing to determine the best generative text.

[0171] The text generation device, host device, text generation system, computer-readable storage medium, and computer program product provided in the embodiments of the present invention are introduced below. The text generation device, host device, text generation system, computer-readable storage medium, and computer program product described below can be referenced to the text generation method described above.

[0172] Please refer to Figure 5 , Figure 5A structural block diagram of a text generation device provided by an embodiment of the present invention, the device may include:

[0173] The acquisition module 501 is used to acquire input text; the input text includes a content summary text and a word reference text, and the word reference text includes a specified word;

[0174] A first processing module 502 is used to perform a text generation operation according to the input text using a pre-trained language model to determine a logic matrix corresponding to a word in the generated text; the pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine a selection probability of the candidate word, wherein the selection probability is a probability that the candidate word is selected as a word in the generated text;

[0175] The second processing module 503 is used to process the logic matrix using the neural network layer in the pre-trained probability transfer model to increase the selection probability of the designated candidate word among the candidate words, and obtain an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words;

[0176] The text generation module 504 is used to determine the words in the generated text by using the adjusted logic matrix to obtain the generated text.

[0177] Optionally, the second processing module 503 may be used to:

[0178] The logic matrix is ​​processed using a fully connected layer in the pre-trained probability transfer model.

[0179] Optionally, the device may include:

[0180] A training text acquisition module, used to acquire training text, and use the pre-trained language model to perform a text generation operation according to the training text to obtain a training logic matrix; the training text includes the content summary text and the word reference text;

[0181] A third processing module is used to process the training logic matrix using the fully connected layer in the initial probability transfer model to obtain a logic matrix to be evaluated;

[0182] A normalization processing module, used for processing the to-be-evaluated logic matrix using a normalization function to obtain a basic selection probability of the candidate word, and using the basic selection probability to form a first probability distribution;

[0183] a truncation processing module, configured to process the basic selection probability using a preset truncation function to obtain an adjusted selection probability of the candidate word, and to form a second probability distribution using the adjusted selection probability; the preset truncation function is configured to determine a target designated candidate word whose basic selection probability is not the maximum basic selection probability, and to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability;

[0184] A loss value calculation module is used to calculate the loss value using the first probability distribution and the second probability distribution, and use the loss value to update the parameters of the initial probability transfer model to obtain the pre-trained probability transfer model.

[0185] Optionally, the truncation processing module may include:

[0186] A first adjustment submodule is used to adjust the basic selection probability of each target designated candidate word to a preset multiple of the maximum basic selection probability;

[0187] Or, an interval construction submodule, used to construct an adjustment interval with 1 as the lower limit of the interval and a preset multiple of the maximum basic selection probability as the upper limit of the interval;

[0188] The second adjustment submodule is used to select a random value from the adjustment interval for each of the target designated candidate words, and use the product of the random value and the basic selection probability of the target designated candidate word as the adjusted selection probability of the target designated candidate word.

[0189] Optionally, the loss value calculation module can be used to:

[0190] A relative entropy between the first probability distribution and the second probability distribution is calculated, and the relative entropy is used as the loss value.

[0191] Optionally, the acquisition module 501 may include:

[0192] A text determination submodule, used for acquiring a historical chapter text and determining the designated word from the historical chapter text;

[0193] The word reference text determination submodule is used to take the sentence or paragraph where the designated word is located in the historical chapter text as the word reference text.

[0194] Optionally, using the text generation module 504 may include:

[0195] A text generation submodule, for determining the selection probability of the candidate words by using the adjusted logic matrix, and sampling words by using different sampling parameters and / or different sampling strategies according to the selection probability to obtain multiple groups of generated texts; each group of the probability distributions includes the selection probability of the candidate words;

[0196] The text selection submodule is used to determine the best generative text among the multiple groups of generative texts.

[0197] Optionally, the text selection submodule may include:

[0198] A text pair construction unit, used for obtaining a historical chapter text and a corresponding chapter summary text, and combining the historical chapter text and the chapter summary text into a first text pair, and combining each group of the generated text and the content summary text into a second text pair;

[0199] a matching degree calculation unit, configured to determine, for each group of second text pairs, a matching degree between the second text pairs and the first text pairs using a trained recurrent neural network model;

[0200] A text selection unit is used to determine the best generated text from multiple groups of generated texts according to the matching degree.

[0201] Optionally, the matching degree calculation unit may be used to:

[0202] The first text pair and the second text pair are sequentially input into each recurrent neural network unit in the recurrent neural network model, and the matching degree output by the recurrent neural network unit corresponding to the second text pair is received.

[0203] Optionally, the device may include:

[0204] A training text acquisition module, used to acquire training text; the training text includes a plurality of text paragraphs with coherent content and paragraph summary texts corresponding to each of the text paragraphs;

[0205] A positive sample construction module, used for forming a third text pair from each of the text paragraphs and the corresponding paragraph summary text, and combining the third text pairs into a sequence according to the paragraph order to obtain a positive sample sequence;

[0206] A negative sample construction module, used for randomly forming fourth text pairs from each of the text paragraphs and each of the paragraph summary texts, and combining the fourth text pairs into a sequence in a random order to obtain a negative sample sequence;

[0207] A training module is used to train the recurrent neural network model using the positive sample sequence and the negative sample sequence.

[0208] Please refer to Figure 6 , Figure 6 This is a structural block diagram of a host device provided in an embodiment of the present invention. The embodiment of the present invention provides a host device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the text generation method provided in the aforementioned embodiment when executing the computer program.

[0209] For the specific process of the above-mentioned text generation method, please refer to the corresponding content provided in the above-mentioned embodiment, which will not be repeated here.

[0210] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0211] In addition, the host device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the host device 10; the communication interface 14 can create a data transmission channel between the host device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0212] Please refer to Figure 7 , Figure 7 A structural block diagram of a text generation system provided by an embodiment of the present invention, the text generation system may include:

[0213] A host device 10, configured to execute the text generation method as described above;

[0214] The user device 20 is used to send input text to the host device and obtain generated text from the host device.

[0215] Since the embodiments of the text generation system part correspond to the embodiments of the text generation method part, the embodiments of the text generation system part refer to the description of the embodiments of the text generation method part, which will not be repeated here.

[0216] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the text generation method described in the above embodiment when executed by a processor.

[0217] Since the embodiments of the computer program product part correspond to the embodiments of the text generation method part, please refer to the description of the embodiments of the text generation method part for the embodiments of the computer program product part, and will not be repeated here.

[0218] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the text generation method described in the above embodiment is implemented.

[0219] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the text generation method part, the embodiments of the storage medium part refer to the description of the embodiments of the text generation method part, which will not be repeated here.

[0220] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0221] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0222] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0223] The above is a detailed introduction to the text generation method, device, host device, system, program product and medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A text generation method, characterized in that: include: Obtaining input text; the input text includes a content summary text and a word reference text, and the word reference text includes a specified word; Using a pre-trained language model to perform a text generation operation according to the input text, obtaining a logic matrix corresponding to words in the generated text; the pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate word, wherein the selection probability is the probability that the candidate word is selected as a word in the generated text; The logic matrix is ​​processed by using a neural network layer in a pre-trained probability transfer model to increase the selection probability of a designated candidate word among the candidate words, thereby obtaining an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words; Determine the words in the generated text using the adjusted logic matrix to obtain the generated text; The training process of the pre-trained probability transfer model includes: Acquire training text, and use the pre-trained language model to perform text generation operation according to the training text to obtain a training logic matrix; the training text includes the content summary text and the word reference text; The training logic matrix is ​​processed by using the fully connected layer in the initial probability transfer model to obtain a logic matrix to be evaluated; Processing the logic matrix to be evaluated by using a normalization function to obtain basic selection probabilities of the candidate words, and forming a first probability distribution by using the basic selection probabilities; The basic selection probability is processed by using a preset truncation function to obtain an adjusted selection probability of the candidate word, and the adjusted selection probability is used to form a second probability distribution; the preset truncation function is used to determine a target designated candidate word whose basic selection probability is not the maximum basic selection probability, and is used to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability; The first probability distribution and the second probability distribution are used to calculate a loss value, and the loss value is used to update the parameters of the initial probability transfer model to obtain the pre-trained probability transfer model.

2. The text generation method according to claim 1, characterized in that: The logic matrix is ​​processed using a neural network layer in a pre-trained probability transfer model, including: The logic matrix is ​​processed using a fully connected layer in the pre-trained probability transfer model.

3. The text generation method according to claim 1, characterized in that: When calling the preset truncation function, execute: For each of the target designated candidate words, adjusting the basic selection probability of the target designated candidate word to a preset multiple of the maximum basic selection probability; Or, construct an adjustment interval with 1 as the lower limit of the interval and a preset multiple of the maximum basic selection probability as the upper limit of the interval; For each of the target designated candidate words, a random value is selected from the adjustment interval, and the product of the random value and the basic selection probability of the target designated candidate word is used as the adjusted selection probability of the target designated candidate word.

4. The text generation method according to claim 1, characterized in that: Calculating a loss value using the first probability distribution and the second probability distribution includes: A relative entropy between the first probability distribution and the second probability distribution is calculated, and the relative entropy is used as the loss value.

5. The text generation method according to claim 1, characterized in that: Get input text, including: Acquire a historical chapter text, and determine the designated word from the historical chapter text; The sentence or paragraph where the designated word is located in the historical chapter text is used as the word reference text.

6. The text generation method according to any one of claims 1 to 5, characterized in that: Determining words in the generated text using the adjusted logic matrix to obtain the generated text includes: Determine the selection probability of the candidate words by using the adjusted logic matrix, and sample words by using different sampling parameters and / or different sampling strategies according to the selection probability to obtain multiple groups of generated texts; An optimal generated text is determined among the multiple groups of generated texts.

7. The text generation method according to claim 6, characterized in that: Determining the best generated text among the multiple groups of generated texts includes: Acquire historical chapter texts and corresponding chapter summary texts, and form a first text pair with the historical chapter texts and the chapter summary texts, and form a second text pair with each group of the generated texts and the content summary texts; For each group of second text pairs, using the trained recurrent neural network model to determine the matching degree between the second text pair and the first text pair; An optimal generated text is determined from the multiple groups of generated texts according to the matching degree.

8. The text generation method according to claim 7, characterized in that: Determining the degree of match between the second text pair and the first text pair using the trained recurrent neural network model includes: The first text pair and the second text pair are sequentially input into each recurrent neural network unit in the recurrent neural network model, and the matching degree output by the recurrent neural network unit corresponding to the second text pair is received.

9. The text generation method according to claim 8, characterized in that: The training process of the recurrent neural network model includes: Acquire training text; the training text comprises a plurality of text paragraphs with coherent contents and paragraph summary texts corresponding to each of the text paragraphs; Combining each of the text paragraphs and the corresponding paragraph summary text into a third text pair, and combining the third text pairs into a sequence according to the order of the paragraphs to obtain a positive sample sequence; Randomly forming fourth text pairs from each of the text paragraphs and each of the paragraph summary texts, and combining the fourth text pairs into sequences in a random order to obtain a negative sample sequence; The recurrent neural network model is trained using the positive sample sequence and the negative sample sequence.

10. A text generation device, characterized in that: include: An acquisition module, used to acquire input text; the input text includes a content summary text and a word reference text, and the word reference text includes a specified word; A first processing module, configured to perform a text generation operation according to the input text using a pre-trained language model, and determine a logical matrix corresponding to a word in the generated text; The pre-trained language model has a vocabulary containing candidate words, and the logic matrix is ​​used to determine the selection probability of the candidate words, wherein the selection probability is the probability that the candidate word is selected as a word in the generated text; A second processing module is used to process the logic matrix using a neural network layer in a pre-trained probability transfer model to increase the selection probability of a designated candidate word among the candidate words, thereby obtaining an adjusted logic matrix; the designated word is composed of at least one of the designated candidate words; A text generation module, used for determining words in the generated text by using the adjusted logic matrix to obtain the generated text; A training text acquisition module, used to acquire training text, and use the pre-trained language model to perform a text generation operation according to the training text to obtain a training logic matrix; the training text includes the content summary text and the word reference text; A third processing module is used to process the training logic matrix using the fully connected layer in the initial probability transfer model to obtain a logic matrix to be evaluated; A normalization processing module, used for processing the to-be-evaluated logic matrix using a normalization function to obtain a basic selection probability of the candidate word, and using the basic selection probability to form a first probability distribution; a truncation processing module, configured to process the basic selection probability using a preset truncation function to obtain an adjusted selection probability of the candidate word, and to form a second probability distribution using the adjusted selection probability; the preset truncation function is configured to determine a target designated candidate word whose basic selection probability is not the maximum basic selection probability, and to adjust the basic selection probability of the target designated candidate word according to the maximum basic selection probability; A loss value calculation module is used to calculate the loss value using the first probability distribution and the second probability distribution, and use the loss value to update the parameters of the initial probability transfer model to obtain the pre-trained probability transfer model.

11. A host device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the text generation method as claimed in any one of claims 1 to 9 when executing the computer program.

12. A text generation system, characterized in that: include: A host device, configured to execute the text generation method according to any one of claims 1 to 9; The user device is used to send input text to the host device and obtain generated text from the host device.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the text generation method according to any one of claims 1 to 9 are implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by a processor, the text generation method according to any one of claims 1 to 9 is implemented.

Images