Text generation device, text generation system, and text generation method

Abstract

To structure and display a text group generated correspondingly to inputted texts, in a text generation device.SOLUTION: A text generation device: sequentially outputs selection probabilities corresponding to generation word candidates on the basis of input texts and a past word series by using a text generation model; generates the selection probability of the generation word candidate in each step received from the text generation model and texts which branch on the basis of a text branch condition for branching the texts; and structures and outputs a group of generated texts.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a text generation device, a text generation system, and a text generation method.

Background Art

[0002] Conventionally, text generation using natural language processing technology has been prevalent. For example, it is used in fields such as translation, summarization, grammar correction, style conversion, dialogue generation, advertisement generation, story generation, news article generation, etc. It is common to generate and present one text corresponding to a given input, or to generate and present multiple texts.

[0003] In Patent Documents 1 and 2, a text generation method model is used to output one or more generated texts corresponding to an input text. In Patent Document 3, conversation data is received as input and modeled into a tree structure for output.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Documents 1 and 2, since the generated text is simply output, there may be inconvenience in analyzing the generation result. For example, when only one text is generated and output, it is impossible to know what other text candidates there are.

[0006] When generating multiple texts, it takes time to read and understand each generated text individually. Furthermore, it's necessary to manually analyze which parts of the generated texts are common and which parts differ. In short, it's difficult to systematically understand the generated texts.

[0007] Patent Document 3 specializes in structuring conversational branching in parts of conversational data on similar topics where different answers are expected.

[0008] Therefore, it cannot handle the diverse language processing capabilities of the text generation model. Furthermore, the method described in Patent Document 3 requires multiple conversational texts as input for structuring, and cannot generate a structured text set from a single text.

[0009] The objective of the present invention is to structure and display the set of text generated from input text in a text generation device. [Means for solving the problem]

[0010] A text generation device according to one aspect of the present invention includes a text generation model that acquires a plurality of selection probabilities for a plurality of word candidates in the next step following the input text based on the input text and a past word sequence, and a control unit that selects a word candidate having a first selection probability among the plurality of selection probabilities from among the plurality of word candidates in the next step as the first word candidate of the next step, determines whether a text branching condition is met for the word candidate having a second selection probability among the plurality of selection probabilities, selects the word candidate that satisfies the branching condition as the second word candidate of the next step, generates a first text following the first word candidate and a second text following the second word candidate, respectively, using the text generation model, and outputs the generated first text and second text as a structured generated text set. [Effects of the Invention]

[0011] According to one aspect of the present invention, a text generation device can display a structured set of text generated from input text. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of a text generation device. [Figure 2] This diagram shows an example of a text generation model architecture (encoder-decoder format). [Figure 3] This diagram shows an example of a text generation model architecture (decoder format). [Figure 4] This diagram shows an example of a processing flowchart performed by a text generation device. [Figure 5] This figure shows an example of news text output generated using a text generation device. [Figure 6] This figure shows an example of the output of translated text generated using a text generation device. [Figure 7] This figure shows an example of output in an embodiment where two or more text branching steps are performed using a text generation device. [Figure 8] This figure shows an example of output in an embodiment in which a text generation device is used to generate text that is a different variation of the input text. [Figure 9] This diagram shows an example of a user interface for a text generation device. [Figure 10] This diagram shows an example of the hardware configuration of a text generation device. [Figure 11] This is a schematic diagram of the text generation system. [Modes for carrying out the invention]

[0013] Hereinafter, embodiments will be described with reference to the drawings. In the description of the drawings, elements having the same function are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the following embodiments are one of the forms for implementing the present invention and are not limited to this embodiment.

[0014] In the embodiments, Japanese text examples are used for the description, but this patent does not limit the language of the text to Japanese. That is, it is also applicable to other natural languages such as English and Chinese, and formal languages such as Java and Python. Also, in this patent, for the sake of simplicity, Japanese short texts are used as generation examples for the description, but the "text" to be generated in this patent is not limited to sentences. That is, it is also applicable to longer texts such as articles.

[0015] FIG. 1 is a diagram showing an overview of the text generation device 101. The text generation device 101 is composed of a control / recording unit 102 and a text generation model 103.

[0016] The text generation device 101 receives at least input text as an input. For example, in tasks such as dialogue generation, story generation, and news generation, the input text is the leading text for controlling the content of the subsequent text. For example, "In order to improve the economy" is an example of the input text, and a set of texts subsequent to the input text such as "Financial policy reform is necessary, and for that, the construction of a stable economy is necessary." is generated by the text generation device.

[0017] Also, for example, in tasks such as translation text generation, summary text generation, and style conversion text generation, the input text is the source text that is the basis of the text to be generated. For example, in translation text generation, "The Japanese economy needs to be revitalized." is an example of the input text, and a set of translation texts corresponding to the input text such as "日本経済を再生する必要がある。" is generated by the text generation device.

[0018] In addition to the input text, some of the various parameters for text generation may also be accepted as input. If the various parameters are not accepted as input, they may be set as fixed values ​​when the text generation device 101 is started. Examples of parameters in the text generation device include the number of texts to be generated by the text generation device, the branching conditions used and the various parameters in the branching conditions, and whether or not the branching probability at the input text and branching position is output.

[0019] The control and recording unit 102 controls the inputs provided to the generation model, determines text branching, and records the generated text and branching probabilities. Once text generation is complete, it outputs a structured set of texts.

[0020] The text generation model 103 is a model that outputs text corresponding to the input text. As the text generation model 103, for example, a generative language model using a neural network (such as GPT (Generative Pre-Training) or BART (Bidirectional and Auto-Regressive Transformers)) may be used.

[0021] The text generation model 103 may be pre-trained using machine learning or other means to obtain an appropriate output for the input text. For example, if the text generation device 101 is intended to receive Japanese text as input and output a text translated into English, the text generation model 103 may be pre-trained to output such text for the input text.

[0022] In this embodiment, we assume that an autoregressive generative language model is used as the text generation model 103. The autoregressive generative language model is a type of model that generates text by referring to the input text and past word sequences generated by the model, and sequentially generating the next word.

[0023] Autoregressive generative language model architectures can be broadly classified into two types: encoder-decoder and decoder-only. The text generation model 103 of this embodiment can use either of these two architectures or other architectures.

[0024] Figure 2 illustrates a common text generation method using an encoder-decoder format, that is, a method for generating only one text from an input text.

[0025] The input text is split into words and then fed to an encoder to obtain a distributed representation of the input text. The decoder receives the distributed representation of the input text and the past word sequences generated by the model, and outputs the selection probability for each word candidate.

[0026] Here, word candidates refer to all the vocabulary registered in the text generation model 103. The selection probability for a word is the probability that each word will be selected as the next word in a sequence of words following the previous word sequence. The selection probability for each word candidate is referenced, and the word with the highest probability, a random word based on the selection probability, or a word selected by another method is generated as the word for the next step.

[0027] A token that indicates the beginning of the text (in this patent, " <bos>The next word is generated sequentially from the first word (in this patent, the next word is "), and a token indicating the end of the text is generated. <eos>This process is repeated until the following is generated: The resulting word sequence becomes the output text. In the example in Figure 2, first the input text and <bos>It receives the input text and generates word 1. <bos>It takes word 1 and generates word 2. Then it continues with word 3, word 4, ... <eos>And so words are generated sequentially, <eos>The word sequence leading up to the generation of the output text will be used as the output text.

[0028] Figure 3 illustrates a typical text generation method using a decoder format.

[0029] The entire text corresponding to the input text is generated by providing the input text to the decoder and sequentially generating the words that follow it, similar to the encoder-decoder format. In the example in Figure 3, the input text generates word 3, word 4, word 5, word 6, ... <eos>The process continues, and this word sequence is used as the output text.

[0030] In this embodiment, "word" does not refer to a word in its general usage, but may refer to any token that exists in a predefined vocabulary database. In other words, the text "building a stable economy" may be treated as the word sequence "stable / economic / construction", or as the word sequence "stable / economic / construction", or as the word sequence "stable / economic / construction".

[0031] Figure 4 is a flowchart illustrating an example of the flow performed by the text generation device 101.

[0032] In step 402, the text generation device 101 receives input.

[0033] In step 403, the input text and past word sequence are given to the text generation model 103 to obtain the selection probability for the word candidates in the next step. For example, if the input text is "In order to improve the economy" and the past word sequence is "Monetary policy reform is necessary, and in order to do so", then the selection probability of the word following the past word sequence might be 50% for "stability", 10% for "interest rate", and 10% for "government ministries".

[0034] In step 404, the word with the highest probability of being selected is chosen from the list of word candidates for the next step. In the example above, this means selecting "stability" as the word following "monetary policy reform is necessary, and for that purpose..."

[0035] In step 405, for each word candidate with a high probability of selection from the second highest, it is checked whether the text branching condition is met. For words that meet the condition, the text generation model is used to generate the rest of the text of that word as a text termination token. <eos>Generate until it appears.

[0036] In the example above, if the word "interest rate" satisfies the text branching condition, the text following the past word sequence "Monetary policy reform is necessary, and for that, interest rates" will be generated using the text generation model 103 to produce a text-end token. <eos>Generate until it appears.

[0037] As a result, if, for example, the text obtained is "Monetary policy reform is necessary, and for that, interest rates need to be raised," the generated text is recorded in the control and recording unit 102. The branching probability for each branch text is also recorded in the control and recording unit 102.

[0038] A text branching condition is a condition used to determine whether or not to branch the text based on each word candidate, excluding the word with the highest probability of selection. An example of a text branching condition is given below.

[0039] For example, text branching occurs when the selection probability of a word candidate is above a fixed probability (e.g., 10% or more). For example, text branching occurs when the selection probability of a word candidate is above a certain multiplier of the selection probability of the word with the highest selection probability (e.g., 1 / 3 or more of the highest selection probability). For example, text branching occurs for a specific number of word candidates with high selection probabilities (from the second highest-selection word candidate to the sixth highest-selection word candidate).

[0040] For example, text branching will not be performed for words of certain parts of speech, such as particles, auxiliary verbs, adjectives, and adverbs. For example, text branching will not be performed for word candidates that are similar to the word in the next step (the word candidate with the highest probability of selection) (e.g., "increase" versus "increase"). The similarity between words here may be determined using character-based matching scores, cosine similarity of word embeddings, or an external dictionary that records combinations of similar words.

[0041] Among the text branching conditions shown above, a combination of multiple conditions may also be treated as a text branching condition.

[0042] In step 406, determine if the word in the next step is a text-ending token. If not, return to step 403. If yes, proceed to step 407.

[0043] In step 407, the generated text that was terminated in step 406 by the appearance of a text termination token is recorded, and the control / recording device 102 structures and outputs the set of generated texts that it has recorded.

[0044] In addition to the output described above, the input text, the branching probability of each text at the branching point, or both may also be output. Figures 5 and 6 show an example of a structured output of the set of generated texts, the input text, and the branching probability of each text at the branching point.

[0045] The text generator may choose not to output some of the generated text so that the total number of generated texts output by the device remains below a specified number. Possible methods for selecting which texts to not output include, for example, not outputting branching texts with a low branching probability, or not outputting branching texts that branched far from the input text (i.e., branching texts that branched near the end of the sentence).

[0046] In the above flowchart, the selection probability of each word is treated as the branching probability at the branching point, but other calculation methods may be used. For example, the likelihood of the text sequence from the branching point to the text end token (the product of the word selection probabilities at each step) may be treated as the branching probability at the branching point.

[0047] The branch probability output by the text generator 101 may be expressed in a different way. For example, if the actual output probability is 53.0234388%, it may be rounded to the third decimal place and written as "53.02%". For example, if the output probability is 0%-10%, it may be written as "low", if it is 11%-30%, it may be written as "medium", and if it is 30%-100%, it may be written as "high".

[0048] Figure 5 shows an example of the output format in a news article generation task generated by the text generation device 101. An example of the generation flow in the example shown in Figure 5 is described below.

[0049] First, the input text is given as "In order to improve the economy". The word with the highest probability of being selected as the next word after this input, "finance", is selected as the word for the next step. In addition, "fiscal policy" and "poor" are selected as words that satisfy the branching conditions. For each word that satisfies the branching conditions, word generation is performed sequentially until a text termination token is generated, and the generated text is recorded in the control / recording unit 102.

[0050] In the example above, the following text is generated following "Fiscal Policy": "Reducing the fiscal deficit is essential, and austerity measures must be implemented..." and the following text is generated following "Non-performing Loans": "Dealing with non-performing loans is essential, but financial institutions..." (Some of the generated text is represented with "..." due to character limits.) Next, let's assume that the words following "finance," which has the highest selection probability, are generated sequentially in the order of "policy," "reform," "but," "necessary," "and," "," "that," "for," and "in order to." These words are the words that had the highest selection probability in each step, following the input text and the sequence of past steps. In the example in Figure 5, this means that no text branching occurred during the generation of this word sequence.

[0051] For example, text branching could occur if the word "policy" is followed by "wo" and "important" is followed by "ga". However, in the example in Figure 5, no text branching occurred because none of the words met the text branching conditions.

[0052] Subsequently, "safety" is selected as the word with the highest probability of being selected as the next word after "monetary policy reform is necessary, and for that purpose," and "interest rate," "each," and "Bank of Japan" are selected as words that satisfy the branching condition. Words that satisfy the branching condition are sequentially generated until a text-end token is generated, and the generated text is recorded in the control / recording unit 102.

[0053] The subsequent text is generated by sequentially generating the words that follow the word "safety," which is the word for the next step. In the example in Figure 5, no further text branching occurs and a text termination token is generated, so generation ends at that point and the generated text is recorded in the control / recording unit 102. The input text, the generated text recorded by the control / recording device 102, the branching position, and the branching probability of each text at the branching position are output in a structured manner.

[0054] Figure 6 shows an example of the output format in the translation text generation task, generated by the text generation device 101.

[0055] As shown in Figure 6, the generated text may be omitted, rather than displaying all characters up to the end of the sentence. For example, in tasks such as translation text generation, where only the words at branching points differ and the subsequent text is often identical, it is possible to omit the identical parts.

[0056] In the example in Figure 6, "Japan's economy needs to be revitalized." is output as the branched text instead of "The Japanese economy needs to be revitalized.", but the part "needs to be revitalized." is the same, so it is written as "<same below>".

[0057] In the flow shown in Figure 4, text branching is not performed for text generated by branching due to the fulfillment of the text branching conditions, but further additional branching may be performed from the branched text.

[0058] In the example shown in Figure 7, given the input text "In order to improve the economy," "finance" is selected as the word with the highest probability of selection, and "public finance" is selected as the word that satisfies the branching condition. The text that follows "Public finance" is then generated as "Reducing the budget deficit is essential, but..." Furthermore, in the process of generating this branched text, "opposition party" is selected as the word with the highest probability of selection, and "austerity" is selected as the word that satisfies the branching condition. In the embodiment shown in Figure 7, the text that follows "austerity" which satisfies the branching condition is also generated.

[0059] Whether or not to perform a second or subsequent text branch from the text generated by branching may be considered a text branching condition, or one of the elements that constitute a text branching condition. Whether or not to use this condition as a text branching condition may be accepted as a parameter of the text generation device 101 at the time of input.

[0060] Instead of structuring and displaying a set of generated texts corresponding to the input text, it is also possible to generate branched texts for the input text, thereby generating a set of texts that are different variations of the input text—that is, texts in which parts of the input text are replaced with other strings—and then structure and output them.

[0061] In other words, at each word position in the input text, the system extracts words from the list of word candidates that follow that word, excluding the word that follows in the original input text, that satisfy the text branching condition. Text branching is then performed using these words to generate the subsequent text. Finally, the set of input text and branched text is structured and displayed.

[0062] In the example in Figure 8, the text generator 101 is given the input text "Monetary easing is necessary to prevent economic deterioration." The word following "to prevent economic deterioration" in the input text is "finance," but the words selected to satisfy the branching conditions are "Japan," "fiscal policy," and "weak yen." For each of these words, the generator generates the text that follows it.

[0063] Furthermore, the word following "To prevent economic deterioration, finance" in another part of the input text is "easing," but among the word candidates, "policy" and "tightening" are selected as words that satisfy the branching conditions. For each of these words, subsequent text is generated. The text set generated by the above procedure and the input text are structured and displayed in a format such as Figure 8 as an example.

[0064] In this embodiment, the branching probability may be higher than the selection probability of the following word in the original input text. In the example in Figure 8, the selection probability of the word "Japan" is 25%, which is higher than the selection probability of the word "finance," which is the following word in the original input text, which is 15%. Figure 9 shows an example of the user interface of the text generation device 101.

[0065] When input text is entered into the input text box 901 and the generate button 902 is pressed, a structured set of generated text is output to the output box 903. As shown in Figure 9, various parameters handled by the text generation device 101 may also be accepted as input.

[0066] Figure 9 accepts input parameters such as whether to display the input text and branching probability, the upper limit of the total number of texts to output, and the lower limit of the probability of text branching at branching points. As shown in Figure 9, the input text, the branching probability of each text at branching points, or both may be output.

[0067] In the user interface of the text generation device 101 shown in Figure 9, input text is received via the input device 1004 in Figure 10, and the structured generated text set is displayed on the output device 1005. In this way, the contents of the screen in Figure 9 are displayed on the output device 1005.

[0068] Figure 10 shows an example of the hardware configuration of the text generation device 101. The text generation device 101 can consist of a processor 1001, memory 1002, auxiliary storage device 1003, input device 1004, output device 1005, and communication device 1006. This is just one example of a hardware configuration, and other configurations capable of operating the data generation device may be used.

[0069] Figure 11 is a diagram illustrating the overview of the text generation system.

[0070] In the text generation system shown in Figure 11, the input / output terminal 1101 is connected to the text generation device 101 shown in Figure 1.

[0071] In the text generation device 101 shown in Figure 1, input text is received via the input device 1004 shown in Figure 10, and the structured generated text set is displayed on the output device 1005. In other words, the content of the screen shown in Figure 9 is displayed on the output device 1005.

[0072] In contrast, in the text generation system shown in Figure 11, input text is received via the input / output terminal 1101, and a structured set of generated text is displayed on the input / output terminal 1101. In other words, the contents of the screen shown in Figure 9 are displayed on the input / output terminal 1101.

[0073] According to the above embodiment, the text generation device can structure and display the set of text generated from the input text. [Explanation of Symbols]

[0074] 101 Text Generator 102 Control and Recording Unit 103 Text Generation Model 1004 Input device 1005 Output device 1101 Input / Output Terminal< / eos> < / eos> < / eos> < / eos> < / eos> < / bos> < / bos> < / eos> < / bos>

Claims

1. A text generation model that obtains multiple selection probabilities for multiple word candidates in the next step following the input text, based on the input text and past word sequences. From among the multiple word candidates in the next step, the word candidate having the first selection probability among the multiple selection probabilities is selected as the first word candidate in the next step. Determine whether the word candidate having a second selection probability among the plurality of selection probabilities satisfies the branching condition of the text, and select the word candidate that satisfies the branching condition as the second word candidate for the next step. Using the text generation model described above, a first text following the first word candidate and a second text following the second word candidate are generated, respectively. A control unit that outputs the generated first text and the generated second text as a set of generated texts, It has, The aforementioned past word sequence is, A text generation device characterized by being a past word sequence generated by the aforementioned text generation model.

2. The control unit, From among the multiple word candidates in the next step, the word candidate with the highest selection probability is selected as the first word candidate in the next step. The text generation device according to claim 1, characterized in that, as the second selection probability, it is determined whether the branching condition is satisfied for the word candidates with the second or subsequent highest selection probabilities.

3. The aforementioned text generation model is an autoregressive generative language model, The text generation device according to claim 1, characterized in that it sequentially generates the next word candidate by referring to the input text and the past word sequence generated by the text generation model, and outputs the selection probability for each of the word candidates.

4. The aforementioned text branching condition is, The text generation device according to claim 1, characterized in that it is a condition for determining whether or not to branch the text based on the word candidate having the second selection probability among the plurality of selection probabilities.

5. The control unit, The text generation device according to claim 1, characterized in that it controls the total number of texts to be generated so that it is less than or equal to a predetermined number.

6. The substring of the input text is input to the text generation model, The control unit, The text generation device according to claim 1, characterized in that it generates text which is a different variation of the input text.

7. The control unit, Generate branch text for the aforementioned input text, The text generation device according to claim 6, characterized in that it outputs a set of generated texts in which a part of the input text is replaced with the substring.

8. The subsequent step following the input text comprises a plurality of steps, The control unit, The text generation device according to claim 1, characterized in that it outputs the set of generated texts through the aforementioned steps.

9. The control unit, Accepts specified parameters as input, The text generation device according to claim 1, characterized in that it outputs the set of generated texts by referring to the parameters.

10. The text generation apparatus according to claim 1, further comprising an output device for displaying the set of generated texts.

11. The output device is The text generation device according to claim 10, characterized in that the first selection probability and the second selection probability at the branching position are displayed in the set of generated texts.

12. The output device is An input text box for entering the aforementioned input text, A generate button for generating the aforementioned set of generated texts, An output box that displays the set of generated texts, The text generation device according to claim 10, characterized in that it displays each of the following.

13. A text generation system comprising a text generation device according to claim 1 and an input / output terminal, The aforementioned input / output terminal is A text generation system characterized by displaying the collection of generated texts.

14. The aforementioned input / output terminal is The text generation system according to claim 13, characterized in that the first selection probability and the second selection probability at the branching position are displayed in the set of generated texts.

15. Using a text generation model, the selection probability for each generated word candidate is output sequentially based on the input text and past word sequences. Based on the selection probability of the generated word candidates at each step received from the text generation model and the text branching conditions for branching the text, the branched text is generated. Output the set of texts that have been generated, The aforementioned past word sequence is, A text generation method characterized by being a past word sequence generated by the aforementioned text generation model.

Images