A text generation method integrating dual language model and sentence detection
Through the bilingual model and sentence detection method, combined with the top_k method and N-Gram statistical model, the problems of low generation efficiency and poor sentence quality in text generation technology are solved, and text generation with correct grammar and reasonable semantics are achieved, which is suitable for creativity Text generation task.
Patent Information
- Application Number
- CN202111499830.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-12-09
AI Technical Summary
Existing text generation techniques have problems such as low generation efficiency, poor sentence randomness, poor coherence and weak semantic rationality. Especially in creative text generation tasks, the generated sentences are low in richness and poor readability.
The bilingual model and sentence detection method are used to synchronize the words in the front and back parts of the text, combine the top_k method and part-of-speech weighting method to improve the sampling method, and use syntactic analysis based on grammar regulations and N-Gram statistical model to check the rationality of sentences. , generate text that is syntactically correct and semantically reasonable.
It improves the efficiency and quality of text generation, makes the generated sentences closer to human sentence making, solves the problems of low sentence richness and poor readability, and is suitable for creative text generation tasks.
Smart Images

Figure CN114298010B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text generation, and in particular to a text generation method integrating a dual language model and sentence detection. Background Art
[0002] Text generation technology is an important branch in the field of natural language processing. It is widely used in specific tasks such as summary generation, machine translation, and dialogue systems. Text generation technology can be applied to real scenarios. For example, in scenarios such as article generation and poetry writing, text generation technology can be used to automatically generate text, greatly reducing manual costs; in scenarios such as news information extraction and comment analysis, text generation technology can be used to quickly obtain key sentences; in scenarios such as intelligent dialogue and machine translation, text generation technology can be used to improve the efficiency of human-computer interaction.
[0003] Common text generation technologies can be roughly divided into two categories: rule-based text generation technology and deep learning-based text generation technology. Rule-based text generation technology mainly relies on rule vocabularies. It is more commonly used in strong rule usage scenarios (such as news summaries, financial report generation, etc.) and has high generation efficiency. However, the modification and expansion of rule vocabularies are more troublesome and require a lot of manual construction. Deep learning-based text generation technology can realize process automation and greatly reduce labor costs. Its intelligence is much higher than rule-based text generation technology and is widely used in more complex scenarios (such as intelligent dialogue, machine translation, poetry writing, etc.). Because of the ambiguity and polysemy of natural language itself, generating text that conforms to human cognition is still a huge challenge.
[0004] There are some solutions. For example, the prior art proposes a rule-based text generation method, which first locates the keywords input by the user, obtains the paragraph S1 containing the keywords from the text corpus through forward and reverse retrieval, and then uses text similarity matching to screen out the paragraph S2 that is most similar to the text content of S1. The above matching operation is repeated until the total number of obtained paragraphs reaches the specified number, and finally all paragraphs are reorganized to generate an article. This method involves manual corpus collection, data processing, and index table construction, so the text generation process is time-consuming and labor-intensive, and the text generated by this method is composed of paragraphs in a fixed corpus, so the text content is limited and the richness is low.
[0005] The paper "A Neural Network Model for Chinese Sentence Generation with KeyWord" proposes a keyword-based text generation method, which takes keywords as input and expands keywords bidirectionally (forward and backward) until a sentence is generated. This method uses two LSTM models. The front LSTM model supplements the words on the right of the keyword to generate a sentence S1, and the rear LSTM model supplements the words on the left of S1 to generate S2. The two LSTM models are used alternately to expand words until a sentence is generated. When predicting the next character, the deep learning model uses maximum sampling or random sampling. The maximum sampling method results in a high repetition rate of the generated sentences, and the random sampling method results in strong randomness, poor coherence, and weak semantic rationality of the generated sentences. In addition, it is difficult to ensure syntactic correctness and semantic rationality when generating text through a deep learning model. Summary of the invention
[0006] In order to overcome the defects and shortcomings of the prior art, the present invention provides a text generation method integrating a dual language model and sentence detection, which uses a dual language model trained by a neural network model to synchronously expand the words in the front and back parts of the text, thereby improving the defect of slow cross-iteration generation speed in the previous technology, and improving the shortcomings of poor randomness and poor coherence of sentences in the traditional random sampling method by using the top_k method and the part-of-speech weighting method, and further checking the rationality of the generated sentences by combining a grammar-restricted syntactic analysis method and a Tri-Gram statistical model-based semantic analysis method, so that the generated sentences are closer to human sentence formation.
[0007] In order to achieve the above object, the present invention adopts the following technical solutions:
[0008] The present invention provides a text generation method integrating a dual language model and sentence detection, comprising the following steps:
[0009] Get the input keyword, and language model M1 and language model M2 synchronously expand the words before and after the keyword;
[0010] The language model M1 predicts the next character forward to obtain a new text S1, the language model M2 predicts the next character backward to obtain a new text S2, and the new text S1 and the new text S2 are concatenated to obtain a sentence;
[0011] Determine whether the sentence meets the output conditions. If it meets the output conditions, output the sentence; otherwise, use it as a new input.
[0012] The syntactic analysis method based on the specification conducts structural analysis on the sentence. If the sentence can be reduced to the grammatical start symbol, the sentence conforms to the syntactic rules.
[0013] The N-Gram statistical model is used to calculate the language score of candidate sentences that meet the syntactic rules. If the language score of a candidate sentence exceeds the set threshold, the candidate sentence is output as the final text.
[0014] As a preferred technical solution, the language model M1 and the language model M2 synchronously expand the words before and after the keyword, and the specific steps include:
[0015] Input keywords to the trained language models M1 and M2. Language models M1 and M2 predict the next character respectively, and obtain the original probability distribution of the next character.
[0016] Use the top_k method to select characters and their corresponding prediction probabilities, and update the prediction set;
[0017] Generate a part-of-speech proximity table through the training set, obtain the part-of-speech proximity values of the corresponding part-of-speech combinations in the updated prediction set through the part-of-speech proximity table, and multiply the part-of-speech proximity values by the original probability to obtain a new probability distribution;
[0018] Redistribute the probability distribution and randomly sample the next character nw and the next character fw;
[0019] The input keyword kw is concatenated with the randomly sampled character nw and character fw to obtain the concatenated sequence kw+nw and the concatenated sequence fw+kw;
[0020] The prediction results of language model M1 and language model M2 are concatenated to obtain text S=fw+kw+nw.
[0021] As a preferred technical solution, the steps of generating a part-of-speech table through a training set include:
[0022] For part of speech pos, part of speech s k The probability that the word follows the part-of-speech pos word is:
[0023]
[0024] Among them, num k Indicates part of speech combination pos+s k The number of times it appears in the training set, n represents the number of parts of speech, and finally a part-of-speech adjacent value table Q of size n*n is generated.
[0025] As an optimal technical solution, the probability distribution is redistributed, and the specific calculation formula is:
[0026]
[0027] Among them, π(w k) represents the probability value after redistribution, temperature represents the scale factor, w k Indicates updating the candidate words in the prediction set, p(w k ) represents the probability corresponding to the candidate word.
[0028] As a preferred technical solution, the grammatical analysis method based on the specification performs structural analysis on the sentence, and the specific steps include:
[0029] Mark the parts of speech of all characters in the sentence. Each character has a corresponding non-terminal symbol. Reduce the sentence according to the grammar G. If the sentence can be reduced to the grammar start symbol S, the sentence conforms to the syntactic rules.
[0030] As a preferred technical solution, the syntactic analysis method based on the protocol performs structural analysis on the sentence, and adopts the CYK algorithm based on the CFG rule to construct a recognition matrix of size [n+1, n+1], where n is the total number of words in the input sentence Sen.
[0031] As a preferred technical solution, the recognition matrix construction step includes:
[0032] Definition i,j To identify the value of coordinate [i,j] in the matrix, construct the main diagonal t 0,0 =0,t 1,1 to n,n Put the words w of the input sentence in sequence i ;
[0033] Construct the diagonal t above and to the right of the main diagonal 0,1 to n-1,n , where t 1,1 to k-1,k =G(w k ), where G(w k ) indicates that w is derived from grammar G k The non-terminal symbol of ;
[0034] Construct the remaining diagonal above and to the right of the main diagonal, for t i,j , if there exists a production A→t in the grammar G i,j- 1 t i+1,j , then t i,j =A, otherwise t i,j =t i,j-1 .
[0035] As a preferred technical solution, the N-Gram statistical model is used to calculate the language score of the candidate sentences that meet the syntactic rules. Specifically, the Tri-Gram model is used to obtain the probability of the sentence appearing by calculating the product of the probability of each word appearing.
[0036] As a preferred technical solution, the language model M1 and the language model M2 have the same structure and adopt any one of the neural network models LSTM, GRU or Transform.
[0037] The present invention also provides a text generation system integrating dual language models and sentence detection, including: a keyword acquisition module, a language model M1, a language model M2, a splicing module, a sentence generation condition judgment module, a sentence structure analysis module, a semantic analysis module, and a text output module;
[0038] The keyword acquisition module is used to acquire input keywords, and the language model M1 and the language model M2 synchronously expand the words before and after the keywords;
[0039] The language model M1 predicts the next character forward to obtain a new text S1, and the language model M2 predicts the next character backward to obtain a new text S2;
[0040] The splicing module is used to splice the new text S1 and the new text S2 to obtain a sentence;
[0041] The sentence generation condition judgment module is used to determine whether the sentence meets the output condition. If the output condition is met, the sentence is output, otherwise it is used as a new input;
[0042] The sentence structure analysis module is used to perform structural analysis on the sentence based on a syntactic analysis method based on the specification. If the sentence can be reduced to the grammatical start symbol, the sentence complies with the syntactic rules;
[0043] The semantic analysis module is used to calculate the language score of the candidate sentences that meet the syntactic rules using the N-Gram statistical model;
[0044] The text output module is used to output the final text. If the language score of a candidate sentence exceeds a set threshold, the candidate sentence is output as the final text.
[0045] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0046] (1) The present invention generates grammatically correct and semantically reasonable text based on keywords through a dual language model and a sentence detector. In the dual language model, a dual model synchronous generation mechanism is adopted to improve the problem of low generation efficiency of the alternating generation method. In the sampling strategy, the random sampling method in the prior art has the disadvantages of poor randomness and poor coherence of sentences, and its defects are improved by the top_k method and the part-of-speech weighting method. In the sentence detector, a syntactic analysis method based on grammar constraints and a semantic analysis method based on the N-Gram statistical model are adopted to detect the syntactic rationality and semantic rationality of sentences, so that the generated sentences are closer to human sentence formation. The present invention is applied to creative text generation tasks such as sentence construction and article generation, and can effectively solve the problems of low richness and poor readability of sentences generated by previous generation technologies.
[0047] (2) The present invention uses the trained Tri-Gram language model to perform semantic scoring for the generated text, further ensuring the quality of the generated text. In the use stage, the model generated based on the open source model KenLM is directly called to quickly calculate the text score and improve efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 A flowchart of a text generation method integrating a dual language model and sentence detection according to the present invention;
[0049] Figure 2 A schematic diagram of a sentence generation process of the present invention;
[0050] Figure 3 It is a flow chart of the parts before and after the expansion of keywords in the sentence generation of the present invention;
[0051] Figure 4 It is a schematic diagram of the sentence detection process of the present invention;
[0052] Figure 5 is a schematic diagram of the grammar rules of the present invention;
[0053] Figure 6 It is a schematic diagram of the recognition matrix of the present invention;
[0054] Figure 7 It is a schematic diagram of the recognition matrix of a specific application example of the present invention. DETAILED DESCRIPTION
[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0056] Example 1
[0057] like Figure 1 , Figure 2 As shown, this embodiment provides a text generation method integrating dual language models and sentence detection, which mainly includes two steps: ① dual language model generates sentences; ② sentence detector detects sentences. In step ①, the two language models synchronously expand the characters before and after the keywords, splice them into new text, re-input them into the two models, and continue to expand, and repeat until a candidate sentence is generated; in step ②, by analyzing the sentence structure, if the sentence can be successfully reduced to the grammar start symbol S, the sentence satisfies a reasonable syntactic structure; then the N-Gram model scores the candidate sentences, and judges whether the sentence meets the semantic rationality by whether the score exceeds the threshold.
[0058] In this embodiment, the dual language model generates sentences: the two language models M1 and M2 synchronously expand the forward and backward characters of the keyword, and perform text splicing after each expansion, and then determine whether the spliced text meets the conditions for outputting the sentence. If it does, it will be output. Otherwise, the generated result of this time is used as the new input of the two models to perform a new round of generation tasks, and this cycle is repeated until a candidate sentence is generated;
[0059] In this embodiment, the two language models are neural network model structures, which are trained by known data sets. When the training set is determined, the construction and training of language models M1 and M2 determine the sentence generation effect. The two models have the same structure, and the only difference is that the construction of the training samples is different, so the knowledge learned by the two models from the training data is inconsistent. As long as the two models meet the sequence-to-sequence input and output structure, they are not limited to neural network models such as LSTM, GRU, and Transform.
[0060] In this embodiment, constructing training samples for the bilingual model includes two steps: data preprocessing and constructing data pairs (x, y) that conform to the model structure. Assuming that there is a data set D, which contains several texts, the main process of constructing training samples is as follows:
[0061] In the data preprocessing stage, we first add the start and end markers for each text, denoted as “Begf” and “Endf”; then fill each text with a fixed length and vectorize the text, denoted as S={x 0 ,x 1 ,...,x t-1 ,x t}.
[0062] In the data pair construction phase, different training data sets are constructed for the two languages, training data set D1 is constructed for model M1, and training data set D2 is constructed for model M2.
[0063] In D1, for a text S = {x 0 ,x1 ,...,x t-1 ,x t}, which is converted into data pairs that conform to the model input and output in the form of:
[0064] (x, y) = ({x 0 ,x 1 ,...,x t-1},{x 1 ,...,x t-1 ,x t})
[0065] In D2, for the text S = {x 0 ,x 1 ,...,x t-1 ,x t}, first reverse to S r ={x t ,x t-1 ,...,x 1 ,x 0} and then transformed into a data pair that conforms to the model input and output as (x, y) = ({x t ,x t-1 ,...,x 1},{x t-1 ,...,x 1 ,x 0}).
[0066] The constructed training data is used for training language model M1 and language model M2. After training, language model M1 and language model M2 are saved locally and can be directly called in the subsequent generation stage.
[0067] The specific steps include:
[0068] S1: Input keyword kw, which is the original input of the text generation task. The purpose of the present invention is to generate a sentence containing the keyword. In order to ensure the generation effect, the keyword kw must be a word with a frequency greater than 1000 in the dictionary (the dictionary generated based on the training set). Because in the model training process, words with high frequency are learned more times, and the model learns more knowledge from them. In the generation stage, the sentences generated based on these high-frequency words are more reasonable in terms of syntax, semantics, etc.
[0069] S2: The trained language models M1 and M2 synchronously expand the words before and after the keyword kw. This embodiment proposes a dual language model synchronous expansion technology to complete the keyword-based sentence generation task. Compared with the method of expanding the parts before and after the keyword by alternate iteration in the prior art, the synchronous expansion greatly speeds up the generation rate;
[0070] like Figure 3As shown, the steps of expanding the keyword kw by the language model M1 specifically include:
[0071] S21: The input of the model is a known text sequence (i.e. keyword) kw.
[0072] S22: The known text sequence is input into the trained language model M1 to obtain a prediction set P, which stores all possible predictions of the next word and the corresponding prediction probability of each word, which can be expressed as:
[0073] P = {w 1 :p 1 ,w:p 2 ,...,w n :p n}
[0074] where w k is the predicted character, p k is the predicted probability. Where n is the total number of words in the training set.
[0075] S23: The size of the prediction set is determined by the size of the training set dictionary (which stores all the words and serial numbers that have appeared in the training set). The size of the text training set ranges from a few thousand to hundreds of thousands, and the total number of words ranges from thousands to tens of thousands. In order to alleviate the problem of generating uncommon words, each time we sample the top t words with the highest probability. Therefore, the prediction set is updated to:
[0076] P1={w 1 :p 1 ,w:p 2 ,...,w t :p t}
[0077] In the present invention, the calculation formula of t is proposed as follows:
[0078]
[0079] Where n is the total number of words in the training set.
[0080] S24: The present invention proposes the concept of part-of-speech proximity, which measures the probability of words of different parts of speech being adjacent in a text. Compared with other languages, English text has a strong regularity, for example, a verb is followed by a noun, an article, etc., but not a verb. In order to reduce the probability of such errors in random sampling, adjacent part-of-speech values are added to the probability distribution, making the sampling process more reasonable.
[0081] In this embodiment, a fixed part-of-speech adjacent value table Q is generated through the training data set. Specifically, for part-of-speech pos, part-of-speech s k The probability that the word follows the pos part-of-speech word is:
[0082]
[0083] where num k It means "pos+s k "The number of times this part-of-speech combination appears in the training set, and n represents a total of n parts of speech. Finally, a part-of-speech adjacent value table Q of size n*n is generated.
[0084] In this step, for all candidate words w in probability distribution P1 k , obtain "kw+w" through Q k "The corresponding part-of-speech combination's part-of-speech value q k , the new probability value Thus, we get the updated prediction set P * .
[0085] S25: After a series of operations in the previous steps, the probability values in the prediction set no longer meet the regularization requirements. * The probability value in is recalculated, and the calculation formula is:
[0086]
[0087] Temperature is a scaling factor used to weight the probability distribution. The closer the temperature is to 0, the more deterministic the sampling is (i.e., the more likely the word with the highest probability is to be selected). A temperature of 1 indicates that each word will be selected according to the model output probability. In the experiment, the word selection results are more coherent when the temperature is 0.2 than when the temperature is 1. In order to balance the richness and coherence of the generated text, the temperature value is optimally controlled in [0.2, 0, 6].
[0088] After calculating the final probability distribution, randomly sample the next word nw.
[0089] The present invention selects top_k words and calculates probability values by weighting part-of-speech probabilities, so that the probability distribution is more reasonable, the uncertainty caused by random sampling is reduced, and the generated sentences are more reasonable.
[0090] S26: The input known sequence and the sampled words are concatenated to obtain kw+nw.
[0091] The expansion steps of language model M2 are the same as those of language model M1, except that in step S26, the concatenation is to concatenate the sampled word in front of kw. Assuming that the word predicted and sampled by M2 is fw, the concatenated sequence is fw+kw.
[0092] S3: Concatenate the prediction results of language model M1 and language model M2 to obtain text S = fw + kw + nw.
[0093] S4: Determine whether the text S meets the output conditions. When the length of the generated character sequence is greater than the maximum value (the maximum value ε=60 is set in advance, and the ε value depends on the specific task and is a manually adjusted hyperparameter) or the last character of the text S is END, the algorithm terminates and the result is output. Otherwise, the text S is regarded as a known sequence and goes to step S2 to continue expanding the text S.
[0094] Sentence detector detects sentences: In the sentence detector, the syntactic analysis method based on reduction and the semantic analysis method based on the N-Gram statistical model are combined to detect the syntactic rationality and semantic rationality of the sentence. In the above steps, a candidate sentence has been generated through the synchronous expansion of the two language models. The detection link is divided into two parts: one is to analyze the sentence structure. If the sentence can be successfully reduced to the grammar start symbol S, the sentence satisfies the reasonable syntactic structure; the second is to use the N-Gram statistical model to score the candidate sentence, and judge whether the sentence satisfies the semantic rationality by whether the score exceeds the threshold. If and only if the candidate sentence passes the two-step detection, it will be output as the final result, otherwise the generated sentence is invalid.
[0095] like Figure 4 As shown, the workflow of the sentence detector specifically includes:
[0096] S5: First, mark the part of speech of all words in the sentence Sen, and each word has a corresponding non-terminal symbol. Then, Figure 5 As shown, according to the grammar G, if the sentence can be successfully reduced to the rule start symbol S, the sentence Sen passes the syntactic rule detection. Otherwise, the generated sentence Sen is invalid and the sentence generation part needs to be re-executed.
[0097] in, Figure 5 The grammar shown is the syntactic rules commonly used in syntactic analysis, covering the syntactic structure of most English texts.
[0098] The present invention checks the rationality of sentence structure through syntactic analysis. Syntactic analysis is one of the basic tasks of natural language processing, but there are few related technologies for text generation at present. The present invention covers most of the syntactic structures of simple English sentences by constructing reasonable grammar rules.
[0099] In the reduction method, the CYK (Coke-Younger-Kasami) algorithm is used. The CYK algorithm is an analysis algorithm based on CFG rules and is a bottom-up analysis algorithm. Figure 6As shown, the CYK algorithm needs to construct a recognition matrix of size [n+1, n+1], where n is the total number of words in the input sentence Sen. The composition of the recognition matrix is as follows:
[0100] ① All elements below the diagonal are 0;
[0101] ②The elements above the main diagonal are composed of non-terminal symbols of grammar G;
[0102] ③The elements on the main diagonal are composed of the terminal symbols (words) of the input sentence Sen.
[0103] Definition i,j To identify the value of the coordinate [i, j] in the matrix, the steps to construct the identification matrix are as follows:
[0104] ①Construct the main diagonal, t 0,0 =0,t 1,1 to n,n Put the words w of the input sentence Sen in sequence i .
[0105] ②Construct the diagonal t above and to the right of the main diagonal 0,1 to n-1,n , where t 1,1 to k-1,k =G(w k ), where G(w k ) indicates that w can be derived from grammar G k A→w k , then G(w k )=A.
[0106] ③Construct the remaining diagonal to the upper right of the main diagonal. For t i,j , if there exists a production A→t in the grammar G i,j- 1 t i+1,j , then t i,j =A, otherwise t i,j =t i,j-1 .
[0107] ④ The necessary and sufficient condition for judging the legitimacy of sentence Sen is t 0,n =S.
[0108] S6: The Tri-Gram model calculates the language score of Sen and judges the rationality of the text. If the score exceeds the specified threshold, Sen is reasonable, otherwise Sen is invalid.
[0109] N-Gram is an algorithm based on statistical language model, including Bi-Gram and Tri-Gram. The basic idea is to perform a sliding window operation of size N on the byte level on the content of the text, forming a sequence of byte segments of length N, each of which is called a gram. The model is based on the assumption that the appearance of the Nth word is only related to the previous N-1 words, but not to any other words, and the probability of the whole sentence appearing is the product of the probability of each word appearing. The Tri-Gram model is used in the present invention, that is, the probability of each word appearing is related to the previous two words. Assume that the text is represented by Sen = {w 1 ,w 2 ,...,w n}, then the calculation formula of Tri-Gram is expressed as:
[0110] P(Sen)=P(w 1 w 2 ... n )=p(w 1 )*P(w 2 |w 1 )*(w 3 |w 2 w 1 )...*P(w n |w n-1 w n-2 )
[0111] Each term in the formula represents the conditional probability of the keyword, which is calculated using maximum likelihood estimation:
[0112]
[0113] In the present invention, the use of a dual language model can generate texts with high richness and high readability, but the rationality of the text grammar cannot be guaranteed. Therefore, the grammatical rationality of the text is evaluated through a language model. Specifically, the text is scored using a trained Tri-Gram language model. The higher the score, the more the text conforms to human language habits and the stronger the grammatical rationality. In the present invention, the text score exceeds the threshold It means the text passes the semantic check.
[0114] The dataset for training Trim-Gram can be the same as that for training the bilingual model, or it can be other high-quality English datasets. The open source statistical language model training tool KenLM is used to train the dataset and generate the fallback weights of keywords and conditional probabilities of Tri-Gram in the language model. After installing KenLM through git, the model can be trained directly through command line instructions in the build file.
[0115] The threshold The value is calculated by random sampling. Specifically, random sampling is used to randomly sample 100 samples from the dataset used to train the dual language model, and the trained Tri-Gram model is used to score the 100 samples, and the lowest score is recorded. The sampling is repeated 100 times, and the scores obtained from the 100 samples are averaged to obtain the threshold. The text score in the corpus represents the Tri-Gram score of normal text. Taking the lowest score is to obtain the dividing line between semantically reasonable samples and illegal samples as much as possible. Repeated sampling can reduce the uncertainty of random sampling.
[0116] The present invention uses the trained Tri-Gram language model to perform semantic scoring for the generated text, further ensuring the quality of the generated text. In the use stage, the model generated based on the open source model KenLM is directly called to quickly calculate the text score and improve efficiency.
[0117] In this embodiment, the data set used for training the model in the keyword-based text generation method includes but is not limited to the data set size and form mentioned in this embodiment, and the network structure, parameters, loss function, activation function, etc. used do not affect the effectiveness of the text generation method in the present invention; the model used for semantic scoring is not limited to the Tri-Gram model, and reasonable training using the N-Gram model can achieve the same effect, and N can be any reasonable natural number, but generally 2 (Bi) or 3 (Tri). The method for determining the threshold is not limited to the average of the minimum value taken by repeated random sampling.
[0118] The method of the present invention is described using a specific example. Assuming that the training data set is Amazon Reading Reviews, two LSTM neural network models M1 and M2 and a Tri-Gram model TG are trained using the data set. The part-of-speech adjacent value table Q is calculated by the training set. The threshold value is described in step S6. The random sampling calculation method is used to obtain
[0119] ① Input sequence kw=reading.
[0120] ②M1’s first prediction generates a prediction set P = {End: 0.1985, books: 0.2332, story: 0.1750, happy: 0.0020, he: 0.0001, …}.
[0121] Take the top t characters (here we assume t=5) in probability ranking and their probability values, P1={books:0.2332, End:0.1985, story:0.1750, lovely:0.1520, romantic:0.0999}.
[0122] Reading books is a combination of "n+n" parts of speech, and the corresponding value in table Q is 0.12, that is, q1 = 0.12. Similarly, the q values of other words in P1 are obtained, and q2 = 0.36, q3 = 0.15, q4 = 0.22, and q5 = 0.19. Multiply the q value with the corresponding element in P1 to get the new prediction set P * ={books:0.0280, End:0.0715, story:0.0262, lovely:0.0334, romantic:0.0190}.
[0123] According to the formula given in step S25, temperature = 0.3, for P * Recalculate and get the final probability distribution P = {books: 0.1271, Endf: 0.5054, story: 0.1203, lovely: 0.1763, romantic: 0.0709}
[0124] Assume that the next word randomly sampled is Endf (sentence end marker). After concatenation, we get readingEndf.
[0125] ③ Similarly, while M1 predicts the next character, M2 predicts the previous character of kw according to step ②, and gets loves, which is concatenated to get loves reading Endf. The sentence length is less than 60, and the first word is not Startf, which does not meet the output conditions described in step S4, and the text continues to be expanded.
[0126] ④ Take "loves reading Endf" as input and proceed to steps ② and ③. Since Endf is the end-of-sentence marker, it is skipped directly when executing step ② and no words after Endf are generated. Step ③ is executed to only expand the sentence forward and obtain the new text "She loves reading Endf". The sentence length is less than 60, and the first word is not Startf, which does not meet the output condition described in step S4, and the text is continued to be expanded.
[0127] ⑤ Take "She loves reading Endf" as input and proceed to steps ② and ③. Since Endf is the end-of-sentence marker, it is skipped directly when executing step ② and the word after Endf is not generated. Step ③ is executed to only expand the sentence forward and obtain the new text "Startf She loves reading Endf". The starting word is Startf and the ending word is Endf, which meets the output condition described in step S4, and the generated sentence Sen is "She loves reading".
[0128] The above steps complete the sentence generation work, and the next step is the sentence detection part.
[0129] Step ⑥ Determine whether the sentence conforms to the syntactic rules through convention. First, the parts of speech of "She loves reading" are P (pronoun), V (verb), and N (noun). Figure 7 As shown, by constructing the recognition matrix, the sentence can be successfully reduced to the grammar start symbol S, which conforms to the syntactic rules.
[0130] ⑦ Call the trained Tri-Gram model and calculate P(Sen)=0.805 through the formula in step S21, which exceeds the threshold of 0.783 and has reasonable semantics.
[0131] Therefore, the final sentence generated by the keyword reading is She loves reading.
[0132] Example 2
[0133] This embodiment provides a text generation system integrating dual language models and sentence detection, including: a keyword acquisition module, a language model M1, a language model M2, a splicing module, a sentence generation condition judgment module, a sentence structure analysis module, a semantic analysis module, and a text output module;
[0134] In this embodiment, the keyword acquisition module is used to acquire the input keyword, and the language model M1 and the language model M2 synchronously expand the words before and after the keyword;
[0135] In this embodiment, the language model M1 predicts the next character forward to obtain the new text S1, and the language model M2 predicts the next character backward to obtain the new text S2;
[0136] In this embodiment, the concatenation module is used to concatenate the new text S1 and the new text S2 to obtain a sentence;
[0137] In this embodiment, the sentence generation condition judgment module is used to determine whether the sentence meets the output condition. If the output condition is met, the sentence is output, otherwise it is used as a new input;
[0138] In this embodiment, the sentence structure analysis module is used to perform structural analysis on the sentence based on the syntactic analysis method of the specification. If the sentence can be reduced to the grammatical start symbol, the sentence complies with the syntactic rules;
[0139] In this embodiment, the semantic analysis module is used to calculate the language score of the candidate sentences that meet the syntactic rules using the N-Gram statistical model;
[0140] In this embodiment, the text output module is used to output the final text. If the language score of a candidate sentence exceeds a set threshold, the candidate sentence is output as the final text.
[0141] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A text generation method integrating a dual language model and sentence detection, characterized in that: The steps include: Get the input keyword, and language model M1 and language model M2 synchronously expand the words before and after the keyword; The language model M1 predicts the next character forward to obtain a new text S1, the language model M2 predicts the next character backward to obtain a new text S2, and the new text S1 and the new text S2 are concatenated to obtain a sentence; The language model M1 and the language model M2 synchronously expand the words before and after the keyword, and the specific steps include: Input keywords to the trained language models M1 and M2. Language models M1 and M2 predict the next character respectively, and obtain the original probability distribution of the next character. Use the top_k method to select characters and their corresponding prediction probabilities, and update the prediction set; Generate a part-of-speech proximity table through the training set, obtain the part-of-speech proximity values of the corresponding part-of-speech combinations in the updated prediction set through the part-of-speech proximity table, and multiply the part-of-speech proximity values by the original probability to obtain a new probability distribution; Redistribute the probability distribution and randomly sample the next character nw and the next character fw; The input keyword kw is concatenated with the randomly sampled character nw and character fw to obtain the concatenated sequence kw+nw and the concatenated sequence fw+kw; Concatenate the prediction results of language model M1 and language model M2 to obtain text S = fw + kw + nw; The specific steps of generating a part-of-speech adjacent value table through a training set include: For part of speech pos, part of speech s k The probability that the word follows the part-of-speech pos word is: Among them, num k Indicates part of speech combination pos+s k The number of times it appears in the training set, n represents the number of types of parts of speech, and finally generates a part-of-speech adjacent value table Q of size n*n; Redistribute the probability distribution, the specific calculation formula is: Among them, π(w k ) represents the probability value after redistribution, temperature represents the scale factor, w k Indicates updating the candidate words in the prediction set, p(w k ) represents the probability corresponding to the candidate word; Determine whether the sentence meets the output conditions. If it meets the output conditions, output the sentence; otherwise, use it as a new input. The syntactic analysis method based on the specification conducts structural analysis on the sentence. If the sentence can be reduced to the grammatical start symbol, the sentence conforms to the syntactic rules. The N-Gram statistical model is used to calculate the language score of candidate sentences that meet the syntactic rules. If the language score of a candidate sentence exceeds the set threshold, the candidate sentence is output as the final text.
2. The text generation method integrating dual language model and sentence detection according to claim 1, characterized in that: The syntactic analysis method based on convention performs structural analysis on a sentence, and the specific steps include: Mark the parts of speech of all characters in the sentence. Each character has a corresponding non-terminal symbol. Reduce the sentence according to the grammar G. If the sentence can be reduced to the grammar start symbol S, the sentence conforms to the syntactic rules.
3. The text generation method integrating dual language model and sentence detection according to claim 1, characterized in that: The syntactic analysis method based on the convention performs structural analysis on the sentence, and adopts the CYK algorithm based on the CFG rule to construct a recognition matrix with a size of [n+1, n+1], where n is the total number of words in the input sentence Sen.
4. The text generation method integrating dual language model and sentence detection according to claim 3 is characterized in that: The recognition matrix construction step comprises: Definition i,j To identify the value of coordinate [i,j] in the matrix, construct the main diagonal t 0,0 =0,t 1,1 to n,n Put the words w of the input sentence in sequence i ; Construct the diagonal t above and to the right of the main diagonal 0,1 to n-1,n , where t 1,1 to k-1,k =G(w k ), where G(w k ) indicates that w is derived from grammar G k The non-terminal symbol of ; Construct the remaining diagonal above and to the right of the main diagonal, for t i,j , if there exists a production A→t in the grammar G i,j-1 t i+1,j , then t i,j =A, otherwise t i,j =t i,j-1 .
5. The text generation method integrating dual language model and sentence detection according to claim 1, characterized in that: The N-Gram statistical model is used to calculate the language score of the candidate sentences that meet the syntactic rules. Specifically, the Tri-Gram model is used to obtain the probability of the sentence appearing by calculating the product of the probability of each word appearing.
6. The text generation method integrating dual language model and sentence detection according to claim 1, characterized in that: The language model M1 and the language model M2 have the same structure and adopt any one of the neural network models LSTM, GRU or Transform.
7. A text generation system integrating a dual language model and sentence detection, characterized in that: include: Keyword acquisition module, language model M1, language model M2, splicing module, sentence generation condition judgment module, sentence structure analysis module, semantic analysis module, text output module; The keyword acquisition module is used to acquire input keywords, and the language model M1 and the language model M2 synchronously expand the words before and after the keywords; The language model M1 predicts the next character forward to obtain a new text S1, and the language model M2 predicts the next character backward to obtain a new text S2; The language model M1 and the language model M2 synchronously expand the words before and after the keyword, specifically including: Input keywords to the trained language models M1 and M2. Language models M1 and M2 predict the next character respectively, and obtain the original probability distribution of the next character. Use the top_k method to select characters and their corresponding prediction probabilities, and update the prediction set; Generate a part-of-speech proximity table through the training set, obtain the part-of-speech proximity values of the corresponding part-of-speech combinations in the updated prediction set through the part-of-speech proximity table, and multiply the part-of-speech proximity values by the original probability to obtain a new probability distribution; Redistribute the probability distribution and randomly sample the next character nw and the next character fw; The input keyword kw is concatenated with the randomly sampled character nw and character fw to obtain the concatenated sequence kw+nw and the concatenated sequence fw+kw; Concatenate the prediction results of language model M1 and language model M2 to obtain text S = fw + kw + nw; The generating of the part-of-speech adjacent value table through the training set specifically includes: For part of speech pos, part of speech s k The probability that the word follows the part-of-speech pos word is: Among them, num k Indicates part of speech combination pos+s k The number of times it appears in the training set, n represents the number of types of parts of speech, and finally generates a part-of-speech adjacent value table Q of size n*n; Redistribute the probability distribution, the specific calculation formula is: Among them, π(w k ) represents the probability value after redistribution, temperature represents the scale factor, w k Indicates updating the candidate words in the prediction set, p(w k ) represents the probability corresponding to the candidate word; The splicing module is used to splice the new text S1 and the new text S2 to obtain a sentence; The sentence generation condition judgment module is used to determine whether the sentence meets the output condition. If the output condition is met, the sentence is output, otherwise it is used as a new input; The sentence structure analysis module is used to perform structural analysis on the sentence based on a syntactic analysis method based on the specification. If the sentence can be reduced to the grammatical start symbol, the sentence complies with the syntactic rules; The semantic analysis module is used to calculate the language score of the candidate sentences that meet the syntactic rules using the N-Gram statistical model; The text output module is used to output the final text. If the language score of a candidate sentence exceeds a set threshold, the candidate sentence is output as the final text.
