Text generation model training method and device

Abstract

The invention relates to the technical field of neural networks, and discloses a text generation model training method and device. The method comprises the steps of obtaining a training statement set; preprocessing the training statement to obtain a target training statement; inputting the target training statement into an initial text generation model, and sequentially encoding and decoding the target training statement based on an encoder and a decoder to generate a prediction text; constructing a basic loss function based on the prediction text and the reference text; dividing the position of the prediction word in the prediction statement according to a preset pivot point to obtain a first target position and a second target position; determining a target loss function based on the first indicator function, the first coefficient, the second indicator function, the second coefficient and the probability distribution of the prediction word; and training an encoder and a decoder of the initial text generation model according to the target loss function until convergence to obtain a pre-trained text generation model.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a training method and device for a text generation model. Background Art

[0002] At present, deep learning models use an encoding-decoding framework to convert input text into a series of vector representations, and then decode these vectors to generate target text. During the training process, the model continuously optimizes its parameters and improves the generation quality by minimizing the difference between the generated text and the real text (such as cross entropy loss).

[0003] However, during the training process, the model mainly learns the statistical laws of language rather than the factual knowledge of the real world. This may cause the model to generate text that is grammatically correct but untrue or illogical, resulting in inaccurate generated text. Therefore, how to improve the accuracy of generated text has become a problem that needs to be solved. Summary of the invention

[0004] In view of this, the present invention provides a training method and device for a text generation model.

[0005] In a first aspect, the present invention provides a training method for a text generation model, the method comprising: obtaining a training sentence set; wherein the training sentence set comprises a plurality of training sentences; preprocessing the training sentences to obtain target training sentences; inputting the target training sentences into an initial text generation model, encoding and decoding the target training sentences in turn based on an encoder and a decoder of the initial text generation model to generate predicted text; wherein the predicted text comprises predicted sentences; constructing a basic loss function based on the predicted text of the target training sentences and the reference text of the training sentences; determining the position of the predicted words in the predicted sentences based on the predicted words in the predicted sentences corresponding to the basic loss function according to the position index of the predicted words in the predicted sentences. Position; divide the position of the predicted word in the predicted sentence according to a preset pivot point to obtain a first target position and a second target position; determine the target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of the predicted word; wherein the first coefficient is less than the second coefficient; wherein the first indicator function is used to indicate the loss corresponding to the first target position, and the second indicator function is used to indicate the loss corresponding to the second target position; train the encoder and decoder of the initial text generation model according to the target loss function until convergence to obtain a pre-trained text generation model.

[0006] The text generation model training method provided in this embodiment can obtain clearer and more standardized target training sentences by preprocessing training sentences, thereby improving the training efficiency and generation quality of the text generation model.

[0007] At the same time, a basic loss function is constructed based on the predicted text of the target training sentence and the reference text of the training sentence, and the position information of the predicted words in the predicted sentence and the preset pivot point are introduced for position division. Different coefficients are assigned according to the predicted words in different positions. The coefficient before the preset pivot point is smaller than the coefficient after the preset pivot point. That is, a smaller weight is given to the loss of the first target position, and a larger weight is given to the loss of the second target position, which ensures the accuracy of the second half of the sentence, thereby improving the factual consistency of the entire sentence and further improving the accuracy of the generated text.

[0008] In one possible implementation, the method for determining the probability distribution of predicted words includes: determining multiple candidate words and an initial core probability corresponding to each candidate word based on a target training sentence; adjusting the initial core probability corresponding to the candidate word based on the association between the candidate word and the training sentence, and determining the adjusted core probability corresponding to the candidate word; and determining the probability distribution of the predicted word based on the adjusted core probability.

[0009] The text generation model training method provided in this embodiment can more accurately reflect the actual situation of language use, thereby improving the accuracy of prediction, by determining multiple candidate words and their initial kernel probabilities and adjusting the initial kernel probabilities using the association between the candidate words and the training sentences.

[0010] At the same time, in this process, not only the frequency or importance of the word itself (i.e., the initial kernel probability) is considered, but also the contextual information (i.e., the association between the training sentence and the candidate word) is combined to improve the quality and accuracy of the generated text.

[0011] In a possible implementation, an encoder and a decoder based on an initial text generation model sequentially encode and decode a target training sentence, and generating a predicted text includes: encoding the target training sentence for the tth time to obtain an encoded training sentence after the tth encoding; decoding the encoded target training sentence for the tth time to determine multiple candidate words of the tth encoded training sentence and an initial kernel probability corresponding to each candidate word; adjusting the initial kernel probability corresponding to the tth candidate word according to the association between the tth candidate word and the encoded training sentence to determine the adjusted kernel probability corresponding to the tth candidate word; determining the tth target candidate word with the highest kernel probability among the adjusted kernel probabilities; adding the tth target candidate word to the tth generated target training sentence to obtain the t+1th target training sentence; wherein t is not less than 1, and the tth initial kernel probability is greater than the t+1th initial kernel probability; when the t+1th target training sentence is the last target training sentence, integrating the target training sentences to generate a predicted text.

[0012] The training method of the text generation model provided in this embodiment, in the process of iterating the initial text generation model to generate the predicted text, as the length of the target training sentence increases, the kernel probability corresponding to the predicted word will decrease, that is, in the beginning stage of the target training sentence, more randomness is allowed to increase the diversity of the text; and in the second half of the target training sentence, as the context is determined, the randomness is reduced to ensure the factual accuracy of the predicted text. Compared with the traditional fixed parameter kernel sampling method, dynamic kernel sampling can more flexibly adapt to the needs of different generation stages, effectively improving the generation quality and factual accuracy of the predicted text.

[0013] In one possible implementation, when there are multiple pre-trained text generation models, the type of training data corresponding to each pre-trained text generation model is different; wherein, the method also includes: obtaining a sentence data set; wherein the sentence data set includes: a sentence to be processed and a question corresponding to the sentence to be processed; inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a first target processing sentence; inputting the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a second target processing sentence; repeating the steps of inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the first target processing sentence to inputting the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the second target processing sentence, until the output of each pre-trained text generation model is consistent.

[0014] The training method of the text generation model provided in this embodiment, when there are multiple pre-trained text generation models, the type of training data corresponding to each pre-trained text generation model is different, that is, the types of training sentence sets used by different types of pre-trained text generation models are different, and the accuracy and consistency of the generated text are improved by allowing multiple pre-trained text generation models to debate and correct each other. Each pre-trained text generation model generates answers from different angles, and then verifies and corrects these answers through the debate process, ensuring that the final generated text is accurate and consistent in all aspects, and improving the fault tolerance of the system, ensuring that even if individual agents make mistakes, the final generated text is still accurate and reliable.

[0015] In a possible implementation, determining the target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of the predicted words includes:

[0016] Among them, L is the target loss function, T is the total length of the sentence, 1 t′≥t is the first indicator function corresponding to the first target position and 1 t′＜t is the second indicator function corresponding to the second target position, α is the first coefficient, β is the second coefficient, P(ω t′ |ω1ω2,...,ω t′-1 ) is the probability distribution of the predicted words in the sentence.

[0017] In one possible implementation, preprocessing the training sentence to obtain the target training sentence includes: determining supplementary information based on document information corresponding to the training sentence; wherein the supplementary information is used to indicate the type of the training sentence; and configuring the supplementary information at a target position of the training sentence to obtain the target training sentence.

[0018] The training method of the text generation model provided in this embodiment ensures that each training sentence can independently express complete factual information by configuring the supplementary information at the target position of the training sentence. This step solves the problem of information fragmentation and improves the independence and accuracy of the generated text.

[0019] In a second aspect, the present invention provides a training device for a text generation model, the device comprising: an acquisition module, used to acquire a training sentence set; wherein the training sentence set includes multiple training sentences; a preprocessing module, used to preprocess the training sentences to obtain target training sentences; an encoding and decoding module, used to input the target training sentences into an initial text generation model, and to sequentially encode and decode the target training sentences based on an encoder and a decoder of the initial text generation model to generate predicted text; wherein the predicted text includes predicted sentences; a construction module, used to construct a basic loss function based on the predicted text of the target training sentences and the reference text of the training sentences; a first determination module, used to determine the predicted words in the predicted sentences corresponding to the basic loss function according to the position index of the predicted words in the predicted sentences The position of the predicted word in the predicted sentence; a division module, used to divide the position of the predicted word in the predicted sentence according to a preset pivot point to obtain a first target position and a second target position; a second determination module, used to determine the target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of the predicted words; wherein the first coefficient is less than the second coefficient; wherein the first indicator function is used to indicate the loss corresponding to the first target position, and the second indicator function is used to indicate the loss corresponding to the second target position; a training module, used to train the encoder and decoder of the initial text generation model according to the target loss function until convergence to obtain a pre-trained text generation model.

[0020] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the text generation model training method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the text generation model training method of the first aspect or any corresponding embodiment thereof.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the text generation model training method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flowchart of a method for training a text generation model according to an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of a training method for a text generation model according to an embodiment of the present invention;

[0026] Figure 3 is a structural block diagram of a training device for a text generation model according to an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] According to an embodiment of the present invention, an embodiment of a training method for a text generation model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] In this embodiment, a text generation model training method is provided, which can be used in computer devices, such as computers, servers, etc. Figure 1 is a flow chart of a method for training a text generation model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0031] Step S101, obtaining a training sentence set; wherein the training sentence set includes a plurality of training sentences.

[0032] The training sentence set can represent a sentence set used to train the initial text generation model; wherein, the training sentence set includes multiple training sentences, and the training sentences can be obtained from a text file or from a database. No specific limitation is made here and can be implemented by technical personnel in this field.

[0033] As an example, training sentences may be downloaded from a public text dataset to form a training sentence set.

[0034] As an example, training sentences may be downloaded from a corpus in a specific domain to form a training sentence set.

[0035] As an example, training sentences may be created to form a training sentence set, etc.

[0036] Step S102, pre-processing the training sentence to obtain the target training sentence.

[0037] The target training sentence is used to represent the training sentence generated after preprocessing the training sentence. The target training sentence can be used as the input of the initial text generation model. Specifically, the training sentence can be cleaned, segmented, and stop words removed to obtain a target training sentence suitable for the input model.

[0038] As an example, use a natural language processing library (such as NLTK, SpaCy) to perform tokenization and remove stop words.

[0039] As an example, normalize the text, such as converting to lowercase, removing punctuation, etc.

[0040] As an example, the title of the document corresponding to each training sentence can be added to a specific position of the training sentence corresponding to the document, such as the beginning of the sentence, the end of the sentence, etc., which is not specifically limited here.

[0041] Step S103, inputting the target training sentence into the initial text generation model, encoding and decoding the target training sentence in sequence based on the encoder and decoder of the initial text generation model to generate predicted text; wherein the predicted text includes the predicted sentence.

[0042] The encoder is used to convert the input target training sentence into an internal representation or vector in the latent space. The decoder generates the predicted text based on the output of the encoder (i.e., the internal representation or vector). This process can be done step by step, generating one word or character at a time until a certain stopping condition is reached (such as generating a specific end symbol or reaching a preset length). The predicted text is generated step by step by the decoder, and it may be a complete sentence, paragraph, or longer text. The generation process depends on the design of the decoder and the stopping condition.

[0043] As an example, a similar sentence is generated based on a given sentence. The following steps can be used to achieve this goal: Preprocess the input sentence by segmenting it and removing stop words. Encode the preprocessed sentence into a vector representation using the Transformer architecture encoder. The decoder using the Transformer architecture gradually generates predicted text based on the output of the encoder. At each decoding step, beam search can be used to retain multiple candidate sequences and finally select the best one as the output. Fact core sampling can also be used to reduce the probability of predicted words as the length of the sentence increases, and then the predicted text is obtained.

[0044] Step S104, constructing a basic loss function based on the predicted text of the target training sentence and the reference text of the training sentence.

[0045] In text generation tasks, the base loss function is a key tool for measuring the difference between the predicted text generated by the model and the true reference text. It guides the model's training process and helps the model gradually optimize its generation ability. The reference text corresponds to the training sentence and is used to evaluate the quality of the predicted text. During the training process, the goal of the initial text generation model is to minimize the difference between the predicted text and the reference text. Specifically, for sequence generation tasks, the loss is usually the sum of the losses of each word in the sequence. At each time step, the initial text generation model predicts the probability distribution of a word and compares it with the true word to calculate the loss.

[0046] As an example, for each target training sentence, the model generates a predicted text. The predicted text is compared word by word with the reference text. At each time step, the cross entropy loss between the probability distribution of the words predicted by the model and the true words is calculated. The losses of all words in the sequence are summed to get the total loss for the target training sentence. During the training process, the parameters of the model are updated through the back-propagation algorithm to minimize the total loss.

[0047] Step S105 , based on the predicted words in the predicted sentence corresponding to the basic loss function, and according to the position index of the predicted words in the predicted sentence, determine the position of the predicted words in the predicted sentence.

[0048] When using the basic loss function to evaluate the difference between the predicted text generated by the model and the reference text, each word in the predicted text will have a corresponding loss value. The loss value does not directly provide information about the position of the predicted word in the predicted sentence. The position information is determined by the position index of the predicted word in the predicted sentence.

[0049] As an example, a prediction sentence is generated by the input of a given training sentence: "The cat saton the mat". In this prediction sentence: "The" is the first word generated, and its position index is 0 (or 1, depending on the starting value of the index). "Cat" is the second word generated, and its position index is 1 (or 2). "sat" is the third word generated, and its position index is 2 (or 3). And so on, until the entire prediction sentence is generated. Then, according to the position index of the predicted word in the prediction sentence, the position of the predicted word in the prediction sentence is determined.

[0050] Step S106, dividing the position of the predicted word in the predicted sentence according to the preset pivot point to obtain a first target position and a second target position.

[0051] Pivot points are used to divide the benchmarks for predicting sentences. The choice of pivot points depends on the specific requirements of the task and may be based on syntax, semantics, topic, or other relevant factors.

[0052] As an example, the pivot point may be a fixed value: t = 0.5 x length (sentence).

[0053] As an example, the pivot point can be a random value: t~uniform[0.3,0.7]×length(sentence).

[0054] As an example, the pivot point can be a complex value. The value of t is determined based on the root value of the dependency parsing result.

[0055] It should be noted that dependency parsing is a technique in natural language processing (NLP) that identifies the dependency relationship between words in a language. In this analysis, sentences are constructed into a dependency structure, usually a tree, where the nodes in the tree represent words and the edges represent the grammatical relationship between words. This structure helps to understand the syntactic structure of the language, such as the relationship between subject, predicate and object, and thus supports more complex language understanding tasks, such as information extraction and machine translation.

[0056] Specifically, after determining the preset pivot point, the position of the predicted word in the predicted sentence can be divided into a first target position and a second target position.

[0057] As an example, the position of the predicted word before the preset pivot point may be classified as the first target position, and the position of the predicted word after the preset pivot point may be classified as the second target position.

[0058] Step S107, determine the target loss function based on the first indicator function corresponding to the first target position, the first coefficient corresponding to the first target position, the second indicator function corresponding to the second target position, the second coefficient corresponding to the second target position, and the probability distribution of the predicted words; wherein the first coefficient is smaller than the second coefficient; wherein the first indicator function is used to indicate the loss corresponding to the first target position, and the second indicator function is used to indicate the loss corresponding to the second target position.

[0059] By combining the indicator functions and coefficients at different locations to construct the target loss function, differentiated evaluation and optimization of the model's generation capabilities in different areas can be achieved.

[0060] The first indicator function is used to indicate the loss corresponding to the first target position. Specifically, it can be a binary function that returns 1 when the word is located at the first target position, otherwise it returns 0. The second indicator function is used to indicate the loss corresponding to the second target position. Similarly, it is also a binary function that returns 1 when the word is located at the second target position, otherwise it returns 0. The first coefficient is a coefficient associated with the loss of the first target position, and is used to adjust the weight of this part of the loss in the total loss. According to the requirements of the question, α should be smaller than the second coefficient. The second coefficient is a coefficient associated with the loss of the second target position, and is used to adjust the weight of this part of the loss in the total loss. Since α<β, the loss of the second target position will account for a larger proportion of the total loss.

[0061] Specifically, for each word in the predicted sentence, the model generates a probability distribution indicating the probability of the word at each possible position, and for each word, the corresponding indicator function and coefficient are used to calculate the loss according to its position (first target position or second target position). The loss can be calculated based on cross entropy, negative log likelihood, or other appropriate loss functions.

[0062] In order to satisfy the condition of α<β, the definitions of α and β are as follows:

[0063] α=γ; wherein α is the first coefficient and γ is the first calculation coefficient. β=γ+|δ|; wherein β is the second coefficient and δ is the second calculation coefficient.

[0064] γ and δ can be updated according to the following formula, where η is the learning rate:

[0065] Wherein, η can be a fixed value.

[0066] From the above content, it can be seen that with the continuous training of the initial text generation model, the first coefficient will always be smaller than the second coefficient.

[0067] As an example, based on a first indicator function corresponding to a first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to a second target position, a second coefficient corresponding to the second target position, and a probability distribution of predicted words, determining a target loss function includes:

[0068] Among them, L is the target loss function, T is the total length of the sentence, 1 t′≥t is the first indicator function corresponding to the first target position and 1 t′＜t is the second indicator function corresponding to the second target position, α is the first coefficient, β is the second coefficient, P(ω t′ |ω1ω2,...,ω t′-1 ) is the probability distribution of the predicted words in the sentence, and t is the number of probabilities.

[0069] Step S108, training the encoder and decoder of the initial text generation model according to the target loss function until convergence, to obtain a pre-trained text generation model.

[0070] The model is iteratively trained using the training data. In each iteration, the model generates predicted text based on the input text. The target loss between the predicted text and the reference text is calculated. The loss value is propagated back to the encoder and decoder of the model through the back-propagation algorithm. According to the gradient of the loss value, the parameters of the model are updated until convergence, and a pre-trained text generation model is obtained.

[0071] The text generation model training method provided in this embodiment can obtain clearer and more standardized target training sentences by preprocessing training sentences, thereby improving the training efficiency and generation quality of the text generation model.

[0072] At the same time, a basic loss function is constructed based on the predicted text of the target training sentence and the reference text of the training sentence, and the position information of the predicted words in the predicted sentence and the preset pivot point are introduced for position division. Different coefficients are assigned according to the predicted words in different positions. The coefficient before the preset pivot point is smaller than the coefficient after the preset pivot point. That is, a smaller weight is given to the loss of the first target position, and a larger weight is given to the loss of the second target position, which ensures the accuracy of the second half of the sentence, thereby improving the factual consistency of the entire sentence and further improving the accuracy of the generated text.

[0073] In a possible implementation, the method further includes:

[0074] Step a1, determining multiple candidate words and the initial kernel probability corresponding to each candidate word according to the target training sentence.

[0075] The candidate words may be words selected from a vocabulary, context or other sources, and may specifically be words that appear in a target training sentence or a predicted sentence. Each candidate word has an initial probability associated with it, and the initial kernel probability may characterize the probability before adjustment, and may be determined based on the frequency of occurrence of the word in the training data, context relevance or other statistical measures. Specifically, context information is extracted from the target training sentence. Based on the context information and the vocabulary, a plurality of candidate words are determined. An initial kernel probability is assigned to each candidate word.

[0076] Step a2, adjusting the initial kernel probabilities corresponding to the candidate words according to the association between the candidate words and the training sentences, and determining the adjusted kernel probabilities corresponding to the candidate words.

[0077] The association relationship can be based on grammar (such as the position of the word in the sentence, part of speech collocation, etc.), or it can be based on semantics (such as the similarity between words, contextual meaning, etc.). Specifically, a semantic similarity metric (such as cosine similarity, Jaccard similarity) can be used to evaluate the semantic relevance between the candidate words and the target training sentence. Statistical methods (such as chi-square test, mutual information) can also be used to evaluate the association relationship between the candidate words and the target training sentence, etc., which is not specifically limited here and can be implemented by those skilled in the art.

[0078] As an example, the initial kernel probability of the candidate words is adjusted according to the quantified association strength. Generally, the probability of a word with a higher association strength should be increased; conversely, the probability of a word with a lower association strength should be reduced.

[0079] As an example, there is a training sentence "I like to eat apples", the candidate word set is {"banana", "apple", "orange", "pear"}, and the initial kernel probabilities are {0.25, 0.25, 0.25, 0.25} respectively.

[0080] If a statistical-based approach is used and it is found that "apple" has the highest association strength with the training sentence (for example, through word frequency co-occurrence analysis), the probability of "apple" can be increased to 0.5, while the probabilities of other words are reduced accordingly. If a machine learning-based approach is used and a classifier is trained to predict association strength, the output of the classifier may indicate that "apple" has the highest association strength, thereby guiding the adjustment of the probability.

[0081] Step a3, determining the probability distribution of the predicted word based on the adjusted kernel probability.

[0082] Collect the adjusted kernel probabilities of all candidate words and use the adjusted probability values ​​to construct a probability distribution. This distribution is a discrete probability distribution, where each candidate word corresponds to a probability value. Each probability value in the probability distribution represents the relative likelihood of the corresponding candidate word appearing in a given context. The higher the probability value, the greater the likelihood of the word appearing.

[0083] As an example, for the candidate words {“banana”, “apple”, “orange”, “pear”}, the adjusted kernel probabilities are {0.1, 0.6, 0.2, 0.1} respectively.

[0084] First, check whether the sum of these probabilities is 1. In this example, the sum is 0.1+0.6+0.2+0.1=1, so no normalization is needed.

[0085] Then, construct the probability distribution: P("banana") = 0.1, P("apple") = 0.6, P("orange") = 0.2, P("pear") = 0.1.

[0086] Finally, in a given context, “apple” has the highest probability of appearing (60%), followed by “orange” (20%), while “banana” and “pear” are less likely to appear (10% each).

[0087] The text generation model training method provided in this embodiment can more accurately reflect the actual situation of language use, thereby improving the accuracy of prediction, by determining multiple candidate words and their initial kernel probabilities and adjusting the initial kernel probabilities using the association between the candidate words and the training sentences.

[0088] At the same time, in this process, not only the frequency or importance of the word itself (i.e., the initial kernel probability) is considered, but also the contextual information (i.e., the association between the training sentence and the candidate word) is combined to improve the quality and accuracy of the generated text.

[0089] In a possible implementation, the encoder and decoder based on the initial text generation model in step S103 sequentially encode and decode the target training sentence to generate the predicted text, including:

[0090] Step b1, encode the target training sentence for the tth time to obtain the encoded training sentence after the tth encoding.

[0091] During the encoding and decoding process, when the predicted words of the current target training sentence are passed, the predicted words are added to the target training sentence as the next target training sentence. In this process, it is necessary to repeat the execution continuously.

[0092] Specifically, the encoding method can be rule-based (such as bag-of-words model, TF-IDF), statistical-based (such as word embedding, topic model) or neural network-based (such as RNN, LSTM, Transformer, etc.), which is not specifically limited here. Determine the target training sentence to be encoded, and use the selected encoding method to encode the target training sentence. After the encoding is completed, an encoded training sentence will be obtained, that is, the encoded training sentence after the tth encoding.

[0093] Step b2, decoding the encoded target training sentence for the tth time, determining multiple candidate words of the tth encoded training sentence and the initial kernel probability corresponding to each candidate word.

[0094] Choose a suitable decoder, which can be rule-based, statistical, or neural network-based. In a neural network model, the decoder is usually a neural network that matches (or mirrors) the encoder structure. For example, if the encoder is an RNN, the decoder may also be an RNN or its variant. Input the encoded target training sentence into the decoder, and the decoder will generate a series of possible candidate words step by step or all at once based on the input encoding information. For each generated candidate word, the decoder will also assign it an initial kernel probability. This probability reflects the possibility of the word as a decoding output given the encoding information.

[0095] Step b3, according to the association between the t-th candidate word and the encoding training sentence, the initial kernel probability corresponding to the t-th candidate word is adjusted to determine the adjusted kernel probability corresponding to the t-th candidate word.

[0096] The association relationship can be based on grammar (such as the position of the word in the sentence, part of speech collocation, etc.), or it can be based on semantics (such as the similarity between words, contextual meaning, etc.). Specifically, a semantic similarity metric (such as cosine similarity, Jaccard similarity) can be used to evaluate the semantic relevance between the candidate words and the target training sentence. Statistical methods (such as chi-square test, mutual information) can also be used to evaluate the association relationship between the t-th candidate words and the encoded training sentence, etc., which is not specifically limited here and can be implemented by those skilled in the art.

[0097] Based on the evaluation results of the above association relationships, the initial core probabilities of the candidate words will be appropriately adjusted. The adjustment strategy may involve weighting or renormalization of the probabilities, or directly increasing or decreasing the probability values ​​according to the association strength, etc., which are not specifically limited here.

[0098] As an example, the encoding training sentence "I like to eat apples" and the decoder generates two candidate words "banana" and "apple" with initial kernel probabilities of 0.4 and 0.6 respectively. By evaluating the association between the candidate words and the encoding training sentence, it is found that "apple" completely matches the keyword "apple" in the encoding training sentence, while "banana" has no direct association. Therefore, the probability value of "apple" can be appropriately increased (such as adjusted to 0.8) while the probability value of "banana" can be reduced (such as adjusted to 0.2) to reflect this difference in association.

[0099] Step b4, determining the t-th target candidate word with the highest kernel probability among the adjusted kernel probabilities.

[0100] According to the correlation between the candidate words and the encoding training sentences, the initial kernel probability is adjusted, and the adjusted kernel probability corresponding to each candidate word is obtained. All the t-th candidate words and their corresponding adjusted kernel probabilities are traversed, and these kernel probabilities are compared to find the one with the highest value.

[0101] As an example, there are two candidate words "apple" and "banana", whose adjusted kernel probabilities are 0.8 and 0.2 respectively. By comparison, we can determine that "apple" is the candidate word with the highest kernel probability, so it is used as the target candidate word for the tth time.

[0102] Step b5, adding the t-th target candidate word to the t-th generated target training sentence to obtain the t+1-th target training sentence; wherein t is not less than 1, and the t-th initial kernel probability is greater than the t+1-th initial kernel probability.

[0103] The identified t-th target candidate word is seamlessly connected to the end of the current target training sentence. This is to gradually build a complete output sequence, which will continue to extend as the decoding progresses. After the above update, a new target training sentence is obtained, that is, the t+1-th target training sentence. This sentence not only contains the output of all previous steps, but also adds the t-th target candidate word.

[0104] It should be noted that t is not less than 1, and the initial kernel probability of the tth time is greater than the initial kernel probability of the t+1th time. That is, in the process of sentence generation, as the content is gradually determined, the value of the kernel probability is reduced, thereby reducing randomness and improving the factuality of the second half of the sentence. At the same time, in order to avoid the kernel probability from decaying too quickly, the diversity of text generation is ensured.

[0105] As the decoding steps progress, the initial kernel probability of the subsequent steps (such as the t+1th step) often changes. This change may be due to various factors, such as the accumulation of context information, the relevance of the candidate word to the generated sequence, etc. As the decoding progresses, the initial kernel probability of the model will decrease, making the prediction of the subsequent output more accurate.

[0106] As an example, dynamically adjust the kernel probability p of generating each word t , the specific formula can be as follows:

[0107] p t =max{ω,p×λ t-1}; where p t is the initial kernel probability, λ is the attenuation factor, ω is the lower limit of probability attenuation, and p is the kernel probability assigned by the model. In the process of sentence generation, as the content is gradually determined, the initial kernel probability p is reduced. t The value of , thereby reducing randomness and improving the factuality of the second half of the statement. At the same time, in order to avoid p t The value decays too quickly, so ω is introduced as p t The lower limit of the value ensures the diversity of text generation.

[0108] Step b6: when the target training sentence of the t+1th time is the last target training sentence, each target training sentence is integrated to generate a predicted text.

[0109] Starting from the first decoding, backtrack and collect the target training sentences generated after each decoding. These sentences together form the preliminary framework of the predicted text. All the collected target training sentences are spliced ​​in order according to the decoding order to obtain the predicted text.

[0110] The training method of the text generation model provided in this embodiment, in the process of iterating the initial text generation model to generate the predicted text, as the length of the target training sentence increases, the kernel probability corresponding to the predicted word will decrease, that is, in the beginning stage of the target training sentence, more randomness is allowed to increase the diversity of the text; and in the second half of the target training sentence, as the context is determined, the randomness is reduced to ensure the factual accuracy of the predicted text. Compared with the traditional fixed parameter kernel sampling method, dynamic kernel sampling can more flexibly adapt to the needs of different generation stages, effectively improving the generation quality and factual accuracy of the predicted text.

[0111] In a possible implementation, when there are multiple pre-trained text generation models, each pre-trained text generation model corresponds to a different type of training data; wherein the method further includes:

[0112] Step c1, obtaining a sentence data set; wherein the sentence data set includes: sentences to be processed and questions corresponding to the sentences to be processed.

[0113] When there are multiple pre-trained text generation models, the types of training data corresponding to each pre-trained text generation model are different. The pre-trained text generation models may be model A, model B, and model C. The training data of model A may be data in a historical database, the training data of model B may be data of the current text, and the training data of model C may be other types of data. Each model uses different training data, so when the same content is input into different models, each model may give different results.

[0114] A sentence dataset may be a collection consisting of a series of sentences and their related information (such as corresponding questions, tags, context, etc.).

[0115] Step c2, inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a first target processing sentence.

[0116] After each type of pre-trained text generation model is trained, the sentence to be processed and the question corresponding to the sentence to be processed can be used as input. Each pre-trained text generation model will give an answer based on the sentence to be processed and the question corresponding to the sentence to be processed, that is, the first target processing sentence.

[0117] Step c3, inputting the questions corresponding to the first target processing sentence and the sentence to be processed into each pre-trained text generation model to obtain the second target processing sentence.

[0118] The first target processing sentences generated by each model are then integrated and input into each pre-trained text generation model again in combination with the question corresponding to the sentence to be processed to obtain the second answer, which is the second target processing sentence.

[0119] Step c4, repeatedly execute the steps of inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the first target processing sentence, and inputting the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the second target processing sentence, until the output of each pre-trained text generation model is consistent.

[0120] Then, the above steps c2 to c3 are repeated until the output of each pre-trained text generation model is consistent.

[0121] Please refer to Figure 2 , Figure 2It is a schematic diagram of a training method for a text generation model provided according to an embodiment of the present invention.

[0122] Figure 2 Each agent in the figure represents an agent, that is, a pre-trained text generation model. The figure includes agent 1, agent 2...agent n pre-trained text generation models. The question is input into each pre-trained text generation model, and each pre-trained text generation model generates the corresponding first answer, such as agent 1 generates answer 1-1, agent 2 generates answer 1-2, and agent n generates answer 1-n. Then, the first answer generated by each pre-trained text generation model is input as a whole into agent 1, agent 2...agent n pre-trained text generation models to generate the second answer.

[0123] The training method of the text generation model provided in this embodiment, when there are multiple pre-trained text generation models, the type of training data corresponding to each pre-trained text generation model is different, that is, the types of training sentence sets used by different types of pre-trained text generation models are different, and the accuracy and consistency of the generated text are improved by allowing multiple pre-trained text generation models to debate and correct each other. Each pre-trained text generation model generates answers from different angles, and then verifies and corrects these answers through the debate process, ensuring that the final generated text is accurate and consistent in all aspects, and improving the fault tolerance of the system, ensuring that even if individual agents make mistakes, the final generated text is still accurate and reliable.

[0124] In a possible implementation, the above step S102 includes:

[0125] Step d1, determining supplementary information according to the document information corresponding to the training sentence; wherein the supplementary information is used to indicate the type of the training sentence.

[0126] The document information is used to indicate the name of the document, the type of the document, etc. The supplementary information can be used to indicate the supplementary information determined according to the document information. The supplementary information is used to indicate the type of the training sentence, and the supplementary information can be the title of the document, the identifier of the document, etc., which is not specifically limited here. Specifically, the supplementary information is added to the training sentence to identify the training sentence.

[0127] As an example, using the Wikipedia dataset as a factual corpus, for each sentence in a document, the title of the document to which it belongs is added as supplementary information before the sentence to form a new sentence unit.

[0128] Step d2, configuring the supplementary information at the target position of the training sentence to obtain the target training sentence.

[0129] The target position may be the beginning or the end of the training sentence, and is not specifically limited here. Specifically, the supplementary information is inserted into the target position of the training sentence to obtain the target training sentence.

[0130] In one possible implementation, it is checked whether the training sentence after the supplementary information is inserted still maintains its original intention and contextual coherence. If necessary, the target training sentence is fine-tuned to ensure its accuracy and effectiveness.

[0131] The training method of the text generation model provided in this embodiment ensures that each training sentence can independently express complete factual information by configuring the supplementary information at the target position of the training sentence. This step solves the problem of information fragmentation and improves the independence and accuracy of the generated text.

[0132] In the present embodiment, a training device for a text generation model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" may implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0133] This embodiment provides a training device for a text generation model, such as Figure 3As shown, it includes: an acquisition module 301, which is used to acquire a training sentence set; wherein the training sentence set includes multiple training sentences; a preprocessing module 302, which is used to preprocess the training sentences to obtain target training sentences; an encoding and decoding module 303, which is used to input the target training sentences into the initial text generation model, and the encoder and decoder based on the initial text generation model sequentially encode and decode the target training sentences to generate predicted text; wherein the predicted text includes the predicted sentence; a construction module 304, which is used to construct a basic loss function based on the predicted text of the target training sentence and the reference text of the training sentence; a first determination module 305, which is used to determine the position of the predicted word in the predicted sentence based on the predicted word in the predicted sentence corresponding to the basic loss function according to the position index of the predicted word in the predicted sentence ; A division module 306 is used to divide the positions of the predicted words in the predicted sentence according to a preset pivot point to obtain a first target position and a second target position; a second determination module 307 is used to determine the target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of the predicted words; wherein the first coefficient is less than the second coefficient; wherein the first indicator function is used to indicate the loss corresponding to the first target position, and the second indicator function is used to indicate the loss corresponding to the second target position; a training module 308 is used to train the encoder and decoder of the initial text generation model according to the target loss function until convergence to obtain a pre-trained text generation model.

[0134] In a possible implementation, the above-mentioned device also includes: a third determination module, which is used to determine multiple candidate words and the initial core probability corresponding to each candidate word according to the target training sentence; a fourth determination module, which is used to adjust the initial core probability corresponding to the candidate word according to the association relationship between the candidate word and the training sentence, and determine the adjusted core probability corresponding to the candidate word; a fifth determination module, which is used to determine the probability distribution of the predicted word according to the adjusted core probability.

[0135] In a possible implementation, the encoding and decoding module 303 includes: an encoding unit, which is used to encode the target training sentence for the tth time to obtain the encoded training sentence after the tth encoding; a decoding unit, which is used to decode the encoded target training sentence for the tth time to determine multiple candidate words of the tth encoded training sentence and the initial core probability corresponding to each candidate word; a first determination unit, which is used to adjust the initial core probability corresponding to the tth candidate word according to the association between the tth candidate word and the encoded training sentence, and determine the adjusted core probability corresponding to the tth candidate word; a second determination unit, which is used to determine the tth target candidate word with the highest core probability among the adjusted core probabilities; a third determination unit, which is used to add the tth target candidate word to the tth generated target training sentence to obtain the t+1th target training sentence; wherein t is not less than 1, and the tth initial core probability is greater than the t+1th initial core probability; an integration unit, which is used to integrate the target training sentences when the t+1th target training sentence is the last target training sentence, to generate a predicted text.

[0136] In a possible implementation, when there are multiple pre-trained text generation models, the type of training data corresponding to each pre-trained text generation model is different; wherein the above-mentioned device also includes: a sentence data set acquisition module, used to acquire a sentence data set; wherein the sentence data set includes: a sentence to be processed and a question corresponding to the sentence to be processed; a first processing module, used to input the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a first target processing sentence; a second processing module, used to input the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a second target processing sentence; a repeated execution module, used to repeatedly execute the steps of inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the first target processing sentence to inputting the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain the second target processing sentence, until the output of each pre-trained text generation model is consistent.

[0137] In a possible implementation, the second determination module 307 is used to determine the target loss function according to the following formula:

[0138] Among them, L is the target loss function, T is the total length of the sentence, 1 t′≥t is the first indicator function corresponding to the first target position and 1 t′＜t is the second indicator function corresponding to the second target position, α is the first coefficient, β is the second coefficient, P(ω t′ |ω1ω2,...,ω t′-1) is the probability distribution of the predicted words in the sentence.

[0139] In one possible implementation, the preprocessing module 302 includes: a fourth determination unit, used to determine supplementary information based on document information corresponding to the training sentence; wherein the supplementary information is used to indicate the type of the training sentence; and a configuration unit, used to configure the supplementary information at a target position of the training sentence to obtain a target training sentence.

[0140] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0141] The training device of the text generation model in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0142] The embodiment of the present invention also provides a computer device having the above Figure 3 The training setup for the text generation model shown.

[0143] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0144] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0145] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0146] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0148] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0149] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0150] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0151] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A training method for a text generation model, characterized in that: The method comprises: Obtaining a training sentence set; wherein the training sentence set includes a plurality of training sentences; Preprocessing the training sentences to obtain target training sentences; Inputting a target training sentence into an initial text generation model, encoding and decoding the target training sentence in sequence based on an encoder and a decoder of the initial text generation model to generate a predicted text; wherein the predicted text includes a predicted sentence; Construct a basic loss function based on the predicted text of the target training sentence and the reference text of the training sentence; Based on the predicted words in the predicted sentence corresponding to the basic loss function, the position of the predicted words in the predicted sentence is determined according to the position index of the predicted words in the predicted sentence; Dividing the position of the predicted word in the predicted sentence according to a preset pivot point to obtain a first target position and a second target position; Determine a target loss function based on a first indicator function corresponding to a first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to a second target position, a second coefficient corresponding to the second target position, and a probability distribution of predicted words; wherein the first coefficient is less than the second coefficient; wherein the first indicator function is used to indicate a loss corresponding to the first target position, and the second indicator function is used to indicate a loss corresponding to the second target position; The encoder and decoder of the initial text generation model are trained according to the target loss function until convergence to obtain a pre-trained text generation model.

2. The text generation model training method according to claim 1, characterized in that: Methods for determining the probability distribution of predicted words include: According to the target training sentence, multiple candidate words and the initial kernel probability corresponding to each candidate word are determined; According to the association between the candidate words and the training sentences, the initial kernel probabilities corresponding to the candidate words are adjusted to determine the adjusted kernel probabilities corresponding to the candidate words; Based on the adjusted kernel probability, the probability distribution of the predicted word is determined.

3. The text generation model training method according to claim 1, characterized in that: The encoder and decoder based on the initial text generation model sequentially encode and decode the target training sentence to generate the predicted text, including: Encode the target training sentence for the tth time to obtain the encoded training sentence after the tth encoding; Decoding the encoded target training sentence for a tth time to determine a plurality of candidate words of the tth encoded training sentence and an initial kernel probability corresponding to each candidate word; According to the association relationship between the candidate words and the encoding training sentences of the tth time, the initial kernel probability corresponding to the candidate words of the tth time is adjusted to determine the adjusted kernel probability corresponding to the candidate words of the tth time; Determine the t-th target candidate word with the highest kernel probability among the adjusted kernel probabilities; The target candidate word of the tth time is added to the target training sentence generated for the tth time to obtain the target training sentence of the t+1th time; wherein t is not less than 1, and the initial kernel probability of the tth time is greater than the initial kernel probability of the t+1th time; When the target training sentence of the t+1th time is the last target training sentence, each target training sentence is integrated to generate a predicted text.

4. The text generation model training method according to claim 2, characterized in that: When there are multiple pre-trained text generation models, each pre-trained text generation model corresponds to a different type of training data; wherein the method further includes: Acquire a sentence data set; wherein the sentence data set includes: sentences to be processed and questions corresponding to the sentences to be processed; Inputting the to-be-processed sentence and the question corresponding to the to-be-processed sentence into each pre-trained text generation model to obtain a first target processing sentence; Inputting the questions corresponding to the first target processing sentence and the sentence to be processed into each pre-trained text generation model to obtain a second target processing sentence; Repeat the steps of inputting the sentence to be processed and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a first target processing sentence, and inputting the first target processing sentence and the question corresponding to the sentence to be processed into each pre-trained text generation model to obtain a second target processing sentence, until the output of each pre-trained text generation model is consistent.

5. The text generation model training method according to claim 3, characterized in that: Determining the target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of the predicted words includes: In this example, L is the target loss function, T is the total length of the sentence, 1 t′≥t is the first indicator function corresponding to the first target position and 1 t′＜t is the second indicator function corresponding to the second target position, α is the first coefficient, β is the second coefficient, P(ω t′ |ω1ω2,...,ω t′-1 ) is the probability distribution of the predicted words in the sentence, and t is the number of probabilities.

6. The text generation model training method according to any one of claims 1 to 5, characterized in that: The preprocessing of the training sentence to obtain the target training sentence comprises: Determining supplementary information according to document information corresponding to the training sentence; wherein the supplementary information is used to indicate the type of the training sentence; The supplementary information is arranged at a target position of the training sentence to obtain a target training sentence.

7. A training device for a text generation model, characterized in that: The device comprises: An acquisition module is used to acquire a training sentence set; wherein the training sentence set includes a plurality of training sentences; A preprocessing module, used for preprocessing the training sentence to obtain a target training sentence; The encoding and decoding module is used to input the target training sentence into the initial text generation model, and the encoder and decoder based on the initial text generation model sequentially encode and decode the target training sentence to generate a predicted text; wherein the predicted text includes the predicted sentence; A construction module, for constructing a basic loss function based on the predicted text of the target training sentence and the reference text of the training sentence; A first determination module is used to determine the position of the predicted word in the predicted sentence according to the position index of the predicted word in the predicted sentence based on the predicted word in the predicted sentence corresponding to the basic loss function; A division module, used for dividing the position of the predicted word in the predicted sentence according to a preset pivot point to obtain a first target position and a second target position; A second determination module is used to determine a target loss function based on a first indicator function corresponding to the first target position, a first coefficient corresponding to the first target position, a second indicator function corresponding to the second target position, a second coefficient corresponding to the second target position, and a probability distribution of predicted words; wherein the first coefficient is less than the second coefficient; wherein the first indicator function is used to indicate a loss corresponding to the first target position, and the second indicator function is used to indicate a loss corresponding to the second target position; The training module is used to train the encoder and decoder of the initial text generation model according to the target loss function until convergence to obtain a pre-trained text generation model.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the text generation method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the text generation method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the text generation method according to any one of claims 1 to 6.

Images