A text generation method and apparatus
By combining a predefined vocabulary, input text sequences, and an external corpus, multiple neural network models are used to generate output text, which solves the problem of insufficient ability of large-scale natural language generation models to learn new knowledge, and achieves the enrichment of knowledge sources and the preservation of general knowledge.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGKE DINGFU BEIJING TECH DEV
- Filing Date
- 2023-11-29
- Publication Date
- 2026-07-31
AI Technical Summary
Large-scale natural language generation models are limited in their ability to learn new knowledge when using a single external corpus.
By combining a predefined vocabulary, input text sequences, and an external corpus, multiple neural network models (a large model, a first neural network, and a second neural network) are used to generate output text. The probability distributions of each model are calculated, and the first and second neural networks are trained using an objective function to enrich the knowledge source.
It enhances the ability of large models to learn new knowledge while retaining existing general knowledge, thus preventing large models from forgetting general knowledge and experiencing a decline in recognition ability.
Smart Images
Figure CN117633183B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a text generation method and apparatus. Background Technology
[0002] Natural language generation models are used to generate matching output text (e.g., answers) based on user input text (e.g., questions). Natural language generation models with a large parameter scale are called large models, which are capable of performing complex processing and task handling on large-scale texts.
[0003] In training large models using training sets, these models typically segment the text in the training set using a predefined vocabulary to generate individual words, which are then encoded. However, due to the limited size of the predefined vocabulary and the training set, a retrieval-enhanced generation algorithm is currently employed to enable the large model to learn new knowledge from an external corpus.
[0004] Currently, large-scale models only learn new knowledge from external corpora, which is a single source, and there is still room for improvement in the ability of large-scale models to learn new knowledge. Summary of the Invention
[0005] This application provides a text generation method and apparatus that can enrich the variety of knowledge sources used to predict output text, thereby enhancing the ability of the new large model, which consists of a large model, a first neural network model, and a second neural network model, to learn new knowledge.
[0006] Firstly, a text generation method is provided, including:
[0007] Input at least one input text sequence into the large model and generate a first probability distribution of each corresponding output word segmentation sequence in the first vocabulary, where the first vocabulary is a predefined vocabulary used to train the large model.
[0008] At least one input text sequence is input into a first neural network, and each corresponding output word segmentation sequence is generated by a second probability distribution in a second vocabulary, which is determined based on the text input into the first neural network. The first neural network adopts an architecture that only includes a decoder.
[0009] Use a retrieval tool to identify the K text segments with the highest similarity to each input text sequence from the external corpus, where K is an integer greater than or equal to 0;
[0010] The second neural network uses a sequence of K text segments corresponding to each input text sequence and each input text sequence to generate a third probability distribution of each corresponding output word segment sequence in the third vocabulary. The third vocabulary is determined based on an external corpus. The second neural network adopts an architecture that only includes the decoder.
[0011] Based on the first probability distribution, second probability distribution, and third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary, calculate at least one predicted sequence of the output text.
[0012] Determine at least one output text based on the predicted sequence of at least one output text and the corresponding vocabulary.
[0013] In a feasible design, it also includes:
[0014] The objective function is determined based on at least one true sequence and one predicted sequence of output text;
[0015] Train the first neural network and the second neural network using the objective function, and obtain the trained first neural network model and the second neural network model respectively;
[0016] Using a large model, a trained first neural network model, and a second neural network model, calculate the predicted sequence of the first output text based on the first input text sequence;
[0017] The first output text is determined based on the predicted sequence of the first output text and the corresponding vocabulary.
[0018] In one feasible design, based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary, at least one predicted sequence of output text is calculated, including:
[0019] Based on the first probability distribution, second probability distribution and third probability distribution corresponding to each target output word segmentation sequence, a prediction label is generated for each target output word segmentation sequence. The prediction label is used to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution or the third probability distribution.
[0020] Based on the predicted label corresponding to each target output word segmentation sequence, determine the target probability distribution of each target output word segmentation sequence;
[0021] Calculate the target output word segmentation sequence for each target based on the target probability distribution and the corresponding vocabulary.
[0022] The target output word segmentation sequences are combined to form the corresponding predicted output text sequence.
[0023] In a feasible design, the objective function includes a weighted cross-entropy loss function, which is determined based on the predicted label and the true label corresponding to each target output word segmentation sequence. The true label is used to identify whether the target probability distribution of each true target output word segmentation sequence is a first probability distribution, a second probability distribution, or a third probability distribution.
[0024] In one feasible design, a second neural network is used to generate a third probability distribution of each corresponding output word segmentation sequence in a third vocabulary based on a sequence of K text segments corresponding to each input text sequence and each input text sequence, including:
[0025] From K text segments, determine k text segments whose similarity to the corresponding first input text sequence is greater than or equal to a similarity threshold, where 0 ≤ k ≤ K, and k is an integer. The first input text sequence belongs to at least one input text sequence.
[0026] If k = 0, the third probability distribution of the first input text sequence is determined to be 0;
[0027] If k > 0, based on the sequence of k text segments and the first input text sequence, generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0028] In one feasible design, based on the sequence of k text segments and the first input text sequence, a third probability distribution of the corresponding output word segmentation sequence in the third vocabulary is generated, including:
[0029] The first input text sequence and the sequences of each text segment are concatenated to generate k concatenated text sequences.
[0030] Based on k concatenated text sequences, generate the corresponding output word segmentation sequence in the third vocabulary with a third probability distribution.
[0031] In one feasible design, based on k concatenated text sequences, a third probability distribution of the corresponding output word segmentation sequence in the third vocabulary is generated, including:
[0032] If the text fragments are not manually labeled, the state vector output by the last hidden layer is generated by calculating each concatenated text sequence based on unsupervised learning.
[0033] If each text segment is manually labeled, a state vector is generated for each concatenated text sequence based on supervised learning, and the output of the last hidden layer is calculated.
[0034] Each state vector is decoded and linearly transformed to generate an intermediate vector corresponding to each concatenated text sequence;
[0035] Normalize each intermediate vector to generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0036] In one feasible design, the objective function is determined based on at least one true sequence and one predicted sequence of output text, including:
[0037] The first loss function corresponding to the second neural network is generated by maximizing the log-likelihood estimate of the true sequence and the predicted sequence of at least one output text.
[0038] Obtain the second loss function corresponding to the first neural network;
[0039] The objective function is determined by the sum of the first loss function, the weighted second loss function, and the weighted cross-entropy loss function.
[0040] In one feasible design, the first neural network is trained based on unsupervised learning.
[0041] Secondly, a text generation apparatus is provided, comprising:
[0042] The first probability distribution acquisition module is used to input at least one input text sequence into the large model and generate a first probability distribution of each corresponding output word segmentation sequence in the first vocabulary. The first vocabulary is a predefined vocabulary used to train the large model.
[0043] The second probability distribution acquisition module is used to input at least one input text sequence into the first neural network and generate a second probability distribution of each corresponding output word segmentation sequence in the second vocabulary. The second vocabulary is determined based on the text input into the first neural network, and the first neural network adopts an architecture that only includes a decoder.
[0044] The text fragment acquisition module is used to use a retrieval tool to determine the K text fragments with the highest similarity to each input text sequence from the external corpus, where K is an integer greater than or equal to 0;
[0045] The third probability distribution acquisition module is used to generate the third probability distribution of each corresponding output word segment sequence in the third vocabulary based on the sequence of K text segments corresponding to each input text sequence and each input text sequence using the second neural network. The third vocabulary is determined based on the external corpus. The second neural network adopts an architecture that only includes the decoder.
[0046] The prediction sequence calculation module is used to calculate the prediction sequence of at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary.
[0047] A text generation module is used to determine at least one output text based on a predicted sequence of at least one output text and a corresponding vocabulary.
[0048] This application uses a large model to calculate the probability distribution of each output word segment sequence in a general predefined vocabulary for at least one input text sequence. A first neural network is used to calculate the probability distribution of each output word segment sequence in a vocabulary composed of the input texts. A retrieval device is used to obtain the K text segments with the highest similarity to each input text sequence from an external corpus. A second neural network is used to predict the probability distribution of each output word segment sequence in a third vocabulary based on the sequences of the K text segments corresponding to each input text sequence and each input text sequence. Based on the first, second, and third probability distributions corresponding to each target output word segment sequence, and the corresponding vocabulary, the prediction sequence of at least one output text is calculated, thereby determining each output text. This approach enables the simultaneous use of a general predefined vocabulary, input text sequences, and an external corpus as knowledge sources to generate output text, enriching the types of knowledge sources used for predicting output texts and enhancing the ability of the new large model (composed of the large model, the first neural network model, and the second neural network model) to learn new knowledge.
[0049] On the other hand, this application considers that large models trained on predefined vocabularies learn a wide range of general knowledge. Currently, retraining large models based on external corpora causes them to forget general knowledge and reduces their ability to recognize input text that requires prediction of output text using predefined vocabularies. Therefore, this application uses an objective function to train the first and second neural networks, without training the large model. This enriches the knowledge sources while allowing the large model to retain its original training information, thereby preserving the learned general knowledge. Attached Figure Description
[0050] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is a schematic flowchart illustrating a text generation method and a model training method provided in an exemplary embodiment of this application;
[0052] Figure 2 This is a schematic flowchart illustrating another text method and model training method provided in an exemplary embodiment of this application;
[0053] Figure 3 This is a schematic diagram of an exemplary text generation apparatus provided in an exemplary embodiment of this application. Detailed Implementation
[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0055] Figure 1 This is a schematic flowchart illustrating an exemplary text generation method provided in an exemplary embodiment of this application, such as... Figure 1 As shown, the method includes:
[0056] S110, input at least one input text sequence into the large model, and generate a first probability distribution of each corresponding output word segmentation sequence in the first vocabulary.
[0057] The first vocabulary is a predefined vocabulary used to train the large model. The vocabulary includes multiple associations, each of which is used to associate numbers and word segments; that is, the vocabulary stores each word segment and its corresponding code.
[0058] Since the model can only process numbers and cannot directly process Chinese characters, the input text needs to be converted into numbers according to a vocabulary to generate an input text sequence. After the model calculates the output word segmentation sequence, it needs to find the corresponding word segments in the vocabulary to obtain the output word segments (output word segments can be understood as the units that make up the output text). Therefore, the input to the large model is the input text sequence. In the process of calculating the output word segmentation sequence of the output text based on the input text sequence, the large model generates the first probability distribution of each output word segmentation sequence in the first vocabulary. Here, "corresponding output word segmentation sequence" can be understood as the output word segmentation sequence calculated by the large model.
[0059] It should be understood that the large model is a large model trained based on a general predefined vocabulary. This application retains the training parameters of the large model and does not train the large model again, but only uses the calculation results of the large model regarding the probability distribution.
[0060] For example, suppose the input text sequence is x = {x1, x2, ..., xm}, the corresponding true output text sequence is y = {y1, y2, ..., yn}, and the predicted output text sequence is z = {z1, z2, ..., zn}. Here, each item in x1, x2, ..., xm is a word segmentation sequence of the input text sequence, each item in y1, y2, ..., yn is a true output word segmentation sequence, and each item in z1, z2, ..., zn is a predicted output word segmentation sequence. Using a large model on the input text sequence x, the i-th output word segmentation sequence (token) zi of the predicted output text sequence z is calculated according to the first probability distribution p of the first word list Vl. i,l(zi|x1,x2,...,xm,z1,z2,...,z(i-1)). In calculating the conditional probability distribution of zi, it is necessary to predict zi based on the preceding word segmentation sequences x1,x2,...,xm,z1,z2,...,z(i-1), where x1 to xm are the word segmentation sequences of the input text sequence x, and z1 to z(i-1) are the first i-1 output word segmentation sequences of the output text sequence.
[0061] S120, input at least one input text sequence into the first neural network to generate a second probability distribution of each corresponding output word segmentation sequence in the second vocabulary.
[0062] The second vocabulary is determined based on the text input to the first neural network, which employs an architecture that includes only a decoder. At least one input text and at least one real output text constitute the dataset used to train the first and second neural networks. Here, "the corresponding output word segmentation sequence" can be understood as the output word segmentation sequence calculated by the first neural network.
[0063] For example, after segmenting the input text, at least one segment is obtained. The association between each segment and its corresponding number is used as the content of a second vocabulary.
[0064] For example, the first neural network is trained based on unsupervised learning.
[0065] The first neural network is a Transformer module that only contains a decoder.
[0066] For example, unsupervised training of the first neural network can be achieved through the following formulas (1), (2), (3) and (4).
[0067] L q (z)=∑ i logP(z i |z i-j ,…,z i-1 Formula (1); Θ)
[0068] h0=ZW e +W p Formula (2);
[0069]
[0070]
[0071] Among them, L qLet Θ be the likelihood estimation loss function corresponding to the first neural network, where Θ is the parameter of the conditional probability P. h0 represents the output of the first hidden layer, and Z = (z0 / h0). -j ,…,z -1 Z represents the first j text vectors of each token, where j represents the size of the preceding text window for the i-th token, and W... e W represents the token embedding matrix. p h represents the position embedding matrix. l This indicates the output of the first hidden layer; `transformer_block` represents the Transformer module; `h` l-1 P(z) represents the output of the (l-1)th hidden layer. i ) represents the second probability distribution corresponding to zi, softmax represents the normalization function, and h n This represents the output of the nth hidden layer, and T represents the transpose.
[0072] For example, using a Transformer module containing only a decoder, the next token zi of the predicted sequence z of the output text is calculated according to the input x via formulas (1) to (4) in the second probability distribution p of the second vocabulary Vq. I，q (zi|x1, x2, ..., xm, z1, z2, ..., z(i-1)), the content of the second probability distribution is the same as the description of the first probability distribution, and will not be repeated here.
[0073] In the example above, the first neural network uses a Transformer module containing only a decoder, which can accurately predict the second probability distribution of each output word segmentation sequence in the second vocabulary. Furthermore, since the second vocabulary is determined based on the input text sequence, by calculating the second probability distribution of each output word segmentation sequence in the second vocabulary, the answer to the question is found, enriching the knowledge source of the output text.
[0074] S130, using a retrieval tool, identify the K text segments with the highest similarity to each input text sequence from the external corpus.
[0075] Where K is an integer greater than or equal to 0.
[0076] The external corpus can be updated in real time so that the second neural network can acquire new knowledge in a timely manner.
[0077] For example, the retrieval tool is a dense-based vector recall (DPR) that retrieves the K text segments with the highest similarity to the input text sequence from an external corpus, specifically including:
[0078] The first encoder is used to obtain the input vector representation of the input text;
[0079] Using the second encoder, obtain the s fragment vector representations of s text fragments from the external corpus respectively;
[0080] Perform dot product operations between the input vector representation and each of the s segment vector representations to obtain s similarity scores;
[0081] From s similarity scores, identify the K text segments corresponding to the K highest similarity scores.
[0082] In another feasible design, the first encoder and / or the second encoder are pre-trained language models based on attention mechanisms and employing a bidirectional encoding approach.
[0083] For example, with the first encoder as E P (·), the second encoder is E Q (·) is used as an example. First, use E. P (·) Map any one of the s text segments to a two-dimensional real-valued vector, generating s segment vector representations, and creating an index for retrieving these s segment vector representations. Use E Q (·) Map the input text to a d-dimensional vector to generate the input vector representation, and retrieve which of the s fragment vector representations are closest to the K indices of the input vector representation. The similarity between the input text and each text fragment is defined using the dot product of the fragment vector representation and the input vector representation, as shown in the following formula (5):
[0084] sim(x, p) = E Q (x) T E P (p) Formula (5);
[0085] Where x is the text sequence of the input text, and p is the p-th text segment among s text segments.
[0086] For example, the first encoder is a Bidirectional Encoder Representations from Transformer (BERT) model, and the second encoder is a BERT model.
[0087] In the example above, the dot product of the input vector representation with the s fragment vector representations is used as the similarity between the input text and the s text fragments. Since the calculation method is simple, the above method is fast and suitable for processing a large amount of data.
[0088] S140, using the second neural network, based on the sequence of K text segments corresponding to each input text sequence and each input text sequence, to generate the third probability distribution of each corresponding output word segmentation sequence in the third vocabulary.
[0089] The third vocabulary is determined based on an external corpus. The second neural network uses an architecture that only includes a decoder. The contents of the third vocabulary are described in the explanations of the first and second vocabularys, and will not be repeated here. Here, "the corresponding output word segmentation sequence" can be understood as the output word segmentation sequence calculated by the second neural network. It can be seen that the output word segmentation sequences of the large model, the first neural network, and the second neural network are different. To distinguish the output word segmentation sequences of the three, this application uses "corresponding" for differentiation.
[0090] For example, each word of the K text segments is retrieved in a third vocabulary to determine the code (i.e., number) corresponding to each word in the third vocabulary. The code corresponding to each word constitutes a sequence of K text segments.
[0091] For example, the second neural network is a Transformer module that contains only a decoder.
[0092] In a feasible design, such as Figure 2 As shown, this is achieved by using a second neural network to generate a third probability distribution of each corresponding output word segmentation sequence in the third vocabulary, based on the sequence of K text segments corresponding to each input text sequence and each input text sequence:
[0093] A similarity discriminator is used to determine k text segments from K text segments whose similarity to the corresponding first input text sequence is greater than or equal to a similarity threshold, where 0 ≤ k ≤ K, and k is an integer. The first input text sequence belongs to at least one input text sequence.
[0094] If k = 0, the third probability distribution of the first input text sequence is determined to be 0;
[0095] If k > 0, based on the sequence of k text segments and the first input text sequence, generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0096] To improve the computational speed of the third probability distribution and reduce the computational burden on the neural network, the above embodiment, after acquiring K text segments, uses a similarity discriminator to filter out k text segments from the K text segments whose similarity to the input text sequence is greater than or equal to a similarity threshold, transforming them into a sequence of k text segments w = {w1, w2, ..., wk}. If k = 0, it indicates that the relevance of the external knowledge base to the input text sequence is low, and there is no need for the neural network to calculate the third probability distribution; the third probability distribution can be directly determined as 0. If k > 0, a second neural network is then used to predict the third probability distribution of each output word segment sequence in the third vocabulary based on the sequence of the k text segments and the input text sequence.
[0097] Based on the above embodiments, the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary is generated by the following method, according to the sequence of k text segments and the first input text sequence:
[0098] The first input text sequence and the sequences of each text segment are concatenated to generate k concatenated text sequences.
[0099] Based on k concatenated text sequences, generate the corresponding output word segmentation sequence in the third vocabulary with a third probability distribution.
[0100] For example, the input text sequence and the sequences of each text segment are concatenated separately to generate k concatenated text sequences, including:
[0101] The input text sequence, the sequence of each text segment, and the title of each text segment are concatenated to generate k concatenated text sequences.
[0102] For example, the format of question:title:text fragment can be used for concatenation. Taking the question sequence as x, the text fragment sequences as k1 and k2, the title sequence of k1 as t1, and the title sequence of k2 as t2 as an example, the concatenated text sequences would be x:t1:k1 and x:t2:k2. If k2 has no title, the concatenated text sequence would be x:0:k2.
[0103] The above example inputs the concatenated text sequence into the second neural network, enabling the second neural network to simultaneously learn the features of the input text sequence, the features of each text segment sequence, and the features of the title sequence of each text segment. This allows it to accurately predict the third probability distribution of each output word segment sequence in the third vocabulary, enabling the search for answers from an external knowledge base and further enriching the knowledge sources of the output text.
[0104] In one feasible design, based on k concatenated text sequences, a third probability distribution of the corresponding output word segmentation sequence in the third vocabulary is generated, including:
[0105] If the text fragments are not manually labeled, the state vector output by the last hidden layer is generated by calculating each concatenated text sequence based on unsupervised learning.
[0106] If each text segment is manually labeled, a state vector is generated for each concatenated text sequence based on supervised learning, and the output of the last hidden layer is calculated.
[0107] Each state vector is decoded and linearly transformed to generate an intermediate vector corresponding to each concatenated text sequence;
[0108] Normalize each intermediate vector to generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0109] For example, refer to Figure 2 The state vector H corresponding to each concatenated text sequence is obtained through n layers of decoder-only. k_r Then, H k_r The input is given to a Transformer's decoder and linear transformation layer, where decoding and linear transformation are performed respectively to obtain the intermediate vector e. r The intermediate vectors corresponding to the k concatenated text sequences include e1, e2, and e3. r , ..., e k Finally, the k intermediate vectors are normalized using a softmax normalization layer to obtain the third probability distribution p of the i-th token zi in the third vocabulary of the output text sequence z. i,r (zi|x1,x2,…,xm,…...,wk,…z2,…,z(i-1)) (For an explanation of the third probability distribution, please refer to the first probability distribution; it will not be repeated here). The linear transformation layer, normalization layer, and Transformer decoder belong to the second neural network. The number of linear transformation layers is k, and the number of normalization layers is 1. The k concatenated text sequences are sequentially input into the n-layer decoder-only network.
[0110] For example, referring to formulas (2) to (4) above, the state vector H output by the last hidden layer is obtained by calculating for each concatenated text sequence based on unsupervised learning. k_r H k_r H represents the state vector output by the last hidden layer corresponding to the r-th concatenated text sequence in a set of k concatenated text sequences. k_r =h n =transformer_block(h n-1 ), 1≤r≤k.
[0111] It should be noted that, when calculating for each concatenated text sequence based on supervised learning, after training the second neural network model using formula (1), the model parameters are adjusted to the supervised target object. Assume the training set consisting of labeled text fragments is DataSet C, where each instance contains a series of input tokens, x1,…,x… m And a label y. The input is a second neural network model pre-trained using unsupervised training to obtain the activation values of the final transform block. Then input a parameter W. y A linear output layer is used to predict y, as shown in the following formula (6):
[0112]
[0113] In the example above, an n-layer decoder-only method is used to convert the text into a state vector. The state vector is then mapped onto the third vocabulary through an encoder and a linear transformation layer to obtain the text vector of each output word segmentation sequence. Finally, after normalization, the third probability distribution of the output word segmentation sequence in the third vocabulary is accurately obtained.
[0114] S150, calculate the prediction sequence of at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary.
[0115] In a feasible design, such as Figure 2 As shown, this is achieved by calculating at least one predicted sequence of the output text based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence, as well as the corresponding vocabulary:
[0116] Using a classifier, a predicted label is generated for each target output word segmentation sequence based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence. The predicted label is used to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution, or the third probability distribution.
[0117] Based on the predicted label corresponding to each target output word segmentation sequence, determine the target probability distribution of each output word segmentation sequence;
[0118] Calculate the target output word segmentation sequence for each target based on the target probability distribution and the corresponding vocabulary.
[0119] The target output word segmentation sequences are combined to form the corresponding predicted output text sequence.
[0120] For example, a classifier Where Tansformer(X) represents the first, second, or third probability distribution corresponding to the target output segmented sequence from the n-layer Transformer decoder-only output. (Classification discriminator) This is used to predict which of the three probability distributions—first, second, and third—the target probability distribution belongs to. Prediction label 0 indicates the target probability distribution is the first probability distribution, and the corresponding vocabulary is the first vocabulary. Prediction label 1 indicates the target probability distribution is the second probability distribution, and the corresponding vocabulary is the second vocabulary. Prediction label 2 indicates the target probability distribution is the third probability distribution, and the corresponding vocabulary is the third vocabulary, as shown in the following formula (7):
[0121]
[0122] Among them, pred v denoted by , v is the set of predicted labels, v includes 0, 1 and 2, 0, 1 and 2 respectively represent that the predicted output word segmentation sequence comes from the first word list, the second word list and the third word list.
[0123] For example, the classification discriminator includes a linear transformation layer and a normalization layer.
[0124] For example, this is achieved by determining the target probability distribution of each target output word segmentation sequence based on the predicted labels; and calculating each target output word segmentation sequence based on the target probability distribution of each target output word segmentation sequence and the corresponding vocabulary.
[0125] Based on the predicted label content, determine the target probability distribution for each target output word segmentation sequence. Then, based on the target probability distribution of the target output word segmentation sequence, determine the code corresponding to the highest probability in the target probability distribution from the relevant vocabulary as the output word segmentation sequence.
[0126] This can be achieved through the following formula (8): determining the target probability distribution p of the i-th output word segmentation sequence based on the content of the predicted label. i :
[0127]
[0128] In a feasible design, the objective function includes a weighted cross-entropy loss function, which is determined based on the predicted label and the true label corresponding to each target output word segmentation sequence. The true label is used to identify whether the target probability distribution of each true target output word segmentation sequence is a first probability distribution, a second probability distribution, or a third probability distribution.
[0129] Among them, the actual target output word segmentation sequence constitutes the actual sequence y of the output text.
[0130] For example, the cross-entropy loss function L v As shown in formula (9):
[0131]
[0132] Among them, y v 'pv' represents the true label of the i-th output word segment sequence in the true sequence of the output text, indicating which vocabulary the i-th output word segment sequence comes from, and 'pv' represents the predicted label of the i-th output word segment sequence predicted by the classifier.
[0133] In the above example, predicting the label to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution, or the third probability distribution helps to accurately determine which vocabulary the target output word segmentation sequence comes from.
[0134] S160, determine at least one output text based on the predicted sequence of at least one output text and the corresponding vocabulary.
[0135] For example, when using a classifier to generate predicted labels for each target output word segmentation sequence, the vocabulary corresponding to each target output word segmentation sequence is first determined based on the predicted labels. Then, for each target output word segmentation sequence, the associated target output words are retrieved from the corresponding vocabulary. The various target output words are then combined to form the corresponding output text.
[0136] Based on the text generation embodiments in steps S110 to S160, this application also provides a method for training the model, including the following steps:
[0137] S170, Determine the objective function based on at least one true sequence and one predicted sequence of output text;
[0138] In one feasible design, the objective function is determined based on at least one true sequence and one predicted sequence of output text, including:
[0139] The first loss function corresponding to the second neural network is generated by maximizing the log-likelihood estimate of the true sequence and the predicted sequence of at least one output text.
[0140] Obtain the second loss function corresponding to the first neural network;
[0141] The objective function is determined by the sum of the first loss function, the weighted second loss function, and the weighted cross-entropy loss function.
[0142] For example, the first loss function L d As shown in the following formula (10):
[0143]
[0144] Among them, y n To output the true sequence of text, z n For the predicted sequence of the output text, x n The input text is a sequence of texts, where N is the number of samples and θ is a preset parameter.
[0145] For example, the objective function L is shown in the following formula (11):
[0146]
[0147] Among them, L d Let L be the first loss function. q Let L be the second loss function, σ be the weighting coefficient of the second loss function, and L be the weighting coefficient of the second loss function. v Let cross-entropy be the loss function. These are the coefficients used to weight the cross-entropy loss function. The two coefficients can be designed according to actual needs; this application does not impose any restrictions on them.
[0148] The example above uses an objective function that allows for simultaneous parameter tuning of both the first and second neural networks, enabling simultaneous training of both networks.
[0149] S180, use the objective function to train the first neural network and the second neural network, and obtain the trained first neural network model and the second neural network model respectively.
[0150] As can be seen, the first neural network model is used to predict the output text from the input text, which can be considered a source of knowledge. The second neural network model is used to predict the output text from an external knowledge base, which can also be considered a source of knowledge.
[0151] S190 uses a large model, a trained first neural network model, and a second neural network model to calculate the predicted sequence of the target output text based on the target input text sequence.
[0152] S1100, determine the target output text based on the predicted sequence of the target output text and the corresponding vocabulary.
[0153] Other implementation methods and effects of the training method of the above model can be found in the embodiments of the text generation method, and will not be repeated here.
[0154] This application considers that large models trained on predefined vocabularies learn a wide range of general knowledge. Currently, retraining large models on external corpora causes them to forget general knowledge and reduces their ability to recognize input text that relies on predefined vocabularies for predicting output text. Therefore, this application uses an objective function to train the first and second neural networks, without training the large model. This enriches the knowledge sources while allowing the large model to retain its original training information, thus preserving the learned general knowledge.
[0155] This application uses a large model to calculate the probability distribution of each output word segment sequence in a general predefined vocabulary for at least one input text sequence. A first neural network is used to calculate the probability distribution of each output word segment sequence in a vocabulary composed of the input texts. A retrieval device is used to obtain the K text segments with the highest similarity to each input text sequence from an external corpus. A second neural network is used to predict the probability distribution of each output word segment sequence in a third vocabulary based on the sequences of the K text segments corresponding to each input text sequence and each input text sequence. Based on the first, second, and third probability distributions corresponding to each target output word segment sequence, and the corresponding vocabulary, the prediction sequence of at least one output text is calculated, thereby determining each output text. This approach enables the simultaneous use of a general predefined vocabulary, input text sequences, and an external corpus as knowledge sources to generate output text, enriching the types of knowledge sources used for predicting output texts and enhancing the ability of the new large model (composed of the large model, the first neural network model, and the second neural network model) to learn new knowledge.
[0156] Based on the above method embodiments, such as Figure 3 As shown, this application also provides a text generation apparatus, including:
[0157] The first probability distribution acquisition module is used to input at least one input text sequence into the large model and generate a first probability distribution of each corresponding output word segmentation sequence in the first vocabulary. The first vocabulary is a predefined vocabulary used to train the large model.
[0158] The second probability distribution acquisition module is used to input at least one input text sequence into the first neural network and generate a second probability distribution of each corresponding output word segmentation sequence in the second vocabulary. The second vocabulary is determined based on the text input into the first neural network, and the first neural network adopts an architecture that only includes a decoder.
[0159] The text fragment acquisition module is used to use a retrieval tool to determine the K text fragments with the highest similarity to each input text sequence from the external corpus, where K is an integer greater than or equal to 0;
[0160] The third probability distribution acquisition module is used to generate the third probability distribution of each corresponding output word segment sequence in the third vocabulary based on the sequence of K text segments corresponding to each input text sequence and each input text sequence using the second neural network. The third vocabulary is determined based on the external corpus. The second neural network adopts an architecture that only includes the decoder.
[0161] The prediction sequence calculation module is used to calculate the prediction sequence of at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary.
[0162] A text generation module is used to determine at least one output text based on a predicted sequence of at least one output text and a corresponding vocabulary.
[0163] In one feasible design, the device also includes an objective function determination module and a model training module for implementing model training functionality:
[0164] The objective function determination module is used to determine the objective function based on at least one true sequence and one predicted sequence of output text.
[0165] The model training module is used to train the first neural network and the second neural network using the objective function, and to obtain the trained first neural network model and the second neural network model respectively.
[0166] The prediction sequence calculation module is used to calculate the prediction sequence of the first output text based on the first input text sequence, using a large model, a trained first neural network model, and a second neural network model.
[0167] The text generation module is used to determine the first output text based on the predicted sequence of the first output text and the corresponding vocabulary.
[0168] In a feasible design, the prediction sequence calculation module is implemented by calculating at least one prediction sequence for the output text based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence, as well as the corresponding vocabulary:
[0169] Based on the first probability distribution, second probability distribution and third probability distribution corresponding to each target output word segmentation sequence, a prediction label is generated for each target output word segmentation sequence. The prediction label is used to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution or the third probability distribution.
[0170] Based on the predicted label corresponding to each target output word segmentation sequence, determine the target probability distribution of each target output word segmentation sequence;
[0171] Calculate the target output word segmentation sequence for each target based on the target probability distribution and the corresponding vocabulary.
[0172] The target output word segmentation sequences are combined to form the corresponding predicted output text sequence.
[0173] In a feasible design, the objective function includes a weighted cross-entropy loss function, which is determined based on the predicted label and the true label corresponding to each target output word segmentation sequence. The true label is used to identify whether the target probability distribution of each true target output word segmentation sequence is a first probability distribution, a second probability distribution, or a third probability distribution.
[0174] In one feasible design, the third probability distribution acquisition module is implemented by using a second neural network to generate a third probability distribution of each corresponding output word segment sequence in the third vocabulary based on the sequence of K text segments corresponding to each input text sequence and each input text sequence:
[0175] From K text segments, determine k text segments whose similarity to the corresponding first input text sequence is greater than or equal to a similarity threshold, where 0 ≤ k ≤ K, and k is an integer. The first input text sequence belongs to at least one input text sequence.
[0176] If k = 0, the third probability distribution of the first input text sequence is determined to be 0;
[0177] If k > 0, based on the sequence of k text segments and the first input text sequence, generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0178] In a feasible design, the third probability distribution acquisition module is implemented by generating the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary based on the sequence of k text segments and the first input text sequence:
[0179] The first input text sequence and the sequences of each text segment are concatenated to generate k concatenated text sequences.
[0180] Based on k concatenated text sequences, generate the corresponding output word segmentation sequence in the third vocabulary with a third probability distribution.
[0181] In a feasible design, the third probability distribution acquisition module is implemented by generating the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary based on k concatenated text sequences:
[0182] If the text fragments are not manually labeled, the state vector output by the last hidden layer is generated by calculating each concatenated text sequence based on unsupervised learning.
[0183] If each text segment is manually labeled, a state vector is generated for each concatenated text sequence based on supervised learning, and the output of the last hidden layer is calculated.
[0184] Each state vector is decoded and linearly transformed to generate an intermediate vector corresponding to each concatenated text sequence;
[0185] Normalize each intermediate vector to generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
[0186] In a feasible design, the objective function determination module is implemented by determining the objective function based on at least one true sequence and one predicted sequence of the output text:
[0187] The first loss function corresponding to the second neural network is generated by maximizing the log-likelihood estimate of the true sequence and the predicted sequence of at least one output text.
[0188] Obtain the second loss function corresponding to the first neural network;
[0189] The objective function is determined by the sum of the first loss function, the weighted second loss function, and the weighted cross-entropy loss function.
[0190] In one feasible design, the first neural network model is trained based on unsupervised learning.
[0191] Other implementation methods and effects are described in the embodiments for the text generation method, and will not be repeated here.
[0192] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.
[0193] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0194] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0195] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.
[0196] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0197] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
Claims
1. A text generation method, characterized in that, include: Input at least one input text sequence into a large model and generate a first probability distribution of each corresponding output word segmentation sequence in a first vocabulary, wherein the first vocabulary is a predefined vocabulary used to train the large model. At least one of the input text sequences is input into a first neural network to generate a second probability distribution of each corresponding output word segmentation sequence in a second vocabulary, the second vocabulary being determined based on the text input into the first neural network, the first neural network employing an architecture that includes only a decoder; The retrieval tool is used to identify the K text segments with the highest similarity to each input text sequence from the external corpus, where K is an integer greater than or equal to 0; The second neural network uses a sequence of K text segments corresponding to each input text sequence and each input text sequence to generate a third probability distribution of each corresponding output word segmentation sequence in a third vocabulary, which is determined based on the external corpus. The second neural network adopts an architecture that only includes a decoder. Based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary, a predicted sequence for at least one output text is calculated; wherein, the step of calculating a predicted sequence for at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary includes: Using a classification discriminator, a predicted label is generated for each target output word segmentation sequence based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence. The predicted label is used to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution, or the third probability distribution. Based on the predicted label corresponding to each target output word segmentation sequence, determine the target probability distribution of each target output word segmentation sequence; Calculate each target output word segmentation sequence based on the target probability distribution of each target output word segmentation sequence and the corresponding vocabulary; The target output word segmentation sequences are combined to form the corresponding predicted output text sequences; At least one output text is determined based on the predicted sequence of at least one output text and the corresponding vocabulary.
2. The method of claim 1, wherein, Also includes: The objective function is determined based on at least one true sequence of the output text and the predicted sequence; The first neural network and the second neural network are trained using the objective function to obtain the trained first neural network model and the second neural network model, respectively. Using the large model, the trained first neural network model, and the second neural network model, calculate the predicted sequence of the first output text based on the first input text sequence; The first output text is determined based on the predicted sequence of the first output text and the corresponding vocabulary.
3. The method of claim 2, wherein, The objective function includes a weighted cross-entropy loss function, which is determined based on the predicted label and the true label corresponding to each target output word segmentation sequence. The predicted label is used to identify the target probability distribution of the target output word segmentation sequence as the first probability distribution, the second probability distribution, or the third probability distribution. The true label is used to identify the target probability distribution of each true target output word segmentation sequence as the first probability distribution, the second probability distribution, or the third probability distribution.
4. The method according to any one of claims 1 to 3, characterized in that, The step of using a second neural network to generate a third probability distribution of each corresponding output word segmentation sequence in the third vocabulary based on a sequence of K text segments corresponding to each input text sequence and each input text sequence includes: From the K text segments, determine k text segments whose similarity to the corresponding first input text sequence is greater than or equal to a similarity threshold, where 0 ≤ k ≤ K, and k is an integer, and the first input text sequence belongs to the at least one input text sequence; If k=0, the third probability distribution of the first input text sequence is determined to be 0; If k > 0, based on the sequence of k text segments and the first input text sequence, generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
5. The method of claim 4, wherein, The step of generating a third probability distribution of the corresponding output word segment sequence in the third vocabulary based on the sequence of k text segments and the first input text sequence includes: The first input text sequence and the sequences of each text segment are concatenated to generate k concatenated text sequences; Based on the k concatenated text sequences, a third probability distribution of the corresponding output word segmentation sequence in the third vocabulary is generated.
6. The method of claim 5, wherein, The step of generating the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary based on the k concatenated text sequences includes: If the text fragments are not manually labeled, the state vector output by the last hidden layer is generated by calculating each of the spliced text sequences based on unsupervised learning. If each of the text fragments is manually annotated, a state vector is generated for each of the concatenated text sequences based on supervised learning, and the output of the last hidden layer is generated. Each state vector is decoded and linearly transformed to generate an intermediate vector corresponding to each concatenated text sequence. The intermediate vectors are normalized to generate the third probability distribution of the corresponding output word segmentation sequence in the third vocabulary.
7. The method of claim 3, wherein, The step of determining the objective function based on at least one true sequence of the output text and the predicted sequence includes: A first loss function corresponding to the second neural network is generated by maximizing the log-likelihood estimate of at least one of the true sequences of the output text and the predicted sequence. Obtain the second loss function corresponding to the first neural network; The sum of the first loss function, the weighted second loss function, and the weighted cross-entropy loss function is determined as the objective function.
8. The method of claim 7, wherein, The first neural network was trained using an unsupervised learning method.
9. A text generating apparatus characterized by comprising: include: The first probability distribution acquisition module is used to input at least one input text sequence into the large model and generate a first probability distribution of each corresponding output word segmentation sequence in a first vocabulary, wherein the first vocabulary is a predefined vocabulary used to train the large model. The second probability distribution acquisition module is used to input at least one of the input text sequences into the first neural network and generate a second probability distribution of each corresponding output word segmentation sequence in a second vocabulary, wherein the second vocabulary is determined based on the text input into the first neural network and the first neural network adopts an architecture that only includes a decoder. The text fragment acquisition module is used to use a retrieval device to determine the K text fragments with the highest similarity to each input text sequence from the external corpus, where K is an integer greater than or equal to 0; The third probability distribution acquisition module is used to generate a third probability distribution of each corresponding output word segment sequence in a third vocabulary based on the sequence of K text segments corresponding to each input text sequence and each input text sequence using a second neural network. The third vocabulary is determined according to the external corpus. The second neural network adopts an architecture that only includes a decoder. The prediction sequence calculation module is used to calculate a prediction sequence of at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary; wherein, the step of calculating a prediction sequence of at least one output text based on the first probability distribution, the second probability distribution, the third probability distribution corresponding to each target output word segmentation sequence, and the corresponding vocabulary includes: Using a classification discriminator, a predicted label is generated for each target output word segmentation sequence based on the first probability distribution, the second probability distribution, and the third probability distribution corresponding to each target output word segmentation sequence. The predicted label is used to identify whether the target probability distribution of the target output word segmentation sequence is the first probability distribution, the second probability distribution, or the third probability distribution. Based on the predicted label corresponding to each target output word segmentation sequence, determine the target probability distribution of each target output word segmentation sequence; Calculate each target output word segmentation sequence based on the target probability distribution of each target output word segmentation sequence and the corresponding vocabulary; The target output word segmentation sequences are combined to form the corresponding predicted output text sequences; A text generation module is used to determine at least one output text based on a predicted sequence of at least one output text and a corresponding vocabulary.