Training method of language model, text generation method and related device

Abstract

This application proposes a language model training method, a text generation method, and related apparatus. The method allows the language model to predict the weights of each word in the original sentence, obtaining weight values ​​for each word. Each word's weight represents the semantic importance expressed by that word in the original sentence. Then, a training sentence is input into the language model, enabling it to generate text based on the training sentence. The language model's loss is determined based on the weight values ​​of the masked portion of the original sentence and the generated text, and the model's parameters are adjusted accordingly. This setup allows the model to distinguish between different words and adaptively adjust the semantic importance of each word, rather than indiscriminately counting incorrect words. This enables the language model to truly understand semantics and improve the accuracy of prediction results.

Description

Technical Field

[0001] This application relates to the field of text generation technology, and in particular to a method for training a language model, a text generation method, and related apparatus. Background Technology

[0002] After being trained on text generation, language models can generate complete sentences based on provided corpora. During training, this type of language model typically adjusts its parameters by comparing the number of differing texts between the model's predictions and the actual labels. However, this approach means that the penalty for prediction errors is only related to the number of predictions, not to the true semantic meaning. This prevents the language model from truly understanding semantics and affects the accuracy of the predictions. Summary of the Invention

[0003] In view of this, this application proposes a language model training method, a text generation method, and related apparatus to solve the problem that existing language models cannot truly perform semantic understanding, thus affecting the accuracy of prediction results. The specific technical solution proposed in this application is as follows:

[0004] Firstly, this application provides a method for training a language model, including:

[0005] The original sentence is input into a language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence; the weight of each word represents the importance of the semantics expressed by each word in the original sentence.

[0006] The training statement is input into the language model so that the language model can generate text based on the training statement to obtain the text generation result; wherein, the training statement is obtained by masking the original statement, and the text generation result includes the prediction result of the masked part in the training statement;

[0007] Based on the weight values ​​of the masked portion of the original sentence and the text generation result, the loss of the language model is determined, and the parameters of the language model are adjusted according to the loss of the language model.

[0008] Secondly, this application provides a language model training device, comprising:

[0009] The first input module is used to input the original sentence into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence; the weight of each word represents the importance of the semantics expressed by each word in the original sentence.

[0010] The second input module is used to input the training statement into the language model so that the language model can generate text based on the training statement and obtain the text generation result; wherein, the training statement is obtained by masking the original statement, and the text generation result includes the prediction result of the masked part in the training statement;

[0011] The adjustment module is used to determine the loss of the language model based on the weight values ​​of the masked portion of the original sentence and the text generation result, and to adjust the parameters of the language model based on the loss of the language model.

[0012] Thirdly, this application provides a text generation method, including:

[0013] The target sentence is input into a pre-trained language model, which then generates text based on the target sentence to obtain the text generation result.

[0014] The training process of the language model includes:

[0015] An original sentence is input into a language model to predict the weights of each word in the original sentence, resulting in a weight value for each word. The weight of each word represents the semantic importance expressed by that word in the original sentence. A training sentence is input into the language model to generate text based on the training sentence, resulting in a text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes predictions of the masked portion of the training sentence. The loss of the language model is determined based on the weight values ​​of the masked portion of the original sentence and the text generation result, and the parameters of the language model are adjusted based on the loss.

[0016] Fourthly, this application provides a text generation apparatus, comprising:

[0017] The third input module is used to input the target sentence into a pre-trained language model, so that the language model can generate text based on the target sentence and obtain the text generation result.

[0018] The training process of the language model includes:

[0019] An original sentence is input into a language model to predict the weights of each word in the original sentence, resulting in a weight value for each word. The weight of each word represents the semantic importance expressed by that word in the original sentence. A training sentence is input into the language model to generate text based on the training sentence, resulting in a text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes predictions of the masked portion of the training sentence. The loss of the language model is determined based on the weight values ​​of the masked portion of the original sentence and the text generation result, and the parameters of the language model are adjusted based on the loss.

[0020] Fifthly, this application provides an electronic device, comprising:

[0021] Memory and processor;

[0022] The memory is used to store programs;

[0023] The processor is configured to implement the language model training method described in any one of the above embodiments, and / or the text generation method described in any one of the above embodiments, by running the program in the memory.

[0024] Sixthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the language model training method described in any one of the above claims, and / or the text generation method described in any one of the above claims.

[0025] The language model training method proposed in this application allows the original sentence to be input into the language model, enabling the model to predict the weights of each word in the original sentence, thus obtaining the weight values ​​of each word. The weights of each word represent the semantic importance expressed by each word in the original sentence. Then, a training sentence is input into the language model, allowing it to generate text based on the training sentence. The generated text includes predictions of the masked portions of the training sentence. Based on the weight values ​​of the masked portions of the original sentence and the generated text, the loss of the language model is determined, and the parameters of the language model are adjusted accordingly. This setup enables the model to distinguish different words and adaptively adjust the semantic importance expressed by each word, rather than indiscriminately counting incorrect words. This allows the language model to truly understand semantics, thereby improving the accuracy of prediction results. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating a language model training method provided in an embodiment of this application;

[0028] Figure 2 This is a schematic diagram of the structure of a language model provided in an embodiment of this application;

[0029] Figure 3 This is a flowchart illustrating another language model training method provided in an embodiment of this application;

[0030] Figure 4 This is a flowchart illustrating a text generation method provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of a language model training device provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the structure of a text generation device provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solution of this application embodiment is applicable to application scenarios of training language models. By using the technical solution of this application embodiment to train the language model, the language model can distinguish different words and adaptively adjust the semantic importance of each word, thereby improving the accuracy of prediction results.

[0035] Currently, language model training methods mainly include the following two paradigms:

[0036] First, the two-way paradigm: The two-way paradigm refers to cloze-style text generation. Inputting a text with randomly blanked sections, the language model learns to fill in the blanks with appropriate words. The language model can use information from both sides to determine what the blanked-out words are. For example, inputting "I have __, and my blood pressure is now 150," the language model determines the blanked-out word is "hypertension" based on the information from both sides.

[0037] Second, the one-way paradigm: The one-way paradigm refers to word-by-word text generation. The first half of the text is input (it can be a sentence or a single character), and the language model continuously generates subsequent text, using information from the left side to generate the content on the right. For example, if the input is "I have high blood pressure," the language model, based on the information from the left side, determines that the generated text is "My blood pressure is now 150."

[0038] Regardless of the training paradigm, the model's parameters are adjusted based on the number of discrepancies between the language model's predictions and the ground truth labels. For example, in the bidirectional paradigm example above, the language model might predict high blood sugar, heart disease, or even mumps, while the ground truth label is high blood pressure. Comparing each word of the prediction with each word of the ground truth label yields the following:

[0039] Comparing "high blood sugar" and "high blood pressure," there's a single typo;

[0040] There are three errors in the comparison between "heart disease" and "hypertension";

[0041] Comparing "mumps" and "hypertension," there are three errors in the wording.

[0042] This comparative approach allows language models to learn to understand input text and generate appropriate sentences and vocabulary, significantly improving their comprehension and generation capabilities. However, this method of comparing each character without distinction has a drawback: the penalty for incorrect predictions is only related to the number of predictions, not to the actual semantic meaning. For example, in the example above, both "heart disease" and "mumps" are three characters wrong, and the language model receives the same feedback, but clearly, "heart disease" is more likely to cause a blood pressure of 150.

[0043] Similarly, the same applies to training methods using a unidirectional paradigm. For example, if the original sentence is "I have high blood pressure," and the training label is "My blood pressure is now 150," the language model's prediction might look like this:

[0044] Prediction result one: The low pressure is now 150. Comparing "high" and "low", there is a mistake in one word.

[0045] Prediction result two: The high voltage is now 120. Comparing "150" and "120", there is a one-character error.

[0046] Prediction result three: The high pressure this year is 150. Comparing "this year" and "now", there are two words wrong.

[0047] The number of incorrect predictions was the highest, but it did not affect the overall semantics; on the contrary, predictions one and two did not conform to common medical knowledge.

[0048] Based on this, this application proposes a language model training method, a text generation method, and related apparatus to solve the problem that existing language models cannot truly perform semantic understanding, thus affecting the accuracy of prediction results.

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] This application proposes a method for training a language model. This method can be executed by an electronic device, which can be any device with data and instruction processing capabilities, such as a computer, a smart terminal, or a server. See also Figure 1 As shown, the method includes:

[0051] S101. Input the original sentence into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence.

[0052] The aforementioned original statements refer to the original, complete statements. Specifically, original, complete statements can be obtained from publicly available and anonymized web pages. Furthermore, original statements within the same domain as the language model can be obtained from publicly available and anonymized web pages. For example, if the currently trained language model is applied to the medical field, original statements within the same domain can be obtained from publicly available and anonymized medical-related web pages. For instance, statements such as "I have high blood pressure, my systolic blood pressure is currently 150" and "If you have a fever, drink some ibuprofen" can be obtained as original statements.

[0053] The vocabulary mentioned above refers to the vocabulary in the original sentence. A single vocabulary may include a single character or a phrase; this embodiment does not impose any limitation. In some embodiments, if a single vocabulary includes a phrase, then Natural Language Processing (NLP) technology can be used to divide the original sentence into multiple phrases. The weight of each vocabulary represents the importance of the semantics expressed by each vocabulary in the original sentence. The higher the weight value of a vocabulary, the more important the semantics expressed by that vocabulary are in the overall semantics expressed by the original sentence.

[0054] In the embodiments of this application, the original sentence is input into a language model so that the language model predicts the weight of each word in the original sentence, thereby obtaining the weight value of each word in the original sentence. The higher the word weight value output by the language model, the more important the semantics expressed by that word is in the overall semantics expressed by the original sentence, according to the language model.

[0055] In some alternative embodiments, no penalty may be applied to the predicted weight values; that is, the language model may not be specifically trained for weight prediction. For example, if a single word consists of one character and the original sentence is "I have high blood pressure, and my systolic blood pressure is now 150", the language model may predict the weight values ​​of each word as [1, 1, 2, 2, 2, 0.5, 0.2, 0.3, 3, 3, 3, 0.1], or it may predict the weight values ​​of each word as [2, 1, 1, 3, 3, 3, 0.7, 1, 1, 1, 2, 0.5], etc.

[0056] S102. Input the training statement into the language model so that the language model can generate text based on the training statement and obtain the text generation result.

[0057] The training statements described above are obtained by masking the original statements. The masked portion can be set according to the actual situation, and this embodiment does not impose any limitations. For example, if the original statement is "I have high blood pressure, and my systolic blood pressure is 150," the second half of the text can be masked to obtain the training statement "I have high blood pressure."

[0058] The aforementioned text generation results include the language model's prediction results for the masked portions of the training statement. In the embodiments of this application, the training statement is input into the language model so that the language model generates text based on the training statement and obtains the prediction results for the masked portions of the training statement. For example, if the training statement is "I have high blood pressure", inputting the training statement into the language model might result in the language model predicting "My diastolic blood pressure is 150 now".

[0059] In the embodiments of this application, the language model includes at least two branches. One branch is used to obtain the original sentence and predict the weight values ​​of each word in the original sentence. The other branch is used to obtain the training sentence, perform text generation, and predict the content of the masked part in the training sentence.

[0060] Figure 2 This is a schematic diagram of the structure of a language model provided in an embodiment of this application. For example... Figure 2The language model shown includes a backbone network, a weight prediction function h(x), and a text generation function f(x). The backbone network and the weight prediction function form one branch of the language model, used to predict the weight values ​​of each word in the original sentence. The backbone network and the text generation function form another branch of the language model, used to generate text and predict the content of the masked portion in the training sentence.

[0061] For example, such as Figure 2 As shown, if the original sentence is "I have high blood pressure, and my systolic blood pressure is 150," and the training sentence obtained after masking the original sentence is "I have high blood pressure," then "I have high blood pressure, and my systolic blood pressure is 150" can be input into the language model. The backbone network and the weight prediction function branch of the language model predict the weight values ​​of each word in the original sentence as [1, 1, 2, 2, 2, 0.5, 0.2, 0.3, 3, 3, 3, 0.1]. When "I have high blood pressure" is input into the language model, the backbone network and the text generation function branch of the language model predict the content of the masked part in the training sentence, and the resulting text generation result is "My diastolic blood pressure is 150."

[0062] S103. Based on the weight values ​​of the masked parts of the original sentences and the text generation results, determine the loss of the language model, and adjust the parameters of the language model according to the loss of the language model.

[0063] Furthermore, in determining the loss of the language model, in the embodiments of this application, regardless of the weight values ​​of each word obtained in the previous step, the loss of the language model is determined based on the weight values ​​of the original sentence masking part and the text generation result.

[0064] Specifically, the masked portion of the original sentence can be used as the training label for text generation training. Based on the difference between the training label and the generated text, the text generation loss for each word in the generated text is calculated. For example, the cross-entropy loss for each word in the generated text can be calculated as the text generation loss for that word; this embodiment is not limited to this.

[0065] By utilizing the weights of each word in the generated text, the text generation loss corresponding to that word is corrected, resulting in the corrected text generation loss for each word in the generated text. Here, the weight of each word in the generated text refers to the weight of the word at the same position in the masked portion of the original sentence.

[0066] For example Figure 2In the illustrated embodiment, for the masked part of the original statement "Now the high pressure is 150", the weights corresponding to each word in the masked part predicted by the language model are: the weight corresponding to "Now" is 0.2, the weight corresponding to "in" is 0.3, the weight corresponding to "high" is 3, the weight corresponding to "blood" is 3, the weight corresponding to "pressure" is 150, and the weight corresponding to "now" is 0.1.

[0067] For the text generation result "Now the low pressure is 150" obtained by prediction, the word "Now" in the masked part is in the same position as the word "Now" in the text generation result, the word "in" in the masked part is in the same position as the word "in" in the text generation result, the word "high" in the masked part is in the same position as the word "low" in the text generation result, the word "blood" in the masked part is in the same position as the word "blood" in the text generation result, the word "pressure" in the masked part is in the same position as the word "pressure" in the text generation result, the word "150" in the masked part is in the same position as the word "150" in the text generation result, and the word "now" in the masked part is in the same position as the word "now" in the text generation result.

[0068] Then correspondingly, in the text generation result, the weight corresponding to "Now" is 0.2, the weight corresponding to "in" is 0.3, the weight corresponding to "low" is 3, the weight corresponding to "blood" is 3, the weight corresponding to "pressure" is 150, and the weight corresponding to "now" is 0.1.

[0069] After separately calculating the text generation loss corresponding to each word in "Now the low pressure is 150", then using the weight of each word in the text generation result to correct the text generation loss corresponding to the corresponding word, the text generation loss correction result corresponding to each word in the text generation result is obtained. For example, if in the text generation result, the text generation loss corresponding to "Now" is X1, then X1 is corrected using the weight 0.2 corresponding to "Now"; the text generation loss corresponding to "in" is X2, then X2 is corrected using the weight 0.3 corresponding to "in"; the text generation loss corresponding to "low" is X3, then X3 is corrected using the weight 3 corresponding to "low", and so on.

[0070] In the text generation result, the higher the weight of a word, the greater the importance of the semantic meaning corresponding to that word. Correcting the text generation loss for each word using its weight means adjusting the text generation loss based on the importance of each word in the text generation result. The corrected text generation loss for that word can be determined by multiplying its weight by the text generation loss for each word in the text generation result, or by summing its weight and the text generation loss for each word in the text generation result. This embodiment does not limit the specific method used.

[0071] The language model loss is determined based on the text generation loss correction results corresponding to each word. Specifically, the sum of the text generation loss correction results corresponding to each word in the text generation result can be calculated. The sum of the text generation loss correction results corresponding to each word in the text generation result is determined as the language model loss, so that the parameters of the language model can be adjusted according to the language model loss.

[0072] Specifically, the parameters of the language model can be adjusted in the direction of reducing the language model loss. After the adjustment is completed, steps S101 to S103 above can be repeated to continuously reduce the loss until the loss of the language model is less than the set loss value. The set loss value can be set according to the actual situation, and this embodiment does not limit it.

[0073] Training a language model in this way allows iterative training to discover data patterns and attribute characteristics. Besides learning to predict text, it can gradually learn to distinguish words with different semantic meanings and the relative importance of those meanings within a sentence, focusing on words of high importance. In other words, the language model trained using the technical solution in this embodiment can generate text based on semantic understanding, significantly improving the accuracy of the generated results.

[0074] The above-described language model training method can greatly reduce human intervention, allowing the model to discover data patterns and attribute characteristics from large-scale corpus training. However, the drawback is slow convergence speed, requiring longer training and iteration times. To address this issue, another embodiment of this application discloses a language model training method, such as... Figure 3 As shown, the specific steps may include the following:

[0075] S301. Using the original sentence as a training sample and the weight values ​​of each word in the original sentence as training labels, the language model is trained to predict weights.

[0076] Specifically, in the embodiments of this application, the language model can be trained first for weight prediction. After the language model's weight prediction training is completed, the original sentence is then input into the language model to obtain the weight values ​​of each word in the original sentence output by the language model. The weight values ​​of each word in the original sentence output by the language model trained for weight prediction will be more accurate.

[0077] A large number of raw sentences can be obtained as training samples for weight prediction training. Then, the actual weight values ​​of each word in the raw sentences are labeled as training labels for weight prediction training. For example, raw sentences relevant to the application domain of the language model can be obtained as training samples for weight prediction training. Alternatively, a domain-specific vocabulary can be collected to form a domain-specific vocabulary list, and different weight values ​​can be assigned to different words to label the actual weight values ​​of each word in the raw sentences according to the domain-specific vocabulary list. For example, the actual weight value of ordinary words in the raw sentences is labeled as 1, while the actual weight value of domain-related words is labeled as 2 or higher according to the domain-specific vocabulary list.

[0078] When training a language model for weight prediction, the main focus is on... Figure 2 The backbone network and the weight prediction function branch are trained. During the weight prediction training process, the training samples of the weight prediction training are input into the language model to obtain the prediction results output by the language model. By comparing the prediction results with the training samples of the weight prediction training, the weight prediction training loss of the language model is determined.

[0079] The formula for weighted prediction of training loss is as follows:

[0080] w i =h(backbone(x) i ))

[0081] loss1 = e i *log(w i )

[0082] In the above formula, e i w represents the actual weight value of the i-th word. i is the weight value of the corresponding word in the prediction result output by the language model, and loss1 is the training loss of the weight prediction.

[0083] To reduce the training loss of weight prediction, the parameters of the language model are adjusted. After adjustment, the training process of weight prediction is repeated until the training loss of weight prediction tends to converge. At this point, the language model has learned the ability to distinguish the semantic importance of different words in a sentence.

[0084] S302. Input the original sentence into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence.

[0085] S303. Input the training statement into the language model so that the language model can generate text based on the training statement and obtain the text generation result.

[0086] S304. Based on the weight values ​​of the masked portion of the original sentence and the text generation result, determine the loss of the language model, and adjust the parameters of the language model according to the loss of the language model.

[0087] The specific execution process of steps S302-S304 is the same as that of steps S101-S103 in the above embodiments. Those skilled in the art can refer to the description in the above embodiments for the specific execution process, which will not be repeated here.

[0088] Specifically, after the weight prediction training is completed, text generation training can be performed according to steps S101-S103 in the above embodiments. The original text is input into the language model so that the language model can predict the weights of each word in the original sentence. Then, the training sentence is input into the language model so that the language model can generate text based on the training sentence and obtain the text generation result. Based on the weight values ​​of the masked part of the original sentence and the text generation result, the loss of the language model is determined, and the parameters of the language model are adjusted according to the loss of the language model.

[0089] In the above embodiments, the language model is trained on weight prediction before text generation training. This method of training on weight prediction before text generation training can effectively improve the convergence speed of the language model and reduce the number of iterations and training time.

[0090] As an optional implementation, the steps in the above embodiments determine the loss of the language model based on the weight values ​​of the masked portion of the original sentence and the text generation result, and may specifically include the following steps:

[0091] Based on the difference between the generated text and the masked portion of the original sentence, the text generation loss of the language model is determined; based on the text generation loss and the weight values ​​of the masked portion of the original sentence, the loss of the language model is determined.

[0092] Specifically, in the embodiments of this application, when determining the loss of the language model, the text generation loss of the language model is first determined based on the difference between the generated text and the masked portion of the original sentence. For example, the cross-entropy loss of the language model can be determined as the text generation loss of the language model based on the difference between the generated text and the masked portion of the original sentence.

[0093] Then, based on the text generation loss and the weights of the masked portion of the original sentence, the language model loss is determined. As described in the above embodiment, the text generation loss for each word in the generated text can be determined by multiplying the text generation loss for that word by its weight, or by summing the text generation losses for each word in the generated text and their weights. This embodiment does not limit this approach. Then, the sum of the text generation losses for each word is calculated as the text generation loss of the language model.

[0094] In the above embodiments, the loss of the language model can be determined based on the text generation loss and the weight values ​​of the masked parts of the original sentence, so that the language model can distinguish different words and adaptively adjust the importance of the semantics expressed by each word.

[0095] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments determine the loss of the language model based on the text generation loss and the weight values ​​of the original sentence masking portion, which may specifically include the following steps:

[0096] Calculate the weighted sum of the text generation loss for each word in the generated text, and determine the language model loss based on the weighted sum. The weight of the text generation loss for each word in the generated text is the same as the weight of the word at the same position in the masked portion.

[0097] Specifically, in the embodiments of this application, the text generation loss corresponding to each word in the text generation result is first calculated, then the weighted sum of the text generation losses corresponding to each word in the text generation result is calculated, and finally, the loss of the language model is determined based on the above weighted sum. In some embodiments, the weighted sum of the text generation losses corresponding to each word in the text generation result can be determined as the loss of the language model, that is, the loss of the language model in the text prediction training process. The specific calculation formula is as follows:

[0098] w i =h(backbone(x) i ))

[0099]

[0100] In the above formula, w i y represents the weight value of the i-th word in the prediction results output by the language model. i denoted as , where is the predicted probability of the corresponding word, N is the number of words in the generated text, and loss2 is the training loss for text prediction.

[0101] In some embodiments, if the language model is trained only on text generation, then the loss of the language model during text generation training is loss2; in some embodiments, if the language model is trained on both text generation and weight prediction, with the loss for weight prediction training being loss1 and the loss for text generation training being loss2, then the loss of the language model is:

[0102] loss = loss1 + loss2

[0103] In the above embodiments, a weighted sum of the text generation loss corresponding to each word in the text generation result can be calculated, so as to determine the loss of the language model based on the weighted sum, thereby enabling the language model to distinguish different words and adaptively adjust the importance of the semantics expressed by each word.

[0104] As an optional implementation, another embodiment of this application discloses that the steps of the above embodiments, which determine the loss of the language model based on a weighted sum, may specifically include the following steps:

[0105] The weighted sum is normalized to obtain the loss of the language model.

[0106] Specifically, in order to achieve relative differences within a text, a language model might directly predict all weight values ​​as 0 during the learning process in order to minimize the loss during text prediction training. In this case, the language model cannot obtain normal training feedback. To solve the above problem, the embodiments of this application normalize the weighted sum, and the normalized weighted sum is used as the loss of the language model. In some embodiments, the weights of the text generation loss corresponding to each word in the text generation result can be normalized to the range [0,1] to achieve the purpose of normalizing the weighted sum.

[0107] In some other embodiments, as an optional implementation, another embodiment of this application discloses that the steps of the above embodiments normalize the weighted sum to obtain the loss of the language model, which may specifically include the following steps:

[0108] Calculate the ratio between the weighted sum and the weighted sum of the text generation loss for each word in the text generation result, and determine the ratio as the loss of the language model.

[0109] Specifically, in the embodiments of this application, when normalizing the weighted sum, in addition to normalizing the weights of the text generation loss corresponding to each word in the text generation result to the range [0,1] according to the steps of the above embodiments, the normalization purpose can also be achieved by calculating the ratio between the weighted sum and the weighted sum of the text generation loss corresponding to each word in the text generation result. The specific formula is as follows:

[0110]

[0111] In the above embodiments, by normalizing the weighted sum, the language model may directly predict all weight values ​​as 0 in order to minimize the loss during the text prediction training process.

[0112] As an optional implementation, another embodiment of this application discloses that the training of the language model in the above embodiments may specifically include the following steps:

[0113] The test statement is input into the language model so that the language model can generate text based on the test statement and obtain the text test result; the training result of the language model is determined based on the text test result.

[0114] After the language model training phase is complete, it needs to be tested. Specifically, the test statements from the testing phase can be input into the language model for testing. The test statements are the same as the training statements in the above embodiments, both of which are obtained by masking the original statements.

[0115] After receiving the input test sentence, the language model generates text only for the masked portion of the test sentence, obtaining the text test result. This text test result is used to determine the training result of the language model. Specifically, when the text test result output by the language model exceeds the preset expectation, for example, when the accuracy of the text test result reaches the set accuracy, the training result of the language model is determined to be complete. When the text test result output by the language model does not exceed the preset expectation, for example, when the accuracy of the text test result does not reach the set accuracy, the training result of the language model is determined to be a failure, and training needs to continue until the training result of the language model exceeds the preset expectation.

[0116] The accuracy rate set above can be adjusted according to actual conditions, such as setting it to 98%, but this embodiment does not limit it.

[0117] The training of the language model mentioned above can include pre-training or downstream training of the language model.

[0118] In one specific embodiment, two training methods for the language model are provided:

[0119] Method 1: End-to-end training. End-to-end training includes a training phase and a testing phase.

[0120] During the training phase, the original sentence is first input into the language model to predict the weights of each word in the original sentence, thus obtaining the weight values ​​of each word. Then, the training sentence is input into the language model to generate text based on it, yielding the generated text. The training sentence is obtained by masking the original sentence, and the generated text includes predictions of the masked portions of the training sentence. Based on the difference between the generated text and the masked portions of the original sentence, the text generation loss of the language model is determined. A weighted sum of the text generation losses for each word in the generated text is calculated, and the weights of the text generation losses for each word in the generated text are the same as the weights of words in the same position in the masked portions. The ratio between this weighted sum and the weighted sum of the text generation losses for each word in the generated text is calculated, and this ratio is determined as the language model loss, allowing for the adjustment of the language model parameters based on the language model's loss.

[0121] During the testing phase, the test statement is input into the language model so that the language model can generate text based on the test statement and obtain the text test result. Then, based on the text test result, the training result of the language model is determined.

[0122] The language model training method described in Method 1 can greatly reduce human intervention, allowing the model to discover data patterns and attribute characteristics from large-scale corpus training. However, its drawback is slow convergence speed, requiring longer training and iteration times. To address this issue, the embodiments of this application provide Method 2 as follows.

[0123] Method 2: Cascaded training. Cascaded training also includes a training phase and a testing phase.

[0124] During the training phase, the original sentences are used as training samples, and the weight values ​​of each word in the original sentences are used as training labels. Weight prediction training is performed on the language model. When the weight prediction loss in the weight prediction training converges, end-to-end training is then performed. The end-to-end training method in Method Two is the same as the end-to-end training phase provided in Method One of the above embodiments. Those skilled in the art can refer to the descriptions in the above embodiments, and will not be repeated here. In this embodiment, the prediction results of the weight prediction training are used to optimize the end-to-end training, and then the output of the end-to-end training is used to calculate the overall loss to optimize the weight prediction training, thus achieving self-supervised training of the language model.

[0125] During the testing phase, only the text generation task needs to be performed. The test statement is input into the language model so that the language model can generate text based on the test statement, obtain the text test result, and then determine the training result of the language model based on the text test result.

[0126] This application proposes a text generation method, which can be executed by an electronic device. The electronic device can be any device with data and instruction processing capabilities, such as a computer, a smart terminal, or a server. See also... Figure 4 As shown, the method includes:

[0127] S401. Input the target sentence into the pre-trained language model so that the language model can generate text based on the target sentence and obtain the text generation result.

[0128] The training process of the language model includes:

[0129] The original sentence is input into the language model, which predicts the weights of each word in the original sentence, thus obtaining the weight values ​​of each word. The weight of each word represents the importance of the semantics expressed by each word in the original sentence. The training sentence is input into the language model, which generates text based on the training sentence, thus obtaining the text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes the prediction results of the masked part of the training sentence. Based on the weight values ​​of the masked part of the original sentence and the text generation result, the loss of the language model is determined, and the parameters of the language model are adjusted according to the loss of the language model.

[0130] The target statement mentioned above refers to the statement used for text generation. The target statement is input into a pre-trained language model, which can then generate text based on the target statement, yielding the generated text result.

[0131] For specific limitations on text generation methods, please refer to the limitations on language model training methods mentioned above, which will not be repeated here.

[0132] Corresponding to the above-described language model training method, this application also discloses a language model training apparatus, see [link to relevant documentation]. Figure 5 As shown, the device includes:

[0133] The first input module 100 is used to input the original sentence into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence; the weight of each word represents the importance of the semantics expressed by each word in the original sentence.

[0134] The second input module 110 is used to input the training statement into the language model so that the language model can generate text based on the training statement and obtain the text generation result; wherein, the training statement is obtained by masking the original statement, and the text generation result includes the prediction result of the masked part in the training statement.

[0135] The adjustment module 120 is used to determine the loss of the language model based on the weight values ​​of the masked part of the original sentence and the text generation result, and to adjust the parameters of the language model based on the loss of the language model.

[0136] As an optional implementation, another embodiment of this application discloses that the language model training device of the above embodiments further includes:

[0137] The training module is used to train the language model by taking the original sentence as a training sample and the weight values ​​of each word in the original sentence as training labels before the language model predicts the weights of each word in the original sentence and before the original sentence is input into the language model.

[0138] As an optional implementation, another embodiment of this application discloses that the adjustment module 120 of the above embodiments, when determining the loss of the language model based on the weight values ​​of the original sentence masking portion and the text generation result, is specifically used for:

[0139] Based on the difference between the generated text and the masked portion of the original sentence, the text generation loss of the language model is determined; based on the text generation loss and the weight values ​​of the masked portion of the original sentence, the loss of the language model is determined.

[0140] As an optional implementation, another embodiment of this application discloses that the adjustment module 120 of the above embodiments, when determining the loss of the language model based on the text generation loss and the weight values ​​of the original sentence masking portion, is specifically used for:

[0141] Calculate the weighted sum of the text generation loss for each word in the generated text; the weight of the text generation loss for each word in the generated text is the same as the weight of the word in the same position in the masked part; determine the loss of the language model based on the weighted sum.

[0142] As an optional implementation, another embodiment of this application discloses that the adjustment module 120 of the above embodiments, when determining the loss of the language model based on the weighted sum, is specifically used for:

[0143] The weighted sum is normalized to obtain the loss of the language model.

[0144] As an optional implementation, another embodiment of this application discloses that the adjustment module 120 of the above embodiments, when normalizing the weighted sum to obtain the loss of the language model, is specifically used for:

[0145] Calculate the ratio between the weighted sum and the weighted sum of the text generation loss for each word in the text generation result, and determine the ratio as the loss of the language model.

[0146] As an optional implementation, another embodiment of this application discloses that the language model training device of the above embodiments further includes:

[0147] The testing module is used to input test statements into the language model, so that the language model can generate text based on the test statements and obtain text test results; based on the text test results, the training results of the language model are determined.

[0148] Specifically, for the specific working content of each unit of the above-mentioned language model training device, please refer to the content of the above-mentioned language model training method embodiment, which will not be repeated here.

[0149] Corresponding to the above-described text generation method, this application also discloses a text generation apparatus, see [link to apparatus]. Figure 6 As shown, the device includes:

[0150] The third input module 200 is used to input the target sentence into the pre-trained language model so that the language model can generate text based on the target sentence and obtain the text generation result.

[0151] The training process of a language model includes:

[0152] The original sentence is input into the language model, which predicts the weights of each word in the original sentence, thus obtaining the weight values ​​of each word. The weight of each word represents the importance of the semantics expressed by each word in the original sentence. The training sentence is input into the language model, which generates text based on the training sentence, thus obtaining the text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes the prediction results of the masked part of the training sentence. Based on the weight values ​​of the masked part of the original sentence and the text generation result, the loss of the language model is determined, and the parameters of the language model are adjusted according to the loss of the language model.

[0153] Specifically, for the specific working content of each unit of the above-mentioned text generation device, please refer to the content of the above-mentioned text generation method embodiment, which will not be repeated here.

[0154] This application also proposes a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input / output component through the interface circuit of the control device.

[0155] The input / output component specifically refers to the hardware component that enables the user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, monitor, speaker, printer, etc.

[0156] The aforementioned interface circuit can be any interface circuit capable of implementing data communication functions, such as a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIe circuit, etc.

[0157] The processor in the control device is a circuit with signal processing capabilities, which is used to execute any of the language model training methods described in the above embodiments, and / or execute any of the text generation methods described in the above embodiments.

[0158] When the control device is applied to a device with human-computer interaction function, the input and output components of the control device can be input components and output components on the device, such as microphone, keyboard, handwriting tablet, touch screen, display, audio player, etc. Meanwhile, the processor of the control device can be the device's built-in CPU or GPU, etc., and the interface circuit of the control device can be the interface circuit between the device's information input components and the CPU or GPU processor.

[0159] Corresponding to the above method, this application also discloses an electronic device, see [link to relevant documentation]. Figure 7 As shown, the electronic device includes:

[0160] Memory 300 and processor 310;

[0161] The memory 300 is connected to the processor 310 and is used to store programs;

[0162] The processor 310 is configured to implement the language model training method disclosed in any of the above embodiments, and / or implement the text generation method disclosed in any of the above embodiments, by running a program stored in the memory 300.

[0163] Specifically, the aforementioned electronic device may further include: a bus, a communication interface 320, an input device 330, and an output device 340.

[0164] The processor 310, memory 300, communication interface 320, input device 330, and output device 340 are interconnected via a bus. Among them:

[0165] A bus can include a pathway for transmitting information between various components of a computer system.

[0166] The processor 310 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0167] Processor 310 may include a main processor, as well as a baseband chip, modem, etc.

[0168] The memory 300 stores a program for executing the technical solution of this application, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 300 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0169] Input device 330 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0170] Output device 340 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0171] The communication interface 320 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0172] The processor 310 executes the program stored in the memory 300 and calls other devices, which can be used to implement the language model training method disclosed in any of the above embodiments, and / or to implement the text generation method disclosed in any of the above embodiments.

[0173] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the language model training method disclosed in any of the above embodiments, and / or to perform the text generation method disclosed in any of the above embodiments.

[0174] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0175] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the language model training method disclosed in any of the above embodiments, and / or, perform the text generation method disclosed in any of the above embodiments.

[0176] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0177] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0178] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0179] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0180] The modules and sub-modules in the apparatus and terminal in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0181] It should be understood, in the several embodiments provided in this application, that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative; for instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0182] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0183] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0184] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0185] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0186] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training a language model, characterized in that, include: The original sentence is input into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence. The weight of each word represents the importance of the semantics expressed by each word in the original sentence; The training statement is input into the language model so that the language model can generate text based on the training statement to obtain the text generation result; wherein, the training statement is obtained by masking the original statement, and the text generation result includes the prediction result of the masked part in the training statement; Based on the difference between the generated text and the masked portion of the original sentence, the text generation loss of the language model is determined. Based on the text generation loss and the weight values ​​of the masked portion of the original sentence, the loss of the language model is determined. Based on the loss of the language model, the parameters of the language model are adjusted.

2. The method according to claim 1, characterized in that, Before inputting the original sentence into the language model to predict the weights of each word in the original sentence and obtain the weight values ​​of each word in the original sentence, the method further includes: The original sentence is used as a training sample, and the weight values ​​of each word in the original sentence are used as training labels to train the language model for weight prediction.

3. The method according to claim 1, characterized in that, The step of determining the loss of the language model based on the text generation loss and the weight values ​​of the original sentence masking portion includes: Calculate the weighted sum of the text generation loss corresponding to each word in the text generation result; the weight of the text generation loss corresponding to each word in the text generation result is the same as the weight of the words at the same position in the masked part; The loss of the language model is determined based on the weighted sum.

4. The method according to claim 3, characterized in that, Determining the loss of the language model based on the weighted sum includes: The weighted sum is normalized to obtain the loss of the language model.

5. The method according to claim 4, characterized in that, The normalization process for the weighted sum to obtain the loss of the language model includes: Calculate the ratio between the weighted sum and the weighted sum of the text generation loss corresponding to each word in the text generation result, and determine the ratio as the loss of the language model.

6. The method according to claim 1, characterized in that, Also includes: The test statement is input into the language model so that the language model generates text based on the test statement and obtains the text test result; Based on the text test results, the training results of the language model are determined.

7. A training device for a language model, characterized in that, include: The first input module is used to input the original sentence into the language model so that the language model can predict the weight of each word in the original sentence and obtain the weight value of each word in the original sentence. The weight of each word represents the importance of the semantics expressed by each word in the original sentence; The second input module is used to input the training statement into the language model so that the language model can generate text based on the training statement and obtain the text generation result; wherein, the training statement is obtained by masking the original statement, and the text generation result includes the prediction result of the masked part in the training statement; The adjustment module is used to determine the text generation loss of the language model based on the difference between the text generation result and the original sentence masking part, determine the loss of the language model based on the text generation loss and the weight value of the original sentence masking part, and adjust the parameters of the language model based on the loss of the language model.

8. A text generation method, characterized in that, include: The target sentence is input into a pre-trained language model, which then generates text based on the target sentence to obtain the text generation result. The training process of the language model includes: An original sentence is input into a language model to predict the weights of each word in the original sentence, resulting in a weight value for each word. The weight of each word represents the semantic importance expressed by that word in the original sentence. A training sentence is input into the language model to generate text based on the training sentence, resulting in a text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes predictions of the masked portion of the training sentence. The text generation loss of the language model is determined based on the difference between the text generation result and the masked portion of the original sentence. The loss of the language model is then determined based on the text generation loss and the weight values ​​of the masked portion of the original sentence. Finally, the parameters of the language model are adjusted based on the loss.

9. A text generation device, characterized in that, include: The third input module is used to input the target sentence into a pre-trained language model, so that the language model can generate text based on the target sentence and obtain the text generation result. The training process of the language model includes: An original sentence is input into a language model to predict the weights of each word in the original sentence, resulting in a weight value for each word. The weight of each word represents the semantic importance expressed by that word in the original sentence. A training sentence is input into the language model to generate text based on the training sentence, resulting in a text generation result. The training sentence is obtained by masking the original sentence, and the text generation result includes predictions of the masked portion of the training sentence. The text generation loss of the language model is determined based on the difference between the text generation result and the masked portion of the original sentence. The loss of the language model is then determined based on the text generation loss and the weight values ​​of the masked portion of the original sentence. Finally, the parameters of the language model are adjusted based on the loss.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to implement the language model training method as described in any one of claims 1 to 6, and / or the text generation method as described in claim 8, by running a program in the memory.

11. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the language model training method as described in any one of claims 1 to 6, and / or the text generation method as described in claim 8.

Images