Text generation model training method, text generation method and device

Abstract

The present disclosure relates to a text generation model training method, a text generation method and a device. The method can include: performing correctness discrimination training on a text generation model based on a correctness discrimination task, the correctness discrimination task being used to adjust model parameters based on text content correctness discrimination; and taking the model parameters obtained through the correctness discrimination training as initialization parameters, and continuing to train the text generation model based on a text generation task, the text generation task being used to adjust the model parameters based on correctness of text generated by the text generation model. Through the technical solution of the present disclosure, the text generation model can be effectively prevented from generating incorrect content and repeatedly correcting, meaningless trial-and-error output is reduced, the redundancy of output content is reduced on the basis of ensuring the accuracy of the output content, and the text generation efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a training method for a text generation model, a text generation method, and an apparatus. Background Technology

[0002] With the continuous development of artificial intelligence technology, text generation models have been widely used in tasks such as question answering, story generation, and dialogue. However, in practical applications, it has been found that their accuracy is low in complex reasoning scenarios.

[0003] Currently, in order to improve the accuracy of text generation models in complex reasoning scenarios, reinforcement learning from verifiable rewards (RLVF) is commonly used to train text generation models. This method guides text generation models to explore reasoning strategies beyond supervised demonstrations by providing sparse but automatically verifiable supervision signals (such as whether the final answer is correct), thereby improving the accuracy of text generation models to some extent.

[0004] However, this training method also leads text generation models to tend to find the final correct answer by enumerating multiple hypotheses and repeatedly correcting intermediate steps, resulting in lengthy output content and significantly increasing inference latency. Summary of the Invention

[0005] This disclosure provides a training method for a text generation model, a text generation method, and an apparatus to address the shortcomings of related technologies.

[0006] According to a first aspect of the present disclosure, a method for training a text generation model is proposed, the method comprising: Based on the correctness judgment task, the text generation model is trained to judge correctness. The correctness judgment task is used to adjust the model parameters based on the correctness of the text content correctness judgment. The model parameters obtained from the correctness discrimination training are used as initialization parameters. Based on the text generation task, the text generation model is further trained. The text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

[0007] In some possible implementations, the correctness-based training of the text generation model includes: Get the first text, where the first text is any text; The first text is input into the text generation model to obtain the discrimination result output by the text generation model. The discrimination result is used to indicate whether the content of the first text is correct. Based on the discrimination results, the text generation model is trained.

[0008] In some possible implementations, the first text includes: a question text and the corresponding answer content, wherein the first text is used to instruct the text generation model to determine whether the answer content is correct based on the question text; The solution content includes at least one of the reasoning text corresponding to the question text and the answer text.

[0009] In some possible implementations, obtaining the first text includes: Input the question text into the large language model to obtain the answer content output by the large language model; The question text is combined with the answer output by the large language model to obtain the first text.

[0010] In some possible implementations, there are multiple large language models, which are large language models with different network architectures, different parameter scales, or different training data.

[0011] In some possible implementations, the number of large language models is multiple; The step of combining the question text with the answer content output by the large language model to obtain the first text includes: If the solutions output by multiple large language models for the same question text meet the preset distribution conditions, then for each large language model, the question text and the solution output by the large language model are combined to obtain the first text. The preset distribution conditions are used to represent the distribution of correct and incorrect categories among several solutions to the same question.

[0012] In some possible implementations, training the text generation model based on the discrimination result includes: The text generation model is trained based on the discrimination results of the text generation model for multiple first texts containing the same question text.

[0013] In some possible implementations, the model parameters adjusted during the training of the text generation model based on the correctness judgment task and the text generation task are not exactly the same.

[0014] In some possible implementations, the text generation model is a model based on a hybrid expert architecture.

[0015] According to a second aspect of the present disclosure, a text generation method is provided, the method comprising: Input the question text into the text generation model; The solution content output by the text generation model is obtained, wherein the text generation model is trained based on the method described in the embodiment of the first aspect above.

[0016] According to a third aspect of the present disclosure, a training apparatus for a text generation model is provided, the apparatus comprising: The first training module is configured to train the text generation model based on a correctness judgment task, wherein the correctness judgment task is used to adjust the model parameters based on the correctness of the text content correctness judgment. The second training module is configured to use the model parameters obtained from the correctness discrimination training as initialization parameters, and continue to train the text generation model based on the text generation task, wherein the text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

[0017] In some possible implementations, the first training module includes: The acquisition unit is configured to acquire a first text, wherein the first text is any text. The generation unit is configured to input the first text into the text generation model and obtain the discrimination result output by the text generation model, wherein the discrimination result is used to indicate whether the content of the first text is correct; The training unit is configured to train the text generation model based on the discrimination result.

[0018] In some possible implementations, the first text includes: a question text and the corresponding answer content; the first text is used to instruct the text generation model to determine whether the answer content is correct based on the question text. The solution content includes at least one of the reasoning text corresponding to the question text and the answer text.

[0019] In some possible implementations, the acquisition unit is configured to input the question text into a large language model to obtain the answer content output by the large language model; and to combine the question text with the answer content output by the large language model to obtain the first text.

[0020] In some possible implementations, there are multiple large language models, which are large language models with different network architectures, different parameter scales, or different training data.

[0021] In some possible implementations, the number of large language models is multiple; The acquisition unit is configured to combine the question text with the answer content output by the large language model for each large language model to obtain the first text if the answer content output by multiple large language models for the same question text meets the preset distribution conditions. The preset distribution conditions are used to represent the distribution of correct and incorrect categories among several solutions to the same question.

[0022] In some possible implementations, the training unit is configured to train the text generation model based on the text generation model's discrimination results for multiple first texts containing the same question text.

[0023] In some possible implementations, the model parameters adjusted during the training of the text generation model based on the correctness judgment task and the text generation task are not exactly the same.

[0024] In some possible implementations, the text generation model is a model based on a hybrid expert architecture.

[0025] According to a fourth aspect of the present disclosure, a text generation apparatus is provided, the apparatus comprising: The generation module is configured to input the question text into the text generation model; and obtain the answer content output by the text generation model, wherein the text generation model is trained based on the method described in the first aspect above.

[0026] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method described in the embodiments of the first or second aspect above.

[0027] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the steps of the methods described in the embodiments of the first or second aspect above.

[0028] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the methods described in the embodiments of the first or second aspect above.

[0029] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: As can be seen from the above embodiments, this disclosure first trains the text generation model for correctness discrimination, enabling the text generation model to identify erroneous content in the text; then, the model parameters obtained from the correctness discrimination training are used as initialization parameters, and the text generation model is further trained based on the text generation task. Since the text generation model has learned to identify various types of erroneous content during the correctness discrimination training process, the text generation model can effectively avoid generating erroneous content and repeatedly correcting it during the text generation training process, reducing meaningless trial-and-error output, thereby reducing the redundancy of the output content while ensuring the accuracy of the output content, and improving the text generation efficiency.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of a training system for a text generation model according to an embodiment of the present disclosure.

[0033] Figure 2 This is a schematic flowchart illustrating a training method for a text generation model according to an embodiment of the present disclosure.

[0034] Figure 3 This is a schematic flowchart illustrating a correctness judgment training method for a text generation model according to an embodiment of the present disclosure.

[0035] Figure 4 This is a schematic flowchart illustrating a training method for a text generation model according to an embodiment of the present disclosure.

[0036] Figure 5 This is a schematic flowchart illustrating a correctness judgment training method for a text generation model according to an embodiment of the present disclosure.

[0037] Figure 6 This is a schematic flowchart illustrating a text generation method according to an embodiment of the present disclosure.

[0038] Figure 7 This is a schematic block diagram illustrating a training apparatus for a text generation model according to an embodiment of the present disclosure.

[0039] Figure 8This is a schematic block diagram illustrating a training apparatus for a text generation model according to an embodiment of the present disclosure.

[0040] Figure 9 This is a schematic block diagram illustrating a text generation apparatus according to embodiments of the present disclosure.

[0041] Figure 10 This is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0042] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0043] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. The singular forms “a” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0044] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0045] For the sake of brevity and ease of understanding, this document uses the terms "greater than" or "less than", "higher than" or "lower than" to describe size relationships. However, it will be understood by those skilled in the art that the term "greater than" also includes the meaning of "greater than or equal to", and "less than" also includes the meaning of "less than or equal to"; the term "higher than" also includes the meaning of "higher than or equal to", and "lower than" also includes the meaning of "lower than or equal to".

[0046] With the continuous development of artificial intelligence technology, text generation models have been widely used in tasks such as question answering, story generation, and dialogue. However, in practical applications, it has been found that their accuracy is low in complex reasoning scenarios.

[0047] Currently, in order to improve the accuracy of text generation models in complex reasoning scenarios, reinforcement learning from verifiable rewards (RLVF) is commonly used to train text generation models. This method guides text generation models to explore reasoning strategies beyond supervised demonstrations by providing sparse but automatically verifiable supervision signals (such as whether the final answer is correct), thereby improving the accuracy of text generation models to some extent.

[0048] However, this training method can also lead text generation models to tend to find the final correct answer by enumerating multiple hypotheses and repeatedly revising intermediate steps, resulting in verbose output and significantly increasing inference latency. Introducing a length penalty heuristic can compress the length of the model's output, but this will also compromise the accuracy of the output.

[0049] In view of this, this disclosure proposes a training method for a text generation model, which can effectively avoid the text generation model generating incorrect content and repeatedly correcting it, reducing meaningless trial and error output, thereby reducing the redundancy of the output content while ensuring the accuracy of the output content, and improving the text generation efficiency.

[0050] In other words, the method provided in this disclosure can reduce the length of the output content without compromising the accuracy of the text generation model's output.

[0051] It should be noted that the text generation model in this embodiment can be a model with an explicit reasoning process or a model without an explicit reasoning process. Having an explicit reasoning process means that the text generation model outputs information such as intermediate reasoning steps, thought processes, or step-by-step derivation paths when generating the final result. Not having an explicit reasoning process means that the text generation model directly outputs the final result.

[0052] Whether the text generation model has an explicit reasoning process or not, the training method provided in this disclosure can effectively prevent the text generation model from outputting content according to incorrect logic, thereby improving the output quality and / or output efficiency of the text generation model.

[0053] The embodiments of this disclosure can be applied to electronic devices, including but not limited to smartphones, desktop computers, tablets, laptops, e-book readers, smartwatches, smart bracelets, and in-vehicle terminals. One or more embodiments of this disclosure are not limited thereto. During operation, the electronic device can run a text generation model training system to train the text generation model. The application program for this training system can be pre-installed on the electronic device, allowing the training system to be started and run on the device. Of course, when using technologies such as HTML5, it is not necessary to install the corresponding application on the electronic device to obtain and run the training system.

[0054] In another embodiment, the training system may include, for example: Figure 1 The diagram shows a server 11, a network 12, and an electronic device 13. During operation, the server 11 can run the server-side program of the training system to perform data processing and other functions. Meanwhile, the electronic device 13 can run the client-side program of the training system to perform data acquisition, human-computer interaction, and other functions. This collaboration between the server 11 and the electronic device 13 enables the training of the text generation model. It should be noted that the server 11 can also communicate with multiple electronic devices to train the text generation model on multiple devices.

[0055] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted by a host cluster. Electronic device 13 can be any type of device, such as a PC, tablet, laptop, PDA (Personal Digital Assistants), mobile phone, wearable device (such as smart glasses, smartwatches, etc.), etc., and this specification does not limit this to any of the embodiments described herein. The network 12 for interaction between electronic device 13 and server 11 can include various types of wired or wireless networks. In one embodiment, network 12 can include the Public Switched Telephone Network (PSTN) and the Internet.

[0056] The following example illustrates the training method of the text generation model provided in this disclosure by executing it on a terminal. Figure 2 This is a schematic flowchart illustrating a training method for a text generation model according to embodiments of the present disclosure. Figure 2 As shown, the method may include the following steps: S201. Based on the correctness discrimination task, the text generation model is trained to perform correctness discrimination. This correctness discrimination task is used to adjust the model parameters based on the correctness of the text content.

[0057] In some embodiments, the text generation model is a model that can generate corresponding text content based on input content. In some embodiments, the text generation model can be a large language model. For example, by inputting prompt words or natural language instructions into the large language model, the large language model can output text content that matches the input content, such as answer text, inference text, etc.

[0058] In some embodiments, the text generation model can be a dialogue model. For example, by inputting the user's current question or the user's historical dialogue context into the dialogue model, the dialogue model can output a response text that fits the current context.

[0059] In some embodiments, the text generation model can be a story generation model. For example, by inputting a story beginning, story outline, or story summary into the story generation model, the model can output the complete story text.

[0060] It should be noted that the embodiments disclosed herein are merely illustrative examples of the types, inputs, and outputs of text generation models and are not intended to limit them. In some embodiments, any model capable of generating corresponding text content based on input content can be called a text generation model. Therefore, a text generation model is not limited to generating plain text content but can also generate multimodal content. For example, this text generation model can generate not only text but also images, audio, video, and other content. The model can selectively output text content, image content, audio content, video content, or multimodal content based on the input content. Alternatively, the text generation model can acquire existing image, audio, video, or other content and insert it into the generated text content to output multimodal content. The embodiments disclosed herein do not limit how the text generation model generates multimodal content.

[0061] In some embodiments, determining the correctness of text content refers to assessing whether the corresponding text content conforms to the relevant norms within its context. In some scenarios, determining the reasonable correctness of text content can be seen as determining whether the text content conforms to facts, scientific principles, etc. For example, if the text generation model is a question-answering model, determining the correctness of text content means determining whether the answer text output by the answering model is correct relative to the input question text. In some scenarios, determining the correctness of text content can be seen as determining whether the logic of the text content is correct, whether it is consistent with the context, etc. For example, if the text generation model is a story generation model, determining the correctness of text content means determining whether the story text output by the story generation model conforms to the story's development logic.

[0062] In some embodiments, determining the correctness of text content may include determining at least one of the following: whether the logic of the text content is correct, whether the text content conforms to the facts, whether the text content conforms to common sense, whether the text content conforms to scientific laws, whether the context of the text content is consistent, etc., and this disclosure does not limit this.

[0063] In some embodiments, since the correctness discrimination task is used to adjust the model parameters based on the correctness of the text content, training the text generation model on the correctness discrimination task enables the text generation model to accurately determine whether the text content is correct, thereby enabling the text generation model to learn to identify erroneous content in the text.

[0064] The reason why some content is wrong is that its logic is flawed. Therefore, when a text generation model learns to identify erroneous content through correctness judgment training, it also learns to identify the erroneous logic that leads to the content error.

[0065] It's important to note that although the function of a text generation model is to generate text, its output is guided by the input. Therefore, by designing appropriate inputs, a correctness judgment task can be transformed into a text generation task. For example, the input could be "Please determine whether the following answers are the correct answers to the following questions: specific questions and specific answers".

[0066] S202. Using the model parameters obtained from the correctness discrimination training as initialization parameters, the text generation model is further trained based on the text generation task. This text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

[0067] Using the model parameters obtained from the correctness discrimination training as initialization parameters means that after completing the correctness discrimination training for text generation, the model parameters of the text generation model at this point are retained and used as the starting parameters for subsequent text generation training. In this way, the knowledge learned by the text generation model in the correctness discrimination training (the identification of incorrect content and flawed logic) can be transferred to the text generation training. This effectively avoids the text generation model outputting content based on incorrect logic, or even repeatedly correcting it, reducing meaningless trial-and-error output. Therefore, while ensuring the accuracy of the output content, the redundancy of the output content is reduced, and the text generation efficiency is improved.

[0068] In this embodiment of the disclosure, when training the text generation model based on the text generation task, any training method can be used.

[0069] In some embodiments, a reinforcement learning method based on verifiable rewards can be used to train the text generation model. For example, if the text generated by the model is correct, a reward is given; if the text generated is incorrect, no reward is given. Based on the obtained rewards, the model parameters are adjusted using a policy gradient algorithm to improve the accuracy of the model's output text.

[0070] It should be noted that, in the embodiments of this disclosure, even when training the text generation model based on a reinforcement learning method with verifiable rewards, the text generation model will not tend to find the final correct answer by enumerating multiple hypotheses and repeatedly correcting intermediate steps. This is because, before training the text generation model based on the text generation task, it is first trained on the correctness discrimination task, enabling the text generation model to learn to identify erroneous content and erroneous logic. Thus, during the text generation training phase, the text generation model can be effectively prevented from outputting content based on erroneous logic, thereby reducing meaningless trial-and-error output. This reduces the redundancy of the output content while ensuring its accuracy, and improves the efficiency of text generation.

[0071] In some possible implementations, the model parameters adjusted during the training of the text generation model based on the correctness judgment task and the text generation task are not entirely the same. Therefore, some of the model parameters adjusted during the correctness judgment training phase are fully retained in the subsequent text generation training phase. This allows the model to still possess a high ability to distinguish between correct and incorrect content while generating text, thereby more effectively avoiding the generation of erroneous content by the text generation model and the need for repeated corrections.

[0072] In some possible implementations, the text generation model is based on a hybrid expert architecture. A hybrid expert architecture-based model determines the model parameters involved in inference computation based on the input content, and calculates the output content based on these parameters. During model training, only the model parameters involved in inference computation are adjusted, while those not involved in inference computation are left unadjusted. Since the input content in the correctness judgment training phase and the text generation training phase are not entirely the same, the model parameters involved in inference computation are also not entirely the same. This naturally leads to the model parameters being adjusted differently during the training of the text generation model based on the correctness judgment task and the text generation task.

[0073] For example, this hybrid expert architecture model includes multiple expert subnetworks, each adept at solving different problems. A gating layer can be placed before these multiple expert subnetworks. The gating layer generates a weight matrix based on the input content. This weight matrix includes the weights corresponding to each expert subnetwork. The N expert subnetworks (N can be any integer) with the largest weights can be identified as target expert subnetworks, or expert subnetworks with weights greater than a preset threshold can be identified as target expert subnetworks. The input content is processed by these target expert subnetworks, and the outputs of the multiple target expert subnetworks are fused. The fused result is then input into a decoder or output layer to output the final result.

[0074] In some possible implementations, after training the text generation model on the correctness discrimination task, at least some of the model parameters adjusted during the training process can be frozen so that the model parameters adjusted in the correctness discrimination training phase are fully preserved in the subsequent text generation training phase.

[0075] As can be seen from the above embodiments, this disclosure first trains the text generation model for correctness discrimination, enabling the text generation model to identify erroneous content in the text; then, the model parameters obtained from the correctness discrimination training are used as initialization parameters, and the text generation model is further trained based on the text generation task. Since the text generation model has learned to identify various types of erroneous content during the correctness discrimination training process, the text generation model can effectively avoid generating erroneous content and repeatedly correcting it during the text generation training process, reducing meaningless trial-and-error output, thereby reducing the redundancy of the output content while ensuring the accuracy of the output content, and improving the text generation efficiency.

[0076] Next, this disclosure will be implemented through... Figure 3 The illustrated embodiment further explains step S201 above. Figure 3 This is a schematic diagram illustrating a correctness judgment training method for a text generation model according to an embodiment of this disclosure. Figure 3 As shown, the method may include the following steps: S301, Obtain the first text.

[0077] The first text can be any text.

[0078] In some possible implementations, the first text is a question-and-answer text, which may include a question text and the corresponding answer content. The answer content may include at least one of the reasoning text and the answer text corresponding to the question text. The reasoning text is used to represent the intermediate logical process of deriving the answer from the question text, and the answer text is used to directly respond to the final conclusion of the question text. Determining whether the first text is correct may involve determining whether the answer content in the first text is the correct answer to the question text in the first text.

[0079] In some possible implementations, the first text is story text. Determining the correctness of the first text can be done by judging whether the story development logic of the story text is correct. In some possible implementations, the first text is dialogue text. Determining the correctness of the first text can be done by judging whether the answers are correct and whether the dialogue is consistent. In some possible implementations, the first text is code text. Determining the correctness of the first text can be done by judging whether the code text is correct, for example, whether it can correctly implement the corresponding function.

[0080] It should be noted that the embodiments disclosed herein are merely illustrative of the first text and do not limit the first text.

[0081] In some possible implementations, the first text may be generated by other models. In some possible implementations, the first text may be manually written. In some possible implementations, the first text may be collected from publicly available data. This disclosure is merely illustrative of the source of the first text and does not limit the source of the first text.

[0082] This disclosure uses only the example of a question-and-answer text as the first text, and the first text being generated by other models, to illustrate the method of obtaining the first text. Obtaining the first text includes: inputting the question text into a large language model to obtain the answer content output by the large language model; and combining the question text with the answer content output by the large language model to obtain the first text.

[0083] In some embodiments, prompt word templates can be designed to clearly define the tasks that the text generation model needs to perform. For example, the prompt word template could be "Please determine whether the following answer is the correct answer to the following question," or "Please determine whether the following answer is the correct answer to the following question and give your reasoning." By guiding the text generation model to output its reasoning within the prompt word template, the model can engage in more complex and detailed thinking when making correctness judgments, resulting in more accurate recognition results. Furthermore, it can guide the text generation model to learn information such as logical errors within the text content.

[0084] The first text may also include a prompt word template. The first text is obtained by combining the question text with the answer output by the large language model, including: combining the question text, the answer output by the large language model, and a preset prompt word template.

[0085] In some embodiments, to obtain a richer first text, there can be multiple large language models, which are different large language models. For example, at least one of the following differences exists between these multiple large language models: (1) Different network architectures.

[0086] For example, autoregressive architectures based on Transformer decoders, hybrid expert architectures, and encoder-decoder architectures that support bidirectional context modeling.

[0087] (2) The parameter scales are different.

[0088] For example, large language models with a parameter count of one billion (1B), ten billion (10B), and hundreds of billions (100B+) can cover different generative capabilities, ranging from lightweight reasoning to complex multi-step deduction.

[0089] (3) The training data are different.

[0090] Of course, the embodiments disclosed herein are merely illustrative examples of the differences between multiple large language models and are not intended to limit them. Of course, the differences between the multiple large language models may also be due to different training methods, or other differences not mentioned in the embodiments disclosed herein.

[0091] Because of at least one of the above differences between multiple large language models, the generated solutions will differ in terms of logical path, reasoning depth, etc., resulting in diverse output content.

[0092] In some possible implementations, to enable the text generation model to learn to identify erroneous content or logic more quickly, correct and incorrect answers can be input into the text generation model for the same question text. In this way, the text generation model can explicitly learn the discrimination boundary of erroneous content by comparing the differences between correct and incorrect answers, thereby improving its sensitivity and accuracy in identifying erroneous content. That is, obtaining the first text includes: obtaining multiple first texts corresponding to the same question text, wherein the multiple first texts include the same question text, and the multiple first texts include different answer contents.

[0093] In some embodiments, the first text is generated based on a large language model. There are multiple large language models; wherein, combining the question text with the answer content output by the large language model to obtain the first text includes: if the answer content input by multiple large language models for the same question text satisfies a preset distribution condition, then for each large language model, the question text is combined with the answer content output by that large language model to obtain the first text. The preset distribution condition is used to represent the distribution of correct and incorrect categories among several answer contents for the same question.

[0094] For example, the preset distribution condition is used to indicate that among several solutions to the same question, there are solutions of the correct category and solutions of the incorrect category. For instance, if multiple large language models output solutions to the same question text that contain both correct and incorrect solutions, then for each large language model, the question text is combined with the solution output by that large language model to obtain the first text.

[0095] For example, the preset distribution condition is used to indicate that among several solutions to the same question, the number of correct solutions and incorrect solutions are the same. For instance, if six large language models output three correct solutions and three incorrect solutions for the same question text, then for each large language model, the question text is combined with the solution output by that large language model to obtain the first text, that is, six first texts are obtained.

[0096] S302. Input the first text into the text generation model and obtain the discrimination result output by the text generation model. The discrimination result is used to indicate whether the content of the first text is correct.

[0097] In some embodiments, the determination result includes a final determination indicating whether the content of the first text is correct. For example, correct or incorrect. Or, a determination flag (such as 0 or 1, where 0 can indicate incorrect and 1 can indicate correct; of course, it can also be 0 indicating correct and 1 indicating incorrect).

[0098] In some embodiments, the judgment result may further include a judgment reason corresponding to the final judgment. The judgment reason may be used to point out logical errors, factual errors, inconsistencies, etc., that exist in the first text.

[0099] In some embodiments, as described in step S301 above, when the first text includes both question text and answer content, the multiple first texts may include the same question text. Therefore, when multiple first texts include the same question text, the multiple first texts can be sequentially input into the text generation model, and the text generation model can sequentially judge and output the judgment results; alternatively, the multiple first texts can be input together into the text generation model, and the text generation model can output the judgment results of the multiple first texts. This disclosure does not limit this aspect.

[0100] S303. Based on the discrimination results, train the text generation model.

[0101] In this embodiment of the disclosure, the text generation model is trained based on the discrimination result, and any training method can be used.

[0102] In some embodiments, the text generation model can be trained based on the difference between the discrimination result and the true discrimination result. In some embodiments, the text generation model can be trained based on a reinforcement learning method with verifiable rewards. For example, if the discrimination result generated by the text generation model is correct, a reward is given; if the discrimination result generated by the text generation model is incorrect, no reward is given. Based on the obtained reward, the model parameters are adjusted through a policy gradient algorithm to improve the accuracy of the discrimination result output by the model.

[0103] In some possible implementations, to enable the text generation model to learn to identify erroneous content or logic more quickly, correct and incorrect answers can be input into the model for the same question text. This allows the text generation model to explicitly learn the discrimination boundary of erroneous content by comparing the differences between correct and incorrect answers, thereby improving its sensitivity and accuracy in identifying erroneous content. Therefore, based on this discrimination result, the text generation model is trained, including: training the text generation model based on its discrimination results for multiple first texts containing the same question text.

[0104] For example, if the judgment result generated by the text generation model is correct, a reward is given; if the judgment result generated by the text generation model is incorrect, no reward is given. Based on the judgment results of the multiple first texts, the total reward is determined. Based on the total reward, the model parameters are adjusted through the policy gradient algorithm to improve the accuracy of the judgment result output by the model.

[0105] Next, this embodiment of the disclosure takes as an example the first text including question text and answer content, and the first text being generated through a large language model, and then... Figure 4 and Figure 5 ,right Figure 3The illustrated embodiments will be further explained. Figure 5 This is a schematic diagram illustrating a correctness judgment training method for a text generation model according to an embodiment of this disclosure. Figure 5 As shown, the method may include the following steps: S501. Input the question text into multiple large language models and obtain the answer content output by each large language model.

[0106] S502. If the answer content output by multiple large language models for the same question text meets the preset distribution conditions, then for each large language model, the question text and the answer content output by the large language model are combined to obtain the first text.

[0107] In some embodiments, when the solution output by the large language model includes both reasoning text and answer text, if the answer text is correct (e.g., the answer in the answer text equals the standard answer), the solution can be considered correct, and a label representing the correct category can be set for the first text composed of the question text and the solution content. If the large language model enumerates multiple hypotheses or repeatedly corrects intermediate steps during the reasoning process, the output of the final correct reasoning process can be controlled, or incorrect reasoning processes can be manually removed. This disclosure does not impose limitations on these embodiments.

[0108] If the answer text is incorrect (e.g., the answer in the answer text is not the same as the standard answer), then the solution content can be considered incorrect, and a label can be set for the first text consisting of the question text and the solution content to indicate the error category.

[0109] S503. Input multiple first texts containing the same question text into the text generation model, and obtain the discrimination results of each first text output by the text generation model.

[0110] S504. Based on the discrimination results of each first text, train the text generation model.

[0111] For example, when the discrimination result of the first text is the same as the label of the first text, a reward can be given to the text generation model; when the discrimination result of the first text is different from the label of the first text, no reward can be given to the text generation model. Based on the discrimination results of multiple first texts, the total reward is determined; based on the total reward, the model parameters are adjusted through a policy gradient algorithm to improve the accuracy of the discrimination result output by the model.

[0112] By training the text generation model with both correct and incorrect answers to the same question, the model can focus more on the logic of the answers and learn to identify erroneous logic.

[0113] like Figure 4 As shown, the embodiments of this disclosure employ a two-stage training method of first determining and then generating to train the text generation model.

[0114] During the discriminator training phase, the model's input can include the question text and the answer to that question. The model's output can include a comment on the answer and a discrimination flag (0 for error, 1 for correct).

[0115] If the answer extracted from the solution content is equal to the standard answer, then label 1 is added to the solution content; if the answer extracted from the solution content is not equal to the standard answer, then label 0 is added to the solution content.

[0116] If the discriminant label output by the model matches the label of the input solution, a positive reward is given (e.g., a reward of 1). If they do not match, no reward is given (e.g., a reward of 0). Based on the reward, the model parameters are adjusted using a policy gradient algorithm, forcing the model to learn to recognize erroneous logic. This discriminant stage can train the text generation model into a discriminator capable of identifying whether the problem-solving steps are correct or incorrect.

[0117] During the generator training phase, the model parameters trained in the discriminator training phase are used as the initialization parameters for the generator training phase. The model's input can be only the question text, and the output can be a complete thought process and the final answer. The model's final answer is compared with the standard answer; if correct, a reward is given (e.g., 1 point) (or 0 points); if incorrect, no reward is given. Based on the reward, the model parameters are adjusted using a policy gradient algorithm. No explicit penalty is needed for the length of the generated content during this phase.

[0118] The text generation model obtained using the above training method has the following functions: (1) Internalized implicit pruning: The model often outputs text such as "Let me try again" for explicit error correction. However, the model trained by the training method provided in this embodiment can suppress erroneous branches in the latent space and directly activate the weights of the correct path when generating each word.

[0119] (2) By using the model parameters obtained from the correctness discrimination training as the initialization parameters of the text generation training stage, the ability to identify errors learned in the correctness discrimination training stage is transferred to the text generation training stage, thus changing the underlying thinking mode of the model.

[0120] Therefore, after two stages of training, the model parameters have internalized the prior knowledge for judging superiority and inferiority. After receiving complex logical questions input by users, the model can output a concise, logically coherent reasoning process and the final answer without redundant trial and error.

[0121] Figure 6 This is a schematic diagram illustrating a text generation method according to an embodiment of the present disclosure. Figure 6 As shown, the method may include the following steps: S601. Input the question text into the text generation model.

[0122] S602. Obtain the solution content output by the text generation model.

[0123] The text generation model is based on Figures 2 to 5 The method provided in any of the embodiments shown is used for training.

[0124] For example, the question text could be: Calculate the integral of a certain function.

[0125] If a text generation model is directly trained using reinforcement learning based on verifiable rewards, the model often first generates a solution, writes out the derivation process, finds an error, outputs "Wrong, let me try another method," and then generates a second solution. This results in the text generation model's output being lengthy and inefficient.

[0126] The text generation model trained using the training method provided in this embodiment has a low score for the first solution path before the first word is generated. The activation of this path is directly suppressed, and the model directly outputs the content according to the second correct solution. The output content is concise and efficient.

[0127] In other words, this disclosure transforms trial and error from the output text to the decision-making within the model, improving the model's reasoning efficiency. Furthermore, without forcibly penalizing the length of the model's output content, the text generation model naturally generates shorter and more accurate answers, solving the problem of the model becoming more verbose as it is trained in reinforcement learning.

[0128] It should be noted that, through experiments, it was found that, based on the same training dataset, compared with directly training the model based on reinforcement learning, the training method provided in this disclosure embodiment reduces the average output length on in-domain math tasks by approximately 42%.

[0129] Furthermore, the training method provided in this embodiment of the present disclosure also improves model performance, with an average accuracy improvement of approximately 3.7% on in-domain mathematical tasks.

[0130] Furthermore, the training method provided in this embodiment changes the model's thought process, significantly reducing the number and frequency of transition words (such as "but," "however," "wait," etc.) in the model's output. This demonstrates that the model has shifted from explicit trial and error to intrinsic decision-making, making the reasoning process more direct and coherent.

[0131] Furthermore, the training method provided by the embodiments of this disclosure can also improve the generalization ability of the model, and the trained model also has superior performance in extra-domain tasks such as scientific reasoning and code writing.

[0132] Corresponding to the aforementioned training method and embodiments of the text generation method for the text generation model, this disclosure also provides embodiments of a training device for the text generation model and a text generation device.

[0133] Please see Figure 7 , Figure 7 This is a schematic block diagram of a training apparatus for a text generation model, provided in an exemplary embodiment. The apparatus may include a first training module 701 and a second training module 702. The first training module 701 is configured to perform correctness discrimination training on the text generation model based on a correctness discrimination task, wherein the correctness discrimination task is used to adjust the model parameters based on the correctness of the text content correctness discrimination. The second training module 702 is configured to use the model parameters obtained from the correctness discrimination training as initialization parameters, and continue to train the text generation model based on the text generation task, wherein the text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

[0134] like Figure 8 As shown, in some possible implementations, the first training module 701 includes: The acquisition unit 7011 is configured to acquire a first text, wherein the first text is any text. The generation unit 7012 is configured to input the first text into the text generation model and obtain the discrimination result output by the text generation model, wherein the discrimination result is used to indicate whether the content of the first text is correct; Training unit 7013 is configured to train the text generation model based on the discrimination result.

[0135] In some possible implementations, the first text includes: a question text and the corresponding answer content; the first text is used to instruct the text generation model to determine whether the answer content is correct based on the question text. The solution content includes at least one of the reasoning text corresponding to the question text and the answer text.

[0136] In some possible implementations, the acquisition unit 7011 is configured to input the question text into a large language model to obtain the answer content output by the large language model; and to combine the question text with the answer content output by the large language model to obtain the first text.

[0137] In some possible implementations, there are multiple large language models, which are large language models with different network architectures, different parameter scales, or different training data.

[0138] In some possible implementations, the number of large language models is multiple; The acquisition unit 7011 is configured to, if the answer content output by multiple large language models for the same question text meets the preset distribution conditions, then for each large language model, combine the question text with the answer content output by the large language model to obtain the first text. The preset distribution conditions are used to represent the distribution of correct and incorrect categories among several solutions to the same question.

[0139] In some possible implementations, the training unit 7013 is configured to train the text generation model based on the text generation model's discrimination results for multiple first texts containing the same question text.

[0140] In some possible implementations, the model parameters adjusted during the training of the text generation model based on the correctness judgment task and the text generation task are not exactly the same.

[0141] In some possible implementations, the text generation model is a model based on a hybrid expert architecture.

[0142] Please see Figure 9 , Figure 9 This is a schematic block diagram of a text generation apparatus provided in an exemplary embodiment. The apparatus may include: a generation module 901. The generation module 901 is configured to input the question text into the text generation model and obtain the answer content output by the text generation model, wherein the text generation model is trained based on the training method of the above-mentioned text generation model.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant methods, and will not be elaborated upon here.

[0144] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0145] Figure 10 This is a schematic block diagram illustrating an electronic device according to embodiments of the present disclosure. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0146] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0147] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0148] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0149] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0150] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0151] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0152] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0153] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0154] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR, or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0155] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described in any of the above embodiments.

[0156] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0157] In an exemplary embodiment, this disclosure also provides a computer program product including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0158] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0159] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a text generation model, characterized in that, The method includes: Based on the correctness judgment task, the text generation model is trained to judge correctness. The correctness judgment task is used to adjust the model parameters based on the correctness of the text content correctness judgment. The model parameters obtained from the correctness discrimination training are used as initialization parameters. Based on the text generation task, the text generation model is further trained. The text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

2. The method according to claim 1, characterized in that, The correctness judgment task-based training of the text generation model includes: Get the first text, where the first text is any text; The first text is input into the text generation model to obtain the discrimination result output by the text generation model. The discrimination result is used to indicate whether the content of the first text is correct. Based on the discrimination results, the text generation model is trained.

3. The method according to claim 2, characterized in that, The first text includes: a question text and the corresponding answer content; the first text is used to instruct the text generation model to determine whether the answer content is correct based on the question text. The solution content includes at least one of the reasoning text corresponding to the question text and the answer text.

4. The method according to claim 3, characterized in that, The acquisition of the first text includes: Input the question text into the large language model to obtain the answer content output by the large language model; The question text is combined with the answer output by the large language model to obtain the first text.

5. The method according to claim 4, characterized in that, The number of large language models is multiple; There are at least one of the following differences among multiple large language models: different network architectures, different parameter sizes, and different training data.

6. The method according to claim 4, characterized in that, The number of large language models is multiple; The step of combining the question text with the answer content output by the large language model to obtain the first text includes: If the solutions output by multiple large language models for the same question text meet the preset distribution conditions, then for each large language model, the question text and the solution output by the large language model are combined to obtain the first text. The preset distribution conditions are used to represent the distribution of correct and incorrect categories among several solutions to the same question.

7. The method according to claim 6, characterized in that, The step of training the text generation model based on the discrimination result includes: The text generation model is trained based on the discrimination results of the text generation model for multiple first texts containing the same question text.

8. The method according to claim 1, characterized in that, The model parameters adjusted during the training of text generation models based on correctness judgment tasks and text generation tasks are not entirely the same.

9. The method according to claim 1, characterized in that, The text generation model is based on a hybrid expert architecture.

10. A text generation method, characterized in that, The method includes: Input the question text into the text generation model; The solution content output by the text generation model is obtained, wherein the text generation model is trained based on the method described in any one of claims 1 to 9.

11. A training device for a text generation model, characterized in that, The device includes: The first training module is configured to train the text generation model based on a correctness judgment task, wherein the correctness judgment task is used to adjust the model parameters based on the correctness of the text content correctness judgment. The second training module is configured to use the model parameters obtained from the correctness discrimination training as initialization parameters, and continue to train the text generation model based on the text generation task, wherein the text generation task is used to adjust the model parameters based on the correctness of the text generated by the text generation model.

12. The apparatus according to claim 11, characterized in that, The model parameters adjusted during the training of text generation models based on correctness judgment tasks and text generation tasks are not entirely the same.

13. A text generation device, characterized in that, The device includes: The generation module is configured to input the question text into the text generation model; and obtain the answer content output by the text generation model, wherein the text generation model is trained based on the method described in any one of claims 1 to 9.

14. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1 to 10 by executing the executable instructions.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 10.

16. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 10.

Images