Method and device for training large language model

Abstract

The embodiment of the invention provides a method and device for training a large language model. The generated data of the first large language model comprises answers for the input questions or query statements for the input questions. The method comprises the steps that a first training sample and a second training sample are obtained, a sample label in the first training sample comprises an answer for an input question in the first training sample, and a sample label in the second training sample comprises a query statement for an input question in the second training sample; and performing supervised training on the first large language model based on the first training sample and the second training sample. And stable training can be ensured.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computers, and in particular, to a method and apparatus for training a large language model. BACKGROUND

[0002] Large language models (LLMs) have shown remarkable capabilities in a wide range of tasks, including mathematical reasoning, question answering, and code generation. Because their capabilities are limited to the data used in training, which is statically encoded in the model, large language models often generate inaccurate or fabricated responses when confronted with complex or real-time problems. By enabling large language models to access external knowledge, more accurate and grounded responses can be produced.

[0003] In the prior art, enabling large language models to interact with search engines in an interleaved manner combined with their own reasoning improves the reasoning capabilities of large language models. Despite the significant progress made, there are still difficulties in stable training. SUMMARY

[0004] One or more embodiments of the present specification describe a method and apparatus for training a large language model, which can ensure stable training.

[0005] In a first aspect, a method for training a large language model is provided. The generated data of a first large language model includes an answer to an input question or a query statement for an input question. The method comprises:

[0006] obtaining a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for an input question in the second training sample;

[0007] performing supervised training on the first large language model based on the first training sample and the second training sample.

[0008] In a possible implementation, the method further comprises:

[0009] obtaining a third training sample, wherein the third training sample includes a sample question and a label answer;

[0010] obtaining a predicted answer using the first large language model based on the sample question included in the third training sample, and performing reinforcement learning training on the first large language model after the supervised training based on the predicted answer and the label answer.

[0011] In one possible implementation, the first large language model obtains an answer based on n rounds of reasoning, where the first n-1 rounds of reasoning include generating a query statement, calling a search engine based on the query statement to obtain retrieval information, and performing the next round of reasoning based on the retrieval information.

[0012] Furthermore, the first training sample or the second training sample includes a sample input to be input into the first large language model, and the sample input includes:

[0013] A first sample input consisting of instructions and questions; or,

[0014] The second sample input consists of instructions, questions, reasoning statements, query statements and retrieval information.

[0015] Furthermore, the first training sample or the second training sample includes at least one of the following marking symbol combinations:

[0016] A first marking symbol combination is used to mark the text segment type in the combination as an inference statement;

[0017] A second marking symbol combination is used to mark the text segment in the combination as a query statement;

[0018] A third marking symbol combination is used to mark the text segment type in the combination as search information;

[0019] The fourth marking symbol combination is used to mark the text segment type in the combination as an answer.

[0020] Furthermore, the first training samples include at least one of the following types of samples:

[0021] For the first type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination;

[0022] The second type of samples, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination.

[0023] Furthermore, the second training samples include at least one of the following types of samples:

[0024] The third type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first tag symbol combination and the query statement in the second tag symbol combination;

[0025] The fourth type of sample, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination.

[0026] Furthermore, the first training sample or the second training sample is obtained by:

[0027] Inputting the instruction and question into a second language model to obtain output text for n rounds of reasoning; the second language model has more parameters than the first language model, and the generated data of any round of reasoning in the n rounds of reasoning includes an answer to the input question or a query statement for the input question;

[0028] Determine whether the answer contained in the output text is correct;

[0029] If the answer is determined to be correct, a first training sample or a second training sample is constructed based on the instruction, question, and output text of the target round reasoning.

[0030] Furthermore, constructing a first training sample based on the output text of the instruction, question, and target round reasoning includes:

[0031] The sample input is composed of instructions and questions, and the sample label is composed of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination included in the output text of the last round, to construct a first training sample; or,

[0032] The sample input is composed of the instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of any round of the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination, which are included in the output text of the last round, to construct the first training sample.

[0033] Furthermore, constructing a second training sample based on the output text of the instruction, question, and target round reasoning includes:

[0034] The second training sample is constructed by forming a sample input consisting of an instruction and a question, and forming a sample label consisting of an inference statement in a first tag symbol combination and a query statement in a second tag symbol combination included in the output text of any round in the first n-1 rounds; or,

[0035] The sample input is composed of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of the first round in the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination, which are included in the output text of the second round in the first n-1 rounds after the first round, to construct the second training sample.

[0036] Furthermore, the instructions include:

[0037] Each time new information is obtained, the generated reasoning statement is placed within the first marking symbol combination;

[0038] When external retrieval is required, the generated query statement is placed in the second mark symbol combination;

[0039] Extract query statements and request external search engines to retrieve relevant documents;

[0040] Append the search information to the generated word sequence and place it in the third token combination as additional context for the next step of generation;

[0041] The entire reasoning process is iterated until the maximum number of retrievals is reached or the model generates a final answer, and the answer is placed in the fourth token combination.

[0042] Furthermore, the third training sample is obtained by:

[0043] Inputting the instruction and question into a second language model to obtain output text; the second language model has more parameters than the first language model, and the generated data includes an answer to the input question or a query statement for the input question;

[0044] Determine whether the answer contained in the output text is correct;

[0045] If the answer is determined to be incorrect, sampling is enabled for the second largest language model. For the question, if the second largest language model is correct at least once out of a preset number of answers, the question and the correct answer are retained to obtain a third training sample.

[0046] Furthermore, the objective function of the reinforcement learning training is determined based on a policy model and a reference model, and both the policy model and the reference model are initialized with a large language model trained by supervised learning.

[0047] Furthermore, the predicted answer belongs to the output text of the first language model, and the output text also includes search information obtained by calling a search engine; and the reinforcement learning training includes:

[0048] performing masking processing on the output text obtained by the first language model to shield the search information included in the output text;

[0049] Calculate the loss for the masked output text.

[0050] Furthermore, the reinforcement learning training includes:

[0051] Based on the difference between the predicted answer and the labeled answer, a reward score is determined.

[0052] In a second aspect, a device for training a large language model is provided, wherein the generated data of the first large language model includes an answer to an input question or a query statement for the input question, and the device includes:

[0053] an acquiring unit, configured to acquire a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample;

[0054] A training unit is configured to perform supervised training on the first large language model based on the first training sample and the second training sample acquired by the acquisition unit.

[0055] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.

[0056] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0057] Through the methods and devices provided in the embodiments of this specification, a novel training framework is adopted that enables a large language model to adaptively utilize internal and external knowledge during the reasoning process. Specifically, the training framework adopts a supervised fine-tuning training method, in which a first training sample and a second training sample are used. The sample labels in the first training sample include the answer to the input question in the first training sample, and the sample labels in the second training sample include the query statement for the input question in the second training sample. This training method combines reasoning and retrieval, enabling the large language model to output specific types of text segments during reasoning and response, thereby performing format learning. The output format meets the requirements, thereby ensuring stable training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0060] Figure 2 A flow chart of a method for training a large language model according to one embodiment is shown;

[0061] Figure 3 A schematic diagram showing categories of a first training sample and a second training sample according to one embodiment is shown;

[0062] Figure 4 A flow chart of a method for training a large language model according to another embodiment is shown;

[0063] Figure 5 A schematic diagram of reinforcement learning according to one embodiment is shown;

[0064] Figure 6 A schematic block diagram illustrating an apparatus for training a large language model according to one embodiment is shown;

[0065] Figure 7 A schematic block diagram of an apparatus for training a large language model according to another embodiment is shown. DETAILED DESCRIPTION

[0066] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0067] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves training a large language model, where a large language model refers to a machine learning model with large-scale parameters and complex computing structures. These models are usually built from deep neural networks, have billions or even hundreds of billions of parameters, and can handle more complex tasks and data. Large language models are widely used in various fields, including natural language processing, computer vision, etc. In the embodiments of this specification, further training the large language model on the basis of pre-training can enhance its ability to handle complex tasks.

[0068] The process of generating output text by a large language model according to input text is generally as follows: first, the input text is converted into a form that can be processed by the large language model, that is, into a sequence of tokens; then reasoning is performed, that is, the input token sequence converted from the input text is input into the large language model, the input token sequence is encoded and represented, and the token sequence is predicted and generated based on the encoded representation; finally, the generated token sequence is converted into text form to obtain the output text. In the field of natural language processing, a token refers to the smallest semantic unit in text, which can be a word, a subword, a character, etc. The large language model usually divides the input text into several tokens for processing and generating output.

[0069] With reference to Figure 1 In the embodiments of the present specification, the retrieval augmented generation (RAG) method is adopted to assist text generation by retrieving relevant knowledge and reduce the fact error rate of the large language model in knowledge-intensive tasks. Retrieval augmented generation expands the capability boundary of the model by providing external knowledge in the generation process. Unlike the usual processing method, instead of retrieving and integrating documents related to the query into the prompt to guide the model's answer, the large language model is enabled to interact with the search engine in an interleaved manner, combined with its own reasoning, so that the model can perform step-by-step retrieval, which is beneficial to solve multi-hop problems with complex processes. The large language model obtains an answer based on n rounds of reasoning, wherein the first n-1 rounds of reasoning include generating a query sentence, calling a search engine based on the query sentence to obtain retrieval information, and performing the next round of reasoning based on the retrieval information. It can be understood that each round of reasoning usually includes a reasoning sentence and other types of output text segments generated as needed, for example, the other types of output text segments include query sentences or answers.

[0070] For example, Figure 1 In the embodiments, the instructions and the question are input into the large language model, the large language model first generates a reasoning sentence with a marker symbol, then generates a query sentence with a marker symbol, and then extracts the query sentence and requests an external search engine to retrieve relevant documents. The retrieved retrieval information is attached in the generated token sequence as additional context for text generation in the next round of reasoning. This process is iterated until the maximum number of retrievals is reached or the model generates a final answer. In the generated text, there are several marker symbol combinations, which are used to mark the type of text segment. The marker symbol combination <think> and< / think> is used to mark the type of text segment in the combination as a reasoning sentence; the marker symbol combination <search> and< / search> is used to mark the type of text segment in the combination as a query sentence; and the marker symbol combination <info> and< / info> is used to mark the type of text segment in the combination as retrieval information.<answer> and< / answer> , used to mark the text segment type in the combination as the answer. The first n-1 rounds of reasoning can include <think> Inference Statement< / think> , <search> Search query< / search> , <info> Retrieve Information< / info> ; The final round of reasoning can include <think> Inference Statement< / think> , <answer> Answer< / answer> It is understandable that the value of n is adapted to the specific problem. When the problem is more complex, the value of n is usually larger, for example, n is 3; when the problem is relatively simple, the value of n is usually smaller, for example, n is 1.

[0071] It should be noted that a marker symbol can be a word unit generated by a large language model, and a sentence can include several word units, wherein a query statement can be a short sentence, for example, including only one search term, or a query statement can be a long sentence, for example, including multiple search terms, or, in addition to the search term, also including its attributives, auxiliary words, etc.

[0072] Figure 2 A flowchart of a method for training a large language model according to an embodiment is shown. The generated data of the first large language model includes the answer to the input question or the query statement for the input question. The method can be based on Figure 1 The implementation scenario shown in Figure 2 uses supervised training. Figure 2 As shown, the method for training a large language model in this embodiment includes the following steps: Step 21, obtaining a first training sample and a second training sample, wherein the sample labels in the first training sample include the answer to the input question in the first training sample, and the sample labels in the second training sample include the query statement for the input question in the second training sample; Step 22, performing supervised training on the first large language model based on the first training sample and the second training sample. The specific execution method of each of the above steps is described below.

[0073] First, in step 21, a first training sample and a second training sample are obtained. The sample label of the first training sample includes the answer to the input question in the first training sample, and the sample label of the second training sample includes the query statement for the input question in the second training sample. It is understandable that the first training sample and the second training sample have different sample labels.

[0074] In one example, the first large language model obtains an answer based on n rounds of reasoning. The first n-1 rounds of reasoning include generating a query statement, calling a search engine based on the query statement to obtain retrieval information, and performing the next round of reasoning based on the retrieval information.

[0075] This example shows a typical application scenario of a large language model, namely a multi-round reasoning scenario. It is understandable that when the question is relatively simple, the large language model can also get the answer after only one round of reasoning.

[0076] In the embodiments of this specification, the output text includes the text generated by each round of reasoning. The text generated by each round of reasoning can be further divided into multiple text segments based on its specific type. For example, several tokens belonging to the reasoning statement can be divided into one text segment, several tokens belonging to the query statement can be divided into another text segment, and several tokens belonging to the answer can be divided into another text segment. In addition, although the search information is obtained by calling the search engine, it can also be added as a text segment to the token sequence generated by the large language model, providing additional context for subsequent text generation.

[0077] In one example, the first training sample or the second training sample includes a sample input to be input into the first large language model, and the sample input includes:

[0078] A first sample input consisting of instructions and questions; or,

[0079] The second sample input consists of instructions, questions, reasoning statements, query statements and retrieval information.

[0080] In this example, for the first sample input, the training model uses internal and external knowledge to respond, and for the second sample input, the training model mainly relies on external knowledge when reasoning and generating text.

[0081] In one example, the first training sample or the second training sample includes at least one of the following marker symbol combinations:

[0082] A first marking symbol combination is used to mark the text segment type in the combination as an inference statement;

[0083] A second marking symbol combination is used to mark the text segment in the combination as a query statement;

[0084] A third marking symbol combination is used to mark the text segment type in the combination as search information;

[0085] The fourth marking symbol combination is used to mark the text segment type in the combination as an answer.

[0086] In this example, the training samples are labeled with four types of text segments using four combinations of marker symbols, enabling the large language model to adaptively utilize internal and external knowledge during reasoning and respond in a specific format.

[0087] In one example, the first training sample includes at least one of the following types of samples:

[0088] For the first type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination;

[0089] The second type of samples, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination.

[0090] In this example, the first and second category samples have sample labels of the same composition and sample inputs of different compositions. The first category samples are used to train the model to respond internally, and the second category samples are used to train the model to be mainly based on external knowledge when reasoning and generating text.

[0091] In one example, the second training samples include at least one of the following types of samples:

[0092] The third type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first tag symbol combination and the query statement in the second tag symbol combination;

[0093] The fourth type of sample, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination.

[0094] In this example, the third and fourth category samples have sample labels of the same composition but sample inputs of different compositions. The third category samples are used to train the model to respond using external knowledge, and the fourth category samples are used to train the model to be mainly based on external knowledge when reasoning and generating text.

[0095] Figure 3 FIG. 1 shows a schematic diagram of the categories of the first training sample and the second training sample according to an embodiment. Figure 3 , which shows four types of samples, which combine reasoning and retrieval. The four types of samples have two sample inputs and two sample labels. For the sake of simplicity, only the combination of marker symbols is shown in the figure, and the statements in the combination are not shown. It can be understood that in the combination of marker symbols <think> and< / think> There are also inference statements in the mark symbol combination <search> and< / search> There are also query statements in the markup symbol combination <info> and< / info> There is also retrieval information in the mark symbol combination <answer> and< / answer>The answer still exists within. It can be understood that the first type of samples can train the model to predict reasoning statements and answers based on instructions and questions; the third type of samples can train the model to predict reasoning statements and query statements based on instructions and questions; the second type of samples can train the model to predict reasoning statements and answers for subsequent reasoning rounds based on instructions, questions, reasoning statements, query statements, and retrieval information from a certain reasoning round; and the fourth type of samples can train the model to predict reasoning statements and query statements for subsequent reasoning rounds based on instructions, questions, reasoning statements, query statements, and retrieval information from a certain reasoning round. It should be noted that sample labels do not contain retrieval information, thereby preventing the model from generating retrieval information and helping to prevent the model from generating hallucinations.

[0096] In one example, the first training sample or the second training sample is obtained in the following manner:

[0097] Inputting the instruction and question into a second language model to obtain output text for n rounds of reasoning; the second language model has more parameters than the first language model, and the generated data of any round of reasoning in the n rounds of reasoning includes an answer to the input question or a query statement for the input question;

[0098] Determine whether the answer contained in the output text is correct;

[0099] If the answer is determined to be correct, a first training sample or a second training sample is constructed based on the output text of the instruction, question, and target round reasoning.

[0100] In this example, the output text containing the correct answer is selected to construct the first training sample or the second training sample. The output text can be segmented according to specific segmentation points and divided into multiple sample labels to construct multiple first training samples or second training samples. It is understood that the judgment result can be obtained based on human instructions or by comparing the answer in the output text with a pre-set standard answer.

[0101] Furthermore, constructing a first training sample based on the output text of the instruction, question, and target round reasoning includes:

[0102] The sample input is composed of instructions and questions, and the sample label is composed of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination included in the output text of the last round, to construct a first training sample; or,

[0103] The first reasoning statement in the first token symbol combination, the query statement in the second token symbol combination, and the retrieval information in the third token symbol combination included in the output text of any one of the first n-1 rounds of instructions, questions, and the first token symbol combination included in the output text of the last round of instructions, questions, and the second reasoning statement in the first token symbol combination and the answer in the fourth token symbol combination are used to construct a first training sample.

[0104] For example, the instruction and the question are input into the second large language model, and the output text obtained includes a plurality of output text segments, which are <think> Inference statement 1< / think> 、 <search> Query statement 1< / search> 、 <info> Retrieve information 1< / info> 、 <think> Inference statement 2< / think> 、 <search> Query statement 2< / search> 、 <info> Retrieve Information 2< / info> 、 <think> Reasoning statement 3< / think> 、 <answer> Answer< / answer> The entire reasoning process includes 3 rounds of reasoning, and the first round of reasoning generates <think> Inference statement 1< / think> 、 <search> Query statement 1< / search> 、 <info> Retrieve information 1< / info> The second round of reasoning generates <think> Inference statement 2< / think> 、 <search> Query statement 2< / search> 、 <info> Retrieve Information 2< / info> The third round of reasoning generates <think> Reasoning statement 3< / think> 、 <answer> Answer< / answer> .

[0105] The instruction and the question are input as sample input, and the generated <think> Reasoning statement 3< / think> 、 <answer> Answer< / answer> in the third round of reasoning are used as sample labels, and a first training sample can be constructed; the instruction, the question, and the generated <think> Inference statement 1< / think> 、 <search> Query statement 1< / search> 、 <info> Retrieve information 1< / info> in the first round of reasoning are used as sample input, and the generated <think> Reasoning statement 3< / think> 、 <answer> Answer< / answer> in the third round of reasoning are used as sample labels, and another first training sample can be constructed. It can be understood that one output text can construct multiple first training samples.

[0106] Further, the second training sample is constructed according to the instruction, the question, and the output text of the target round of reasoning, comprising:

[0107] The instruction and the question are used as sample input, and the reasoning statement in the first token symbol combination and the query statement in the second token symbol combination included in the output text of any one of the first n-1 rounds are used as sample labels to construct a second training sample; or

[0108] The sample input is composed of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of the first round in the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination, which are included in the output text of the second round in the first n-1 rounds after the first round, to construct the second training sample.

[0109] For example, following the previous example, we input the instruction and question as samples and <think> Inference statement 1< / think> 、 <search> Query statement 1< / search> As a sample label, a second training sample can be constructed; the instructions and questions are input as samples, and the <think> Inference statement 2< / think> 、 <search> Query statement 2< / search> As sample labels, another second training sample can be constructed; instructions, questions, and the first round of reasoning generated <think> Inference statement 1< / think> 、 <search> Query statement 1< / search> 、 <info> Retrieve information 1< / info> As sample input, the generated <think> Inference statement 2< / think> 、 <search> Query statement 2< / search> As a sample label, another second training sample can be constructed. It can be understood that one output text can construct multiple second training samples.

[0110] Furthermore, the instruction includes:

[0111] Each time new information is obtained, the generated reasoning statement is placed within the first marking symbol combination;

[0112] When external retrieval is required, the generated query statement is placed in the second mark symbol combination;

[0113] Extract query statements and request external search engines to retrieve relevant documents;

[0114] Append the search information to the generated word sequence and place it in the third token combination as additional context for the next step of generation;

[0115] The entire reasoning process is iterated until the maximum number of retrievals is reached or the model generates a final answer, and the answer is placed in the fourth token combination.

[0116] In this example, instructions are used to guide the reasoning process of the large language model, so that it generates a response in a specific format. The specific form of the instructions can be diverse and is not limited here.

[0117] Then, in step 22, supervised training is performed on the first large language model based on the first training samples and the second training samples. It will be appreciated that after obtaining a certain number of first training samples and second training samples, supervised fine-tuning can be performed using these samples to guide the model to generate responses in a specific format and enable it to adaptively utilize internal and external knowledge.

[0118] Among them, supervised fine-tuning (SFT) refers to further training based on the pre-trained model using labeled data to improve the performance of the model or adapt it to specific tasks.

[0119] In the embodiments of this specification, the difference between the predicted output of a sample and the sample label can be measured based on a pre-set loss function, and the model parameters can be adjusted with the goal of reducing the total prediction loss.

[0120] Through the methods and devices provided in the embodiments of this specification, a novel training framework is adopted that enables a large language model to adaptively utilize internal and external knowledge during the reasoning process. Specifically, the training framework adopts a supervised fine-tuning training method, in which a first training sample and a second training sample are used. The sample labels in the first training sample include the answer to the input question in the first training sample, and the sample labels in the second training sample include the query statement for the input question in the second training sample. This training method combines reasoning and retrieval, enabling the large language model to output specific types of text segments during reasoning and response, thereby performing format learning. The output format meets the requirements, thereby ensuring stable training.

[0121] In addition, the embodiments of this specification also provide a two-stage training method, which further performs reinforcement learning training on the large language model based on the aforementioned supervised training.

[0122] Figure 4 A flowchart of a method for training a large language model according to another embodiment is shown. The generated data of the first large language model includes the answer to the input question or the query statement for the input question. The method can be based on Figure 1 The implementation scenario shown in Figure 2 uses a two-stage training approach, including supervised training and reinforcement learning training. Figure 4As shown, the method for training a large language model in this embodiment includes the following steps: Step 41, obtaining a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample; Step 42, based on the first training sample and the second training sample, supervised training is performed on the first large language model; Step 43, obtaining a third training sample, wherein the third training sample includes a sample question and a labeled answer; Step 44, based on the sample question included in the third training sample, using the first large language model to obtain a predicted answer, and based on the predicted answer and the labeled answer, reinforcement learning training is performed on the first large language model after the supervised training. The specific execution method of each of the above steps is described below.

[0123] First, in step 41, a first training sample and a second training sample are obtained, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample.

[0124] Then, in step 42, supervised training is performed on the first large language model based on the first training sample and the second training sample;

[0125] Among them, step 41 is the same as the aforementioned step 21, and step 42 is the same as the aforementioned step 22, which will not be repeated here.

[0126] Next, in step 43, a third training sample is obtained, wherein the third training sample includes a sample question and a labeled answer. It is understandable that obtaining the third training sample includes screening the sample questions.

[0127] In one example, the third training sample is obtained in the following manner:

[0128] Inputting the instruction and question into a second language model to obtain output text; the second language model has more parameters than the first language model, and the generated data includes an answer to the input question or a query statement for the input question;

[0129] Determine whether the answer contained in the output text is correct;

[0130] If the answer is determined to be incorrect, sampling is enabled for the second largest language model. For the question, if the second largest language model is correct at least once out of a preset number of answers, the question and the correct answer are retained to obtain a third training sample.

[0131] In this example, questions that generate incorrect answers are selected. These questions are relatively challenging for the large language model. Subsequently, questions that may not be answered correctly due to missing information retrieval or insufficient model capabilities are further eliminated, and sampling is enabled for the large language model. For example, the temperature is set to 1.2 and the maximum number of retrievals is set to 10. For each question, a maximum of 10 attempts are made. Only questions and correct answers that provide the correct answer in at least one attempt are retained to construct challenging but answerable samples.

[0132] In addition, a certain proportion of samples that generate correct answers can be randomly selected to form a training data set consisting of the final third training samples.

[0133] Finally, in step 44, the first large language model is used to obtain predicted answers based on the sample questions included in the third training sample. Reinforcement learning training is then performed on the supervised first large language model based on the predicted answers and the labeled answers. It will be appreciated that the second stage of reinforcement learning training is based on the supervised first large language model.

[0134] Among them, reinforcement learning (RL) is a machine learning method that focuses on training models to learn how to make decisions by interacting with the environment. In RL, the agent affects the environment by taking actions and adjusts its strategy based on the feedback (rewards or penalties) of its actions to maximize the cumulative reward.

[0135] In the embodiments of this specification, reinforcement learning is applied to the interaction between a large language model and a search engine, and the search engine is modeled as part of the environment to enhance retrieval-based decision-making capabilities.

[0136] Figure 5 FIG. 1 shows a schematic diagram of reinforcement learning according to an embodiment. Figure 5 , the question is input into the large language model, and the large language model obtains the answer through multiple rounds of reasoning. During the multiple rounds of reasoning, the search engine is called to obtain retrieval information, and the retrieval information is applied as additional context in the subsequent reasoning process. The final answer may be correct or wrong. When the answer is wrong, it is beneficial to adjust the parameters of the large language model through reinforcement learning to improve its ability to solve tasks.

[0137] In one example, the objective function of the reinforcement learning training is determined based on a policy model and a reference model, and both the policy model and the reference model are initialized with a large language model trained by supervised learning.

[0138] In this example, both the policy model and the reference model are initialized with a large language model trained using supervised learning. Since the model is relatively stable after supervised learning training, stable training is possible, and the output format meets the requirements.

[0139] In the embodiments of this specification, reinforcement learning is extended to utilize an external search engine for retrieval. The RL objective function using the external retrieval system R can be expressed as follows:

[0140]

[0141] Among them, π θ and π ref Represent the strategy model and reference model respectively, both of which are initialized from the model after SFT, r φ is the reward function, is the KL divergence. x represents the sample input extracted from the dataset D, and y represents the sample input extracted from the policy model π θ The output after sampling from the retrieval system R and retrieving it. Among them, the proximal policy optimization (PPO) algorithm is used to optimize the policy model, which is widely used in reinforcement learning due to its efficiency and reliability.

[0142] In one example, the predicted answer is output text of the first language model, and the output text also includes search information obtained by calling a search engine; and the reinforcement learning training includes:

[0143] performing masking processing on the output text obtained by the first language model to shield the search information included in the output text;

[0144] Calculate the loss for the masked output text.

[0145] In this embodiment, the episode sequence includes both tokens generated by the LLM and tokens retrieved from an external retrieval system. To reduce the influence of external information on the model, a mask loss is introduced for the retrieval tokens. This ensures that the policy gradient objective is only calculated for the LLM-generated tokens, excluding the retrieval content from the optimization process. This stabilizes the training process while maintaining the flexibility of retrieval enhancement generation.

[0146] In one example, the reinforcement learning training includes:

[0147] Based on the difference between the predicted answer and the labeled answer, a reward score is determined.

[0148] In this example, a simple outcome-based reward score is used to avoid the complexity of process-based rewards.

[0149] In the examples of this specification, the reward function serves as the primary training signal, determining the optimization direction of RL. A rule-based reward system, including a final answer reward, is used to assess whether a response is correct. For example, in factual reasoning tasks, rule-based criteria such as exact string matching are used for evaluation:

[0150] r φ (x,y)=EM(a pre , a gold );

[0151] where a pre is the final answer extracted from the response y, a gold is the labeled answer. Because the SFT model has already learned to respond in a specific format, no format reward was added. Furthermore, no neural reward model was used to avoid reward speculation and additional computational cost. By extending RL and retrieval and providing the model with explicit rewards, the model was trained to enhance its reasoning ability and correctly answer questions.

[0152] The experimental results of the embodiments of this specification show that the model can effectively utilize internal and external knowledge in the reasoning process.

[0153] Through the method provided in the embodiments of this specification, a novel training framework is adopted, which enables the large language model to adaptively utilize internal and external knowledge during the reasoning process. Specifically, the training framework includes two stages, the first stage adopts a supervised fine-tuning training method, and the second stage adopts a reinforcement learning training method. In the first stage, the first training sample and the second training sample used include different sample labels, combining reasoning with retrieval, so that the large language model can output a specific type of text segment when reasoning and responding, thereby performing format learning. In the second stage, reinforcement learning and retrieval environment are further used to enhance the reasoning ability of the model, and dynamically retrieve external knowledge to solve problems. Since the second stage is based on the first stage, through the first stage, the model can generate a response in a specific format and enable it to adaptively utilize internal and external knowledge, so that the initial model of the subsequent reinforcement learning is relatively stable and the output format meets the requirements, thereby ensuring stable training in reinforcement learning.

[0154] According to another embodiment, a device for training a large language model is also provided. The generated data of the first large language model includes the answer to the input question or the query statement for the input question. The device is used to perform the embodiment of this specification. Figure 2 Described method. Figure 6 FIG. 1 is a schematic block diagram of an apparatus for training a large language model according to an embodiment. Figure 6 As shown, the apparatus 600 includes:

[0155] An acquiring unit 61 is configured to acquire a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample;

[0156] The training unit 62 is configured to perform supervised training on the first large language model based on the first training samples and the second training samples acquired by the acquisition unit.

[0157] Optionally, as an embodiment, the first large language model obtains an answer based on n rounds of reasoning, the first n-1 rounds of reasoning include generating a query statement, calling a search engine based on the query statement to obtain retrieval information, and performing the next round of reasoning based on the retrieval information.

[0158] Furthermore, the first training sample or the second training sample includes a sample input to be input into the first large language model, and the sample input includes:

[0159] A first sample input consisting of instructions and questions; or,

[0160] The second sample input consists of instructions, questions, reasoning statements, query statements and retrieval information.

[0161] Furthermore, the first training sample or the second training sample includes at least one of the following marking symbol combinations:

[0162] A first marking symbol combination is used to mark the text segment type in the combination as an inference statement;

[0163] A second marking symbol combination is used to mark the text segment in the combination as a query statement;

[0164] A third marking symbol combination is used to mark the text segment type in the combination as search information;

[0165] The fourth marking symbol combination is used to mark the text segment type in the combination as an answer.

[0166] Furthermore, the first training samples include at least one of the following types of samples:

[0167] For the first type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination;

[0168] The second type of samples, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination.

[0169] Furthermore, the second training samples include at least one of the following types of samples:

[0170] The third type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first tag symbol combination and the query statement in the second tag symbol combination;

[0171] The fourth type of sample, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination.

[0172] Furthermore, the acquisition unit 61 includes:

[0173] a first prediction subunit, configured to input the instruction and question into a second large language model to obtain output text for n rounds of reasoning; the second large language model having more parameters than the first large language model, and the generated data of any one round of reasoning in the n rounds of reasoning including an answer to the input question or a query statement for the input question;

[0174] a first judging subunit, configured to judge whether the answer contained in the output text obtained by the first predicting subunit is correct;

[0175] The first construction subunit is configured to construct a first training sample or a second training sample according to the instruction, question, and target round reasoning output text if the first judgment subunit determines that the answer is correct.

[0176] Furthermore, the first construction subunit is specifically used to:

[0177] The sample input is composed of instructions and questions, and the sample label is composed of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination included in the output text of the last round, to construct a first training sample; or,

[0178] The sample input is composed of the instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of any round of the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination, which are included in the output text of the last round, to construct the first training sample.

[0179] Furthermore, the first construction subunit is specifically used to:

[0180] The second training sample is constructed by forming a sample input consisting of an instruction and a question, and forming a sample label consisting of an inference statement in a first tag symbol combination and a query statement in a second tag symbol combination included in the output text of any round in the first n-1 rounds; or,

[0181] The sample input is composed of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of the first round in the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination, which are included in the output text of the second round in the first n-1 rounds after the first round, to construct the second training sample.

[0182] Furthermore, the instruction includes:

[0183] Each time new information is obtained, the generated reasoning statement is placed within the first marking symbol combination;

[0184] When external retrieval is required, the generated query statement is placed in the second mark symbol combination;

[0185] Extract query statements and request external search engines to retrieve relevant documents;

[0186] Append the search information to the generated word sequence and place it in the third token combination as additional context for the next step of generation;

[0187] The entire reasoning process is iterated until the maximum number of retrievals is reached or the model generates a final answer, and the answer is placed in the fourth token combination.

[0188] According to another embodiment, another apparatus for training a large language model is provided, wherein the generated data of the first large language model includes an answer to an input question or a query statement for an input question, and the apparatus is used to execute the method provided in the embodiment of this specification. Figure 4 Described method. Figure 7 FIG. 5 is a schematic block diagram of an apparatus for training a large language model according to another embodiment. Figure 7As shown, the apparatus 700 includes:

[0189] A first acquiring unit 71 is configured to acquire a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample;

[0190] A first training unit 72 is configured to perform supervised training on the first large language model based on the first training sample and the second training sample acquired by the first acquiring unit 71;

[0191] A second acquiring unit 73 is configured to acquire a third training sample, where the third training sample includes a sample question and a labeled answer;

[0192] The second training unit 74 is used to obtain predicted answers using the first large language model based on the sample questions included in the third training samples obtained by the second acquisition unit 73, and to perform reinforcement learning training on the first large language model obtained by the first training unit 72 after supervised training based on the predicted answers and the labeled answers.

[0193] Among them, the first acquisition unit 71 is the same as the aforementioned acquisition unit 61, and the first training unit 72 is the same as the aforementioned training unit 62, which will not be repeated here.

[0194] Optionally, as an embodiment, the second obtaining unit 73 includes:

[0195] a second prediction subunit, configured to input the instruction and question into a second language model to generate output text; the second language model has more parameters than the first language model, and the generated data includes an answer to the input question or a query statement for the input question;

[0196] a second judging subunit, configured to judge whether the answer contained in the output text obtained by the second predicting subunit is correct;

[0197] The second construction subunit is used to enable sampling for the second largest language model if the second judgment subunit determines that the answer is incorrect. For the question, if the second largest language model is correct at least once in a preset number of answers, retain the question and the correct answer to obtain a third training sample.

[0198] Optionally, as an embodiment, the objective function of the reinforcement learning training in the second training unit 74 is determined based on the policy model and the reference model, and both the policy model and the reference model are initialized with a large language model trained by supervised learning.

[0199] Optionally, as an embodiment, the second training unit 74 is specifically configured to determine a reward score based on the difference between the predicted answer and the labeled answer.

[0200] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 or Figure 4 The method described.

[0201] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 2 or Figure 4 The method described.

[0202] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0203] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a large language model, wherein generated data for a first large language model includes an answer to an input question or a query statement for the input question, the method comprising: Obtaining a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample; Based on the first training sample and the second training sample, supervised training is performed on the first large language model.

2. The method according to claim 1, wherein The method further comprises: Obtaining a third training sample, wherein the third training sample includes a sample question and a labeled answer; Based on the sample questions included in the third training sample, the first large language model is used to obtain a predicted answer, and based on the predicted answer and the labeled answer, reinforcement learning training is performed on the first large language model after supervised training.

3. The method according to claim 1, wherein The first language model obtains an answer based on n rounds of reasoning. The first n-1 rounds of reasoning include generating a query statement, calling a search engine based on the query statement to obtain retrieval information, and performing the next round of reasoning based on the retrieval information.

4. The method according to claim 3, wherein: The first training sample or the second training sample includes a sample input to be input into the first large language model, and the sample input includes: A first sample input consisting of instructions and questions; or, The second sample input consists of instructions, questions, reasoning statements, query statements and retrieval information.

5. The method according to claim 3, wherein: The first training sample or the second training sample includes at least one of the following marker symbol combinations: A first marking symbol combination is used to mark the text segment type in the combination as an inference statement; A second marking symbol combination is used to mark the text segment in the combination as a query statement; A third marking symbol combination is used to mark the text segment type in the combination as search information; The fourth marking symbol combination is used to mark the text segment type in the combination as an answer.

6. The method of claim 3, wherein: The first training samples include at least one of the following types of samples: For the first type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination; The second type of samples, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination.

7. The method of claim 3, wherein: The second training samples include at least one of the following types of samples: The third type of samples, the sample input consists of instructions and questions, and the sample label consists of the reasoning statement in the first tag symbol combination and the query statement in the second tag symbol combination; The fourth type of sample, the sample input consists of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination and the retrieval information in the third mark symbol combination, and the sample label consists of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination.

8. The method of claim 3, wherein: The first training sample or the second training sample is obtained in the following manner: Inputting the instruction and question into a second language model to obtain output text for n rounds of reasoning; the second language model has more parameters than the first language model, and the generated data of any round of reasoning in the n rounds of reasoning includes an answer to the input question or a query statement for the input question; Determine whether the answer contained in the output text is correct; If the answer is determined to be correct, a first training sample or a second training sample is constructed based on the instruction, question, and output text of the target round reasoning.

9. The method of claim 8, wherein: The step of constructing a first training sample based on the output text of the instruction, question, and target round reasoning includes: The sample input is composed of instructions and questions, and the sample label is composed of the reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination included in the output text of the last round, to construct a first training sample; or, The sample input is composed of the instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of any round of the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the answer in the fourth mark symbol combination, which are included in the output text of the last round, to construct the first training sample.

10. The method of claim 8, wherein: The constructing of a second training sample based on the output text of the instruction, question, and target round reasoning includes: The second training sample is constructed by forming a sample input consisting of an instruction and a question, and forming a sample label consisting of an inference statement in a first tag symbol combination and a query statement in a second tag symbol combination included in the output text of any round in the first n-1 rounds; or, The sample input is composed of instructions, questions, the first reasoning statement in the first mark symbol combination, the query statement in the second mark symbol combination, and the retrieval information in the third mark symbol combination, which are included in the output text of the first round in the first n-1 rounds. The sample label is composed of the second reasoning statement in the first mark symbol combination and the query statement in the second mark symbol combination, which are included in the output text of the second round in the first n-1 rounds after the first round, to construct the second training sample.

11. The method of claim 8, wherein: The instructions include: Each time new information is obtained, the generated reasoning statement is placed within the first marking symbol combination; When external retrieval is required, the generated query statement is placed in the second mark symbol combination; Extract query statements and request external search engines to retrieve relevant documents; Append the search information to the generated word sequence and place it in the third token combination as additional context for the next step of generation; The entire reasoning process is iterated until the maximum number of retrievals is reached or the model generates a final answer, and the answer is placed in the fourth token combination.

12. The method of claim 2, wherein: The third training sample is obtained by: Inputting the instruction and question into a second language model to obtain output text; the second language model has more parameters than the first language model, and the generated data includes an answer to the input question or a query statement for the input question; Determine whether the answer contained in the output text is correct; If the answer is determined to be incorrect, sampling is enabled for the second largest language model. For the question, if the second largest language model is correct at least once out of a preset number of answers, the question and the correct answer are retained to obtain a third training sample.

13. The method of claim 2, wherein: The objective function of the reinforcement learning training is determined based on a policy model and a reference model, and both the policy model and the reference model are initialized with a large language model trained by supervised learning.

14. The method of claim 2, wherein: The predicted answer belongs to the output text of the first language model, and the output text also includes the search information obtained by calling the search engine; The reinforcement learning training includes: performing masking processing on the output text obtained by the first language model to shield the search information included in the output text; Calculate the loss for the masked output text.

15. The method of claim 2, wherein: The reinforcement learning training includes: Based on the difference between the predicted answer and the labeled answer, a reward score is determined.

16. A device for training a large language model, wherein generated data for a first large language model includes an answer to an input question or a query statement to the input question, the device comprising: an acquiring unit, configured to acquire a first training sample and a second training sample, wherein the sample label in the first training sample includes an answer to an input question in the first training sample, and the sample label in the second training sample includes a query statement for the input question in the second training sample; A training unit is configured to perform supervised training on the first large language model based on the first training sample and the second training sample acquired by the acquisition unit.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 15.

18. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 15 is implemented.

Images