Large language model training method and computer equipment

Abstract

The invention provides a large language model training method and computer equipment, and the method comprises the steps: obtaining a first passing rate track of a large language model for a label-free first problem text in the latest t rounds of training, the first passing rate track comprising a sequence formed by all reasoning accuracy rates for the first problem text in the t rounds of training; determining a first similarity between the first passing rate track and a standard passing rate track; the standard passing rate track is determined according to the passing rate track corresponding to each question and answer sample in the target sample set; under the condition that the first similarity meets a preset condition, forming a first question and answer sample based on the first question text and a pseudo tag thereof, and adding the first question and answer sample to the target sample set; and executing the next round of training, and updating the large language model according to the target sample set.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method for training a large language model and a computer device. Background Technology

[0002] With the development of artificial intelligence technology, Large Language Models (LLMs) have been increasingly used. LLMs generally employ supervised training methods. Supervised training relies on manually labeled data, which can lead to high labeling costs if large-scale data or data from multiple professional domains are used to train the model.

[0003] To reduce the reliance on labeled data for training, unsupervised training methods have been employed in related techniques. However, unsupervised training methods lack real-world feedback, which may lead to the repeated reinforcement of erroneous inference paths, resulting in model performance degradation. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a method and computer device for training a large language model.

[0005] According to a first aspect of one or more embodiments of this specification, a method for training a large language model is proposed, comprising:

[0006] Obtain the first pass rate trajectory of the large language model for the first unlabeled question text in the most recent t rounds of training. The first pass rate trajectory includes a sequence of inference accuracies for the first question text in the t rounds of training.

[0007] Determine the first similarity between the first pass rate trajectory and the standard pass rate trajectory; the standard pass rate trajectory is determined based on the pass rate trajectories corresponding to each question-and-answer sample in the target sample set.

[0008] If the first similarity meets the preset conditions, a first question-and-answer sample is constructed based on the first question text and its pseudo-tags, and added to the target sample set;

[0009] Perform the next round of training, in which the large language model is updated based on the target sample set.

[0010] According to a second aspect of one or more embodiments of this specification, a large language model training apparatus is provided, comprising:

[0011] The pass rate trajectory acquisition module is used to acquire the first pass rate trajectory of the large language model for the first unlabeled question text in the most recent t rounds of training. The first pass rate trajectory includes a sequence of inference accuracies for the first question text in the t rounds of training.

[0012] A similarity determination module is used to determine the first similarity between the first pass rate trajectory and the standard pass rate trajectory; the standard pass rate trajectory is determined based on the pass rate trajectories corresponding to each question-and-answer sample in the target sample set.

[0013] The judgment module is used to construct a first question-and-answer sample based on the first question text and its pseudo-labels and add it to the target sample set when the first similarity meets the preset conditions.

[0014] The training module is used to perform the next round of training, wherein the large language model is updated based on the target sample set.

[0015] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the large language model training method as described in the first aspect of the embodiments of this specification.

[0016] According to a fourth aspect of the embodiments of this specification, a computer device is provided, the computer device comprising:

[0017] processor;

[0018] Memory used to store processor-executable instructions;

[0019] The processor implements the large language model training method as described in the first aspect of the embodiments of this specification by running the executable instructions.

[0020] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, which, when executed by a processor, implements the large language model training method as described in the first aspect of the embodiments of this specification.

[0021] This specification proposes a large language model training method that determines the pass rate trajectory for unlabeled question-and-answer samples. The pass rate trajectory indicates the change in inference accuracy of unlabeled samples over multiple training rounds. The pass rate trajectory of unlabeled samples is compared with a standard pass rate trajectory (determined based on the pass rate trajectory of reliable question-and-answer samples), and reliable unlabeled samples similar to the standard pass rate trajectory are selected. These selected reliable question-and-answer samples are then used for model training. Compared to related techniques that rely solely on static features for label alignment, this specification formalizes the evolution of the model's mastery of each sample during training into a quantified signal, namely the pass rate trajectory. This pass rate trajectory characterizes the dynamic development of inference ability. Furthermore, dynamic alignment is achieved through learning the pass rate trajectory, calculating the similarity between the pass rate trajectory of unlabeled question-and-answer samples and the standard pass rate trajectory, thus realizing knowledge transfer and reliability assessment across heterogeneous problems. Consequently, training can be performed using fewer labeled question-and-answer samples, effectively reducing labeling costs.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0024] Figure 1 This is a schematic diagram illustrating the relevant technology shown in this specification.

[0025] Figure 2 This is a flowchart illustrating a large language model training method as shown in this specification.

[0026] Figure 3 This is a schematic diagram illustrating a large language model training method as shown in this specification.

[0027] Figure 4 This is a hardware structure diagram of a computer device shown in this specification. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0029] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0030] Reinforcement learning with verifiable rewards (RLVR) has become an important technique for training Large Reasoning Models (LRMs). RLVR utilizes question-answer pairs with verifiable, unique answers (hereinafter referred to as labeled question-answer samples) to optimize preferences in LRMs. This specification involves two types of question-answer samples: labeled question-answer samples and unlabeled question-answer samples. Labeled question-answer samples include the corresponding question text and a label (also called the answer text), while unlabeled question-answer samples include only the corresponding question text. For distinction, the question text of labeled question-answer samples will be referred to as labeled question text, and the question text of unlabeled question-answer samples will be referred to as unlabeled question text.

[0031] like Figure 1 As shown, in a typical RLVR (i.e., supervised RLVR), LLM generates multiple inference paths from the input labeled question text q, resulting in multiple answer texts o. Rewards are calculated by verifying whether the answer texts are consistent with the pre-labeled answer texts a (i.e., labels), and the model is then fine-tuned.

[0032] However, a typical RLVR requires a large number of labeled tags, which results in high labeling costs.

[0033] In related technologies, unsupervised RLVR methods are used to reduce reliance on labeled question-answer samples. For example... Figure 1 As shown, in unsupervised RLVR, a reward mechanism is constructed using internal model signals (such as output entropy, self-confidence, majority voting, etc.). Among these, Figure 1The dashed box 'a' represents a missing label. Unsupervised RLVR methods assume that the LLM (Limited Learning Module) internalizes relatively rich knowledge during the pre-training phase, and then the LLM stimulates its intrinsic reasoning ability through self-supervision during the unsupervised RLVR process. However, this method lacks external, realistic feedback, and its reward signal is affected by the LLM's own biases, making it prone to falling into false consensus. That is, the LLM generates high confidence in incorrect answers and continuously reinforces them, leading to a self-circulating reward signal and performance degradation. For example, in reasoning tasks involving mathematical problem texts, the LLM may repeatedly use false consensus to arrive at consistent but incorrect results.

[0034] like Figure 1 As shown, related technologies also employ semi-supervised methods, using a small number of labeled question-and-answer samples to guide pseudo-label learning on a large number of unlabeled question-and-answer samples. This method typically calculates the similarity between labeled and unlabeled question texts. For any unlabeled question text, the label corresponding to the labeled question text with the highest similarity is used as the pseudo-label for that unlabeled question text. However, this method assumes that different samples share a discrete and fixed label space, making it suitable only for scenarios with a fixed label space, such as classification tasks. In large language model training scenarios, question texts are rich and diverse, and the corresponding answers are also diverse, making it difficult for existing labels to cover all unlabeled question texts. Such training may lead to erroneous knowledge transfer, affecting the training effect of LLM.

[0035] Based on this, this specification proposes a large language model training method. It determines the pass rate trajectory for unlabeled question-and-answer samples, which indicates the change in inference accuracy of unlabeled samples over multiple training rounds. The pass rate trajectory of unlabeled samples is compared with a standard pass rate trajectory (determined based on the pass rate trajectory of reliable question-and-answer samples). Reliable unlabeled samples similar to the standard pass rate trajectory are selected, and these selected reliable question-and-answer samples are used for model training.

[0036] Compared to related technologies that rely solely on static features for label alignment, this specification formalizes the evolution of the model's mastery of each sample during training into a quantified signal, namely the pass rate trajectory. This trajectory characterizes the dynamic development of reasoning ability. Furthermore, dynamic alignment is learned through the pass rate trajectory, calculating the similarity between the pass rate trajectory of unlabeled question-and-answer samples and the standard pass rate trajectory, thus achieving knowledge transfer and reliability assessment across heterogeneous problems. Consequently, training can be performed using fewer labeled question-and-answer samples, effectively reducing labeling costs.

[0037] Furthermore, the method described in this specification uses pass rate trajectories to screen reliable question-and-answer samples in each training round, dynamically updates the target sample set, gradually incorporates validated unlabeled question-and-answer samples, and continuously expands the target sample set to achieve better model training.

[0038] Compared to unsupervised methods, the method described in this specification does not directly use the model's current output internal signals (such as output entropy, self-confidence, majority vote, etc.) as rewards. Instead, it filters reliable unlabeled question-and-answer samples through the pass rate trajectory and trains the model using these selected reliable unlabeled question-and-answer samples. Unlabeled question-and-answer samples that are not selected are not used for training. In this way, even if the model produces a high-confidence incorrect answer to a certain unlabeled question-and-answer sample, the pass rate trajectory of that sample differs significantly from the standard pass rate trajectory, thus preventing the model from being used for training. This effectively avoids erroneous consensus and ensures the model's accuracy.

[0039] Compared to semi-supervised methods, this specification no longer focuses on the correlation between similar question texts, nor does it generate pseudo-labels for unlabeled question texts based on static features. Instead, it focuses on the similarity of the learning process across samples, that is, whether the model gradually masters a sample in a similar way, and the level of mastery of the sample during the model's dynamic learning process. By using the pass rate trajectory as a universal representation across question-answering samples, the alignment problem caused by the open solution space is bypassed, achieving effective knowledge transfer between heterogeneous questions.

[0040] The following section will explain a large language model training method provided in this manual. First, it should be noted that although RLVR was used as an example in the preceding explanation, the method described here can be applied not only to the fine-tuning of LLMs but also to the training process of LLMs. Furthermore, this method can also be applied to the training of other reinforcement learning models.

[0041] First, we can obtain multiple question-and-answer samples with labels, as well as multiple unlabeled question-and-answer samples. As mentioned earlier, labeled question-and-answer samples include tagged question text and labels. Unlabeled question-and-answer samples include unlabeled question text. A label can be a complete paragraph or sentence. In some cases, the question text has a fixed answer, such as a math problem or a factual question; in this case, the label can be that fixed answer.

[0042] Before training, a target sample set consisting of multiple labeled question-and-answer samples is first constructed.

[0043] In the initial warm-up phase, during the n rounds of training, the model is trained using the target sample set, and the LLM is updated. Each of the n rounds of warm-up includes determining the labeled question-and-answer sample set as the target sample set, updating the large language model using the target sample set, and updating the pass rate trajectory for each question-and-answer sample in the target sample set. This allows the model to be trained based on reliable question-and-answer samples during the warm-up phase, while simultaneously accumulating the pass rate trajectory for each question-and-answer sample for later use.

[0044] The pass rate trajectory consists of a sequence of inference accuracies for each question text corresponding to the question-answer sample in each round of training. For example, if the inference accuracy for a certain question text is 0.9 in the first round of training, 0.92 in the second round, and 0.93 in the third round, then the pass rate trajectory for that question text can be [0.9, 0.92, 0.95].

[0045] In one optional implementation, for a second question-and-answer sample with labeled tags (the corresponding question text is called the second question text, and the tags are called the second tags), the method for determining the inference accuracy during any round of training may include:

[0046] First, obtain multiple second answer texts obtained by the large language model in this round of training for the second question text through multiple inference paths.

[0047] Specifically, multiple second answer texts (rollouts) can be generated for the second question text. This utilizes multiple inference paths, which follows the characteristic of reinforcement learning to utilize multiple inference paths. It can also avoid the problem that the accuracy of a certain round of inference may be inaccurate due to model errors in a certain inference process, which may rely on only one inference path.

[0048] Second, for each second response text, the sub-accuracy of the second response text is determined based on its comparison with the second tag.

[0049] Since the second question text is the question text of a question-and-answer sample with labels, the second label of the question-and-answer sample can be directly used to determine the sub-accuracy that characterizes whether the second answer text is accurate.

[0050] In one alternative implementation, the sub-accuracy can be determined based on the similarity between the second answer text and the second tag.

[0051] In one optional implementation, when the second question text is a question text with a fixed answer, and the second label is that fixed answer, the answer output by the LLM for the second question text can be determined from the second answer text. In some cases, the second answer text only includes the answer; in other cases, the second answer text may include reasoning processes, etc., in addition to the answer. Then, the second label and the answer can be input into an indicator function to determine the sub-accuracy. Specifically, the indicator function can be II(a i (t) =y), where i is the identifier of the second answer text among multiple second answer texts output in response to the second question text, t is the current loop iteration, y is the second tag, and a i (t) This is the answer determined from the text of the second question. The indicator function is 1 if the second label and the answer are equal, and 0 otherwise.

[0052] Third, based on the sub-accuracy corresponding to multiple second answer texts, determine the inference accuracy of the second question-and-answer sample in this round of training.

[0053] The inference accuracy for the second question text in this round of training can be obtained by combining the sub-accuracies of multiple second answer texts. In an optional implementation, the inference accuracy can be obtained by averaging the multiple sub-accuracies.

[0054] In another alternative implementation, the inference accuracy of a certain round of training in the pass rate trajectory of the second question-answering sample is obtained through the following steps:

[0055] Obtain the second answer text output by the large language model for the second question text during this round of training; determine the second score of the second answer text based on the pre-trained evaluation model; and determine the inference accuracy for the second question text during this round of training based on the second score.

[0056] An evaluation model can be pre-trained, and the accuracy of the LLM output can be determined based on the evaluation model. In one optional implementation, the second answer text can be directly input into the evaluation model. In another optional implementation, the second question text and / or the second label, along with the second answer text, can be input into the evaluation model, so that the evaluation model can provide a more comprehensive evaluation of the second answer text.

[0057] In an optional implementation, multiple second answer texts can be generated, and multiple second scores corresponding to the multiple second answer texts can be obtained through an evaluation model. The inference accuracy rate is then determined based on the multiple second scores.

[0058] For the second question-answer sample with labels, the inference accuracy can be determined not only in the warm-up phase as described above, but also in every subsequent training round (including...). Figure 2 In the rounds shown, the inference accuracy can be determined as described above, and the newly obtained inference accuracy is added to the end of the pass rate trajectory corresponding to that sample to update the pass rate trajectory. In some cases, the length of the pass rate trajectory is not limited, so the pass rate trajectory can be updated directly based on the inference accuracy. For example, the pass rate trajectory can be obtained... In this context, the subscript q represents the representation of the question-and-answer sample, and the superscript number represents the number of iterations. This represents the pass rate trajectory obtained after the t-th round of the cycle. This represents the inference accuracy in the t-th cycle.

[0059] In another optional implementation, if the length of the pass rate trajectory exceeds a preset length, the inference accuracy at the beginning of the pass rate trajectory, i.e., the inference accuracy obtained from the round furthest from the current time point, can be deleted. The pass rate trajectory can then be updated based on the inference accuracy obtained from the current round after deletion.

[0060] The pass rate trajectory reflects the evolution of LLM’s mastery of the sample: a correctly learned sample should show a steady increase in pass rate, while incorrect or confused reasoning may show fluctuations or stagnation.

[0061] Furthermore, in any of the n training rounds during the warm-up phase, not only is the pass rate trajectory corresponding to each question-answer sample in the target sample set updated, but LLM is also used to infer the question texts for each unlabeled question text to determine the inference accuracy for each unlabeled question text in that training round, and the pass rate trajectory of the unlabeled question-answer sample is updated based on this. This allows the pass rate trajectory of unlabeled question texts to be accumulated. The method for updating the pass rate trajectory of unlabeled question-answer samples (hereinafter referred to as the first question-answer sample, and its corresponding question text is the first question text) based on the inference accuracy is the same as the method for updating the pass rate trajectory of the second question-answer sample mentioned above, and will not be repeated here.

[0062] Regarding the method for determining inference accuracy, in one optional implementation, when the first question text is a question text with a fixed answer, the inference accuracy for the first question text in any round of training is obtained through the following steps:

[0063] First, obtain multiple first answer texts from the large language model obtained through multiple inference paths for the first question text.

[0064] This step is similar to the process described above for obtaining multiple second answer texts for the second question text, and will not be repeated here.

[0065] Second, based on the self-consistency mechanism, pseudo-labels are determined according to the multiple first-response texts.

[0066] Based on the self-consistency mechanism, the answer supported by most reasoning paths can be regarded as the expected answer, i.e., a pseudo-label. This pseudo-label can be the answer determined from the answer text, rather than a long text. For example, if a user asks a math question, the answer text may include a long calculation process, while the pseudo-label can only include the final calculation result.

[0067] Third, for each first answer text, the sub-accuracy of the first answer text is determined based on its comparison with the pseudo-tags.

[0068] This process is similar to the previous method of comparing the second answer text with the pseudo-label. For example, it could involve checking whether the first answer text contains a pseudo-label. Alternatively, an indicator function can be used to obtain the sub-accuracy; the indicator function outputs 1 if the text indicating the answer to the first question in the first answer text is equal to the pseudo-label, and outputs 0 otherwise. A detailed explanation of the indicator function can be found above and will not be repeated here.

[0069] Fourth, the inference accuracy for the first question text in this round of training is determined based on the sub-accuracy of multiple first-response texts.

[0070] This process is similar to determining the inference accuracy of the second question-and-answer sample, and will not be repeated here. For example, the inference accuracy can be determined based on the following formula (1):

[0071]

[0072] Wherein, the subscript q represents the identifier of the first question-and-answer sample among all samples, the subscript i represents the identifier of the first answer text among all first answer texts, and G is the total number of first answer texts obtained for the first question text. The inference accuracy is the result obtained after the t-th iteration. For the t-th iteration of the loop, the answer is the i-th first response text obtained from the first question text. The pseudo-tags are determined for the first question text in the t-th iteration of the loop.

[0073] In another alternative implementation, the inference accuracy for the first question text in any round of training is obtained by the following steps: obtaining the first answer text output by the large language model for the first question text in that round of training; determining the first score of the first answer text based on the pre-trained evaluation model; and determining the inference accuracy for the first question text in that round of training based on the first score.

[0074] Specifically, an evaluation model can be pre-trained, and the accuracy of the first response text can be determined based on the evaluation model.

[0075] In one alternative implementation, the first answer text can be input into the evaluation model. In another alternative implementation, the first answer text and the first question text can be input into the evaluation model together to obtain a more accurate result.

[0076] Similar to the previous method, multiple first-response texts can be generated here, and multiple first scores corresponding to the multiple first-response texts can be obtained through the evaluation model. The accuracy of the above inference can be determined based on the multiple first scores.

[0077] Furthermore, in the training rounds following the warm-up phase, the above methods can also be used to determine the inference accuracy and update the pass rate trajectory, which will not be elaborated further below.

[0078] During any training round after the n rounds of warm-up training, the following can be executed: Figure 2 The methods shown include training an LLM, including:

[0079] Step 201: Obtain the first pass rate trajectory of the large language model for the first unlabeled question text in the most recent t rounds of training.

[0080] The first pass rate trajectory includes a sequence of inference accuracy rates for each inference question text during the t rounds of training.

[0081] Specifically, the pass rate trajectory for unlabeled question texts (i.e., the first question text) is first collected to facilitate the subsequent selection of reliable question-and-answer samples. The method in this specification can be applied to multiple first question texts, that is, the method in this specification can be applied to multiple unlabeled question texts.

[0082] The most recent t round here is executed after the n rounds of training in the model warm-up phase, or includes the last round in the n rounds of training.

[0083] Here, the most recent 't' rounds can specifically refer to all rounds since the model began training; that is, 't' is not a fixed value and will gradually increase as training progresses. Alternatively, 't' can be a preset fixed value, which can be used to obtain the first pass rate trajectory obtained from the most recent 't' rounds.

[0084] The method for obtaining the first pass rate trajectory is detailed above and will not be repeated here.

[0085] Step 203: Determine the first similarity between the first pass rate trajectory and the standard pass rate trajectory.

[0086] The standard pass rate trajectory is determined based on the pass rate trajectory corresponding to each question and answer sample in the target sample set.

[0087] In one optional implementation, the method can be executed only once after the warm-up phase, that is, only after the warm-up phase, based on the pass rate trajectory of the target sample set consisting of labeled question-and-answer samples, to filter unlabeled question-and-answer samples whose first pass rate trajectory meets preset conditions. The selected unlabeled question-and-answer samples and labeled question-and-answer samples are then used to complete subsequent rounds of LLM training.

[0088] Therefore, the pass rate trajectory of this standard can be determined based on the pass rate trajectory obtained from n rounds of training with labeled question-answer samples during the warm-up phase.

[0089] In this way, by using the pass rate trajectory of labeled question-and-answer samples as "cognitive anchors" and calculating the similarity between unlabeled question-and-answer samples and them in learning dynamics, the semantic gap between supervised and unsupervised data can be effectively bridged.

[0090] In another alternative implementation, the method can be executed cyclically, that is, the method shown in this specification is executed in each round after the warm-up phase to select new unlabeled question-and-answer samples that meet the preset conditions to be added to the target sample set, and the next round of training is completed based on the target sample set.

[0091] Therefore, the pass rate trajectory of this standard can be determined based on the pass rate trajectories of the two types of question and answer samples, which include labeled question and answer samples in the target sample set and reliable unlabeled question and answer samples selected in the past.

[0092] Optionally, the standard pass rate can be determined based on the average of the pass rate trajectories corresponding to each question-and-answer sample in the target sample set. Alternatively, the pass rate trajectories of each question-and-answer sample in the target sample set can be clustered to obtain the standard pass rate trajectory.

[0093] After obtaining the standard pass rate trajectory, the similarity between the standard pass rate trajectory and the first pass rate trajectory can be calculated. The similarity can be calculated based on cosine similarity, dot product similarity, etc. This manual does not limit the calculation method of similarity.

[0094] Step 205: If the first similarity meets the preset conditions, construct a first question-and-answer sample based on the first question text and its pseudo-tags, and add it to the target sample set;

[0095] Specifically, it can be determined whether the first pass rate trajectory is similar to the standard pass rate trajectory. If they are similar, the first question-and-answer sample can be considered a reliable sample and can be added to the target sample set. Otherwise, the first question-and-answer sample is not added to the target sample set.

[0096] The method for obtaining pseudo-labels can adopt the self-consistency mechanism mentioned above, which uses the answer agreed upon by the majority of reasoning paths as the pseudo-label.

[0097] The aforementioned preset conditions may include: the first similarity is greater than a preset threshold, and / or the first similarity ranks in the top p positions among the similarities corresponding to each unlabeled question text.

[0098] In this way, a dual screening mechanism of Top-N selection and similarity threshold filtering can be combined to dynamically identify and include high-quality unlabeled question-answer samples that are consistent with the learning process and the validated samples (i.e., reliable question-answer samples included in the target sample set), thereby avoiding the introduction of noise and error patterns and improving training stability.

[0099] Step 207: Perform the next round of training, wherein the large language model is updated based on the target sample set.

[0100] By using the above steps to filter the data, the selected reliable target sample set can be used to train the large language model and update its parameters.

[0101] The next training round here refers to the upcoming training, and the steps described above in this manual can be performed between two training rounds.

[0102] In an optional implementation, after step 207 is completed, inference can be performed on each labeled and unlabeled question text according to the updated large language model, based on the method mentioned above. The inference accuracy of this training round is calculated based on the inferred answer text, and the pass rate trajectory of each question-answer sample is updated accordingly. Step 201 can then be repeated to complete the training of the large language model.

[0103] There are several methods that can be used to update the LLM. Here, we will take the Group Relative Policy Optimization (GRPO) algorithm as an example to illustrate how to update the LLM.

[0104] The GRPO algorithm uses reinforcement learning to train the model by calculating the relative advantage of the model's multiple responses to the same question (the advantage is greater if the response is correct), so that the model's probability of answering correctly increases.

[0105] Accordingly, the training objective of LLM incorporates the GRPO loss of labeled question-answering samples. GRPO loss of unlabeled question-answering samples weighted by mask M As shown in formula (2):

[0106]

[0107] Wherein, if the above preset conditions include the first similarity being greater than the preset threshold, or the first similarity ranking in the top p positions among the similarities corresponding to each unlabeled question text, then the mask M of any unlabeled question-and-answer sample is determined according to the following formula (3):

[0108]

[0109]

[0110] in, This represents the trajectory of the average pass rate of each question-and-answer sample in the target sample set during the most recent t rounds of iteration. TCS represents the pass rate trajectory of the most recent t rounds of the unlabeled question-answering sample u, and TCS represents the cosine similarity between two pass rate trajectories. The parentheses represent an indicator function that outputs 1 if the condition in parentheses is met, and 0 otherwise. Γ represents a preset threshold, and ∨ represents an OR operation.

[0111] As can be seen from the above formula (3), if the first question text meets the above preset conditions, its corresponding mask is 1, otherwise it is 0. Therefore, according to formula (2), the loss in the method of this specification is calculated based on the question-and-answer samples of the target sample set.

[0112] The GPRO objective function can take the form of a policy gradient with clipping and include a KL regularization term to prevent the policy from deviating too far. The specific calculation method can be found in the following formula (4):

[0113]

[0114] Where, γ i，l (θ) is the importance sampling ratio, A i Both within the group and through estimation, the clip operation ensures a stable update step size.

[0115] As mentioned earlier, the target sample set may include a second question-and-answer sample consisting of the second question text and manually labeled second tags, as well as the first question text and its pseudo-labels. Figure 3As shown, this specification utilizes LLM to infer the first question text q1 and the second question text q2, respectively, yielding multiple answer texts o1-oG. The sub-accuracy r for each answer text can be calculated based on the label a2 or the pseudo-label a1 obtained through group voting. Then, the inference accuracy Pt for this round of iteration is determined based on the sub-accuracy r, and the pass rate trajectory is updated based on the inference accuracy Pt. Based on the average pass rate trajectory of the existing target sample set and the average pass rate trajectory among unlabeled question-and-answer samples, unlabeled question-and-answer samples that meet preset conditions are selected and added to the target sample set.

[0116] The method described in this specification utilizes dynamic evolution signals from multiple rounds of training, enabling a more comprehensive evaluation of the learning value of samples. For example, some unlabeled question-answering samples may initially have low inference accuracy, but their pass rate trajectory shows a stable upward trend, indicating that the model is gradually correcting errors and approaching the correct inference path. Conversely, some unlabeled question-answering samples may initially have high inference accuracy, but their pass rate trajectory fluctuates or stagnates, potentially reflecting that they have fallen into local errors. The trajectory similarity mechanism described above can identify the former and give them learning opportunities, ensuring that the selected samples are those that provide training gains for the large language model.

[0117] This specification presents a paradigm shift from "what to learn" to "how to learn." Compared to related technologies, this specification's approach shifts the learning focus from the model's output (what) to the learning process itself (how), using "learning dynamics" as a guiding signal to drive the evolution of semi-supervised reinforcement learning from outcome-driven to process-driven, which has significant methodological implications.

[0118] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of the apparatus and the computer equipment to which it is applied.

[0119] This specification describes a large language model training device, which includes:

[0120] The pass rate trajectory acquisition module is used to acquire the first pass rate trajectory of the large language model for the first unlabeled question text in the most recent t rounds of training. The first pass rate trajectory includes a sequence of inference accuracies for the first question text in the t rounds of training.

[0121] A similarity determination module is used to determine the first similarity between the first pass rate trajectory and the standard pass rate trajectory; the standard pass rate trajectory is determined based on the pass rate trajectories corresponding to each question-and-answer sample in the target sample set.

[0122] The judgment module is used to construct a first question-and-answer sample based on the first question text and its pseudo-labels and add it to the target sample set when the first similarity meets the preset conditions.

[0123] The training module is used to perform the next round of training, wherein the large language model is updated based on the target sample set.

[0124] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0125] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0126] As shown in Figure N, Figure 4 A hardware structure diagram of a computer device is shown. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0127] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above-described methods by running executable instructions.

[0128] The memory 1020 for storing processor-executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020.

[0129] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0130] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0131] Bus 1050 includes a pathway for transmitting information between various components of the device (e.g., processor 1010, memory 1020, input / output interface 1030, and communication interface 1040).

[0132] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0133] This specification also provides a computer program product that, when executed by a processor, implements the above-described large language model training method.

[0134] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described large language model training method.

[0135] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0138] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

Claims

1. A method for training a large language model, comprising: Obtain the first pass rate trajectory of the large language model for the first unlabeled question text in the most recent t rounds of training. The first pass rate trajectory includes a sequence of inference accuracies for the first question text in the t rounds of training. Determine the first similarity between the first pass rate trajectory and the standard pass rate trajectory; the standard pass rate trajectory is determined based on the pass rate trajectories corresponding to each question-and-answer sample in the target sample set. If the first similarity meets the preset conditions, a first question-and-answer sample is constructed based on the first question text and its pseudo-tags, and added to the target sample set; Perform the next round of training, in which the large language model is updated based on the target sample set.

2. The method according to claim 1, wherein, The most recent t round is executed after the n rounds of training in the model warm-up phase, or includes the last round in the n rounds of training; any round of the n rounds of training includes determining the question-answer sample set with labels as the target sample set, updating the large language model using the target sample set, and updating the pass rate trajectory corresponding to each question-answer sample in the target sample set.

3. The method according to claim 1, wherein, The inference accuracy for the first question text in any round of training (t rounds) is obtained through the following steps: The large language model is used to obtain multiple first answer texts for the first question text through multiple inference paths. Based on the self-consistency mechanism, pseudo-labels are determined according to the multiple first-response texts; For each first answer text, the sub-accuracy of that first answer text is determined based on its comparison with the pseudo-tags; The inference accuracy for the first question text in this round of training is determined based on the sub-accuracy of multiple first-response texts.

4. The method according to claim 3, wherein, The sub-accuracy of the first response text was determined by: The sub-accuracy is obtained using an indicator function; the indicator function outputs 1 if the text indicating the answer to the first question in the first answer text is equal to the pseudo-label, and outputs 0 otherwise.

5. The method according to claim 1, wherein, The inference accuracy for the first question text in any round of training (t rounds) is obtained through the following steps: Obtain the first answer text output by the large language model in this round of training for the first question text, and determine the first score of the first answer text according to the pre-trained evaluation model; Based on the first score, the inference accuracy for the first question text in this round of training is determined.

6. The method according to claim 1, wherein, The target sample set includes a second question-and-answer sample consisting of a second question text and manually labeled second tags; the inference accuracy of a certain round of training in the pass rate trajectory of the second question-and-answer sample is obtained through the following steps: Obtain multiple second answer texts from the large language model during this training round, obtained through multiple inference paths for the second question text; For each second response text, the sub-accuracy of that second response text is determined based on its comparison with the second tag; Based on the sub-accuracy corresponding to multiple second answer texts, the inference accuracy of the second question-answer sample in this round of training is determined.

7. The method according to claim 1, wherein, The target sample set includes a second question-and-answer sample consisting of a second question text and manually labeled second tags; the inference accuracy of a certain round of training in the pass rate trajectory of the second question-and-answer sample is obtained through the following steps: Obtain the second answer text output by the large language model in this round of training for the second question text, and determine the second score of the second answer text according to the pre-trained evaluation model; Based on the second score, the inference accuracy for the second question text in this round of training is determined.

8. The method according to claim 1, wherein, The preset conditions include: the first similarity is greater than a preset threshold, and / or, the first similarity ranks in the top p positions among the similarities corresponding to each unlabeled question text.

9. The method according to claim 1, wherein, The standard pass rate is determined based on the average pass rate trajectory corresponding to each question and answer sample in the target sample set.

10. A computer device, comprising: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-9 by executing the executable instructions.

Images