Large language model distribution preserving training method and device, and storage medium

Abstract

This invention relates to a method, apparatus, and storage medium for training a large language model that preserves its distribution, applicable to the field of artificial intelligence technology. The method includes: after acquiring the original base model, directly training the base model with full parameters using a group-relative strategy-optimized GRPO training framework, without starting with the SFT model. This avoids the cold start problem of the SFT model and also prevents the model from over-fitting the target training set, thus avoiding the loss of the original language distribution, knowledge organization, and generalization ability of the base model. Furthermore, this application structures the business task samples, enabling the model to fully understand and correctly read the input samples, providing multiple candidate responses under the same input, and reducing the risk of premature shrinkage of the output distribution.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method, apparatus, and storage medium for distributed training of large language models. Background Technology

[0002] Large language models typically undergo large-scale pre-training to form a base model, followed by post-training using methods such as SFT (Supervised Fine-Tuning), RLHF (Reinforcement Learning from Human Feedback), DPO (Direct Preference Optimization), PPO (Proximal Policy Optimization), and GRPO to adapt to specific tasks or business scenarios. In tasks such as customer service ticket classification, ride-hailing complaint risk identification, Chinese logical judgment, and multi-label text classification, post-training can significantly improve the model's ability to recognize business labels.

[0003] However, post-training also brings new risks. The model may become overly fitted to the target training set, performing better on business tasks, but losing the original language distribution, knowledge organization, and generalization ability of the base model. Especially when the training data has a fixed template, a narrow label set, or strong output format constraints, post-training on top of models such as SFT can easily cause the model's output distribution to shrink towards the training label template. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a large language model distribution-preserving training method, apparatus and storage medium, which aims to solve the problems in the prior art, where post-training is usually carried out on the basis of the SFT model, which has the problems of SFT cold start, the model may be overfitting the target training set, which may perform better in business tasks, but loses the original language distribution, knowledge organization and generalization ability of the base model.

[0005] According to a first aspect of the present invention, a method for training a large language model while maintaining its distribution is provided, the method comprising:

[0006] Obtain multiple business task samples and a base model. Each business task sample includes text input and a correct answer label. The business task samples are constructed into structured data that can be recognized by the group relative strategy optimization GRPO training framework to obtain structured business task samples. Multiple structured business task samples are combined to obtain the GRPO training set. The pedestal model is iteratively trained using the GRPO training set to obtain the GRPO model; the training process includes: The prompt field of the model input message in each of the structured business task samples is used to generate multiple candidate responses according to preset sampling parameters; The reward score for each candidate response is calculated based on the reward_model field configured in the structured business task sample. The estimated relative advantage of each candidate response within the group is calculated based on the reward score of each candidate response. Using the estimated intra-group relative advantage of each candidate response as a weight, all parameters of the base model are updated through the policy gradient algorithm to complete one round of iterative update; The base model is iteratively updated multiple times until a preset stopping condition is met, thus obtaining the GRPO model.

[0007] Preferably, The structured business task sample includes: a data source identifier field (data_source), a model input message field (prompt), a capability label field (ability), a reward model configuration field (reward_model), and an additional information field (extra_info); The model input message field prompt is a list of messages containing individual user messages. The content field of the user message contains the model role description, business input text, tag system, and output format requirements. The model input message field prompt replaces the system prompt during the model training phase.

[0008] Preferably, it further includes: After obtaining the structured business task sample, length detection is performed on the structured business task sample, and multiple structured business task samples after length detection are combined to obtain the GRPO training set; The length detection performed on the structured business task sample includes: Set a maximum length threshold for the model input message field prompt, and perform an error reporting and filtering operation on model input message fields prompt that exceed the maximum length threshold.

[0009] Preferably, it further includes: Perform a response evaluation on the GRPO model, including: Supervised fine-tuning of the SFT is performed on the base model to obtain the SFT model, and the GRPO model, base model, and SFT model are obtained. The business task samples from the same batch are respectively constructed into structured business task samples that can be recognized by the GRPO model, the base model, and the SFT model, and the GRPO verification dataset, the base verification dataset, and the SFT verification dataset are obtained based on the structured business task samples. The GRPO validation dataset, pedestal validation dataset, and SFT validation dataset are input into the corresponding GRPO model, pedestal model, and SFT model, respectively, to obtain the inference response of each of the GRPO model, pedestal model, and SFT model. MI metric verification is performed based on the inference responses of the GRPO model, the base model, and the SFT model, respectively. The GRPO model, base model, and SFT model employ a dual-message structure during the inference phase, consisting of the system prompt and the model input message field prompt, including: Write the business input text into the content field of the user message; write the model role description, tag system, and output format requirements into the system prompt word systemprompt.

[0010] Preferably, The step of performing MI metric verification based on the inference responses of the GRPO model, the base model, and the SFT model includes: The inference responses of the GRPO model, the base model, and the SFT model are input into the server of the base model, and the conditional log probability score of a single business task sample is generated through the server of the base model. The server also outputs the MI index of the GRPO model, the base model, and the SFT model based on the conditional log probability score of the single business task sample; the MI index is the average conditional log probability of the GRPO model, the base model, and the SFT model on the corresponding GRPO validation dataset, the base validation dataset, and the SFT validation dataset.

[0011] Preferably, The server also outputs the Delta distribution offset index of the GRPO model relative to the base model based on the MI index of the GRPO model and the base model; the Delta distribution offset index is the difference between the MI index of the GRPO model and the base model.

[0012] Preferably, The step of inputting the inference responses of the GRPO model, the base model, and the SFT model into the server of the base model, and generating the conditional log probability score of a single business task sample through the server of the base model, includes: For any inference response output by the GRPO model, the base model, and the SFT model, the model input message field prompt, the system prompt word, and the corresponding inference response are concatenated into a complete text full_text. The full text is input into the server of the base model, and the server of the base model generates a sequence of conditional log probability scores for each token in the full text. Filter out the portion of the prompt token in the conditional log probability score sequence, and sum the conditional log probability scores of the remaining tokens to obtain the conditional log probability score of the single business task sample.

[0013] Preferably, The step of filtering the portion of the prompt token in the conditional log probability score sequence, and summing the conditional log probability scores of the remaining tokens to obtain the conditional log probability score of the single business task sample includes: Concatenate the model input message field prompt and the system prompt word into a prefix text prompt_prefix; The prefix text prompt_prefix is ​​encoded by the tokenizer of the base model to obtain the number of prompt tokens; Based on the number of prompt tokens, skip the portion of the preceding prompt tokens in the conditional log probability score sequence, and sum the conditional log probability scores of the remaining tokens in the conditional log probability score sequence to obtain the conditional log probability score of the single business task sample.

[0014] According to a second aspect of the present invention, a large language model distribution-preserving training apparatus is provided, the apparatus comprising: Sample acquisition module: used to acquire multiple business task samples and base models. Each business task sample includes text input and correct answer labels; Training set construction module: used to construct the business task samples into structured data that can be recognized by the group relative strategy optimization GRPO training framework, to obtain structured business task samples, and to obtain a GRPO training set by combining multiple structured business task samples. Iterative training module: used to iteratively train the base model using the GRPO training set to obtain the GRPO model; the training process includes: The prompt field of the model input message in each of the structured business task samples is used to generate multiple candidate responses according to preset sampling parameters; The reward score for each candidate response is calculated based on the reward_model field configured in the structured business task sample. The estimated relative advantage of each candidate response within the group is calculated based on the reward score of each candidate response. Using the estimated intra-group relative advantage of each candidate response as a weight, all parameters of the base model are updated through the policy gradient algorithm to complete one round of iterative update; The base model is iteratively updated multiple times until a preset stopping condition is met, thus obtaining the GRPO model.

[0015] According to a third aspect of the present invention, a storage medium is provided, the storage medium storing a computer program, which, when executed by a host controller, implements the steps of the above-described method.

[0016] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This application, after obtaining the original base model, directly uses a group-relative strategy to optimize the GRPO training framework for full-parameter training of the base model, without starting with the SFT model. This avoids the cold start problem of the SFT model and also avoids the problem of the model overfitting to the target training set, resulting in the loss of the original language distribution, knowledge organization, and generalization ability of the base model. This application also performs structured processing on the business task samples, enabling the model to fully understand and correctly read the input samples, providing multiple candidate responses under the same input, and reducing the risk of premature shrinkage of the output distribution.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] Figure 1This is a flowchart illustrating a distribution-preserving training method for a large language model according to an exemplary embodiment; Figure 2 This is a schematic diagram of an iterative training process according to another exemplary embodiment; Figure 3 This is a system schematic diagram of a large language model distribution-preserving training device according to another exemplary embodiment; In the attached diagram: 1-Sample acquisition module, 2-Training set construction module, 3-Iterative training module. Detailed Implementation

[0020] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0021] Example 1: Figure 1 This is a flowchart illustrating a distribution-preserving training method for a large language model according to an exemplary embodiment, such as... Figure 1 As shown, the method includes: S1, obtain multiple business task samples and base models. Each business task sample includes text input and correct answer labels; S2, construct the business task samples into structured data that can be recognized by the group relative strategy optimization GRPO training framework to obtain structured business task samples, and combine multiple structured business task samples to obtain the GRPO training set. S3, the pedestal model is iteratively trained using the GRPO training set to obtain the GRPO model; Among them, as attached Figure 2 As shown, the training process includes: S301, generate multiple candidate responses from the prompt field of the model input message in each of the structured business task samples according to preset sampling parameters; S302, calculate the reward score for each candidate response based on the reward_model field in the structured business task sample; S303, calculate the estimated intra-group relative advantage of each candidate response based on the reward score of each candidate response; S304, using the estimated intra-group relative advantage of each candidate response as a weight, update all parameters of the base model through the policy gradient algorithm to complete one round of iterative update; S305, perform multiple rounds of iterative updates on the base model until a preset stopping condition is met, and obtain the GRPO model; It is understood that this embodiment specifically includes: The original pedestal model is obtained as the model to be trained. The pedestal model can be Qwen3-14B or a similar model. In this embodiment, GRPO training uses the Qwen3-14B pedestal model to start directly. Obtain business task samples and tags. Samples can come from ride-hailing customer service complaints, identification of major safety risks, classification of customer service work orders, Chinese logical judgment or other text understanding tasks. Each sample must include at least the text input and the corresponding correct answer tag. Each business sample is organized into structured data that can be recognized by the GRPO training framework, including at least the following fields:

[0022] The business input, model role description, label system and output format requirements of the prompt field are all written into the content field of the user message, and the system prompt word is not set separately in the training samples. The reward_model field has the following structure: it contains a ground_truth field (which stores the text of the correct answer label) and a style field (with a value of "rule", indicating that a rule-based reward calculation method is used). The extra_info field has the following structure: it includes the answer field (using...). <answer> Authentic Labels< / answer> The data includes a format package, an index field (sample index number), a question field (original question text), and a split field (data partition identifier, such as "train"). All of the above fields are stored in JSON string format; The technical effect of this data organization protocol is that it enables the GRPO training framework to stably identify prompts (model inputs), reward labels, and sample supplementary information, avoiding training link anomalies caused by differences in field structure or message role format. Before writing to the training set, the prompt length is checked. In this embodiment, the training length constraint parameters are as follows:

[0023] In addition to limiting the input length, this embodiment also limits the length of the output response, restricting the maximum length of the model's inference response to 1024 tokens; In summary, for samples exceeding the length threshold, summary truncation or filtering is performed to ensure that the samples meet the GRPO training length requirements. This mechanism is used to maintain the stability of the training structure of batch and micro-batch (minimum batch training samples) and avoid excessively long prompts from disrupting the training process. Experiments have shown that the training effect will decrease when the prompt length exceeds 5000 tokens. Therefore, the maximum prompt length is preferably 4028 tokens. The GRPO algorithm is used to train the Qwen3 pedestal model with full parameters. The key point of this step is not the specific training script parameters, but the training path: GRPO does not continue training from the SFT model, but directly performs full parameter training from the Qwen3 pedestal model. This can avoid the SFT distribution shift becoming the initial state of the subsequent model, and make the subsequent MI / Delta evaluation able to clearly compare the distances of the "Qwen3 original model output, SFT model output, and GRPO model output" under the same pedestal distribution. The framework used is VERL. During the GRPO training phase, multiple candidate responses are generated for each prompt. In this embodiment, the rollout (the process by which the model generates a complete response from the prompt) sampling parameters are as follows:

[0024] Here, n=8 indicates that the same input generates 8 candidate responses. This setting allows the model to form multiple possible inference paths and answer paths for the same business input, and then reinforces the better output through rule-based reward and intra-group relative advantage estimation. The training samples require the model output structure to include an inference section and an answer section: the inference section consists of... <think> and< / think> The label wraps the code to contain the model's reasoning process; the answer section is composed of... <answer> and< / answer> Tag wrapper, used to output the final category tags; The rule-based reward is based on reward_model.ground_truth, judging the model output. <answer>Whether the label content is consistent with the real label, the model not only learns the task label, but also learns how to put the answer into a fixed structure for output; The above training process specifically includes: For each prompt in the training samples, n candidate responses are independently generated using the policy model trained in the current GRPO according to preset sampling parameters; the sampling parameters include sampling temperature=1.0, kernel sampling threshold top_p=0.7, and the number of candidate responses n=8; each candidate response contains... <think>< / think> The inference section wrapped in the label and by <answer>< / answer> The answer section is wrapped in tags; For each generated candidate response, a reward score is calculated based on reward_model.ground_truth (correct answer label) and a predefined rule-based reward function; the rule-based reward function determines the candidate response's... <answer>Whether the text content within the label is completely consistent with the actual label text in the ground_truth; if it is consistent, a first reward value is assigned; if it is inconsistent, a second reward value is assigned. The n candidate responses corresponding to the same prompt are grouped together. Based on the calculated reward scores of each candidate response, the mean and standard deviation of the reward within the group are calculated, and then the estimated relative advantage of each candidate response within the group is calculated. The estimated relative advantage of the group is the reward score of the candidate response minus the mean reward within the group and divided by the standard deviation within the group. The calculated relative advantage estimates within the group are used as weights to update all parameters of the base model using the policy gradient algorithm. The update direction is to increase the generation probability of candidate responses with positive relative advantage and decrease the generation probability of candidate responses with negative relative advantage. Repeat the above steps until the preset iteration stopping conditions are met. The iteration stopping conditions include: the base model's task metrics on the training set converge or the preset number of training steps are reached. The above training mechanism increases the diversity of candidate responses under the same prompt; provides sufficient candidate samples for intra-group comparison of GRPO; avoids premature shrinkage to a single templated output during the training phase; helps to preserve the original multi-path generation distribution of the Qwen3 base model; and helps the model form the internal reasoning ability required for think / no_think output switching.

[0025] It is worth emphasizing that during the training phase, this embodiment does not set a separate system prompt. Instead, it encapsulates the model role description, business input text, tag system, and output format constraints into a single user message. This avoids differences in how the training framework or chat template interprets the system role.

[0026] During the inference phase, this embodiment restores the control function of the system prompt. The inference message adopts a dual message structure of system prompt plus user: the system prompt message contains model role descriptions, tag system and output format constraints; the user message only contains business input text. The inference sampling configuration adopts a low-temperature strategy, for example, the maximum number of output tokens is 500 and the sampling temperature is set to 0.01 to ensure output stability and reproducibility.

[0027] This heterogeneous message organization method for training and inference enables the model to learn complete task semantics and structured output capabilities during the training phase. During the inference phase, the system prompt controls whether the thought process is explicitly output, thus forming a hybrid inference retention and no-think output capability. This is also one of the important settings in this embodiment. If the system prompt is used during the training phase, the hybrid output capability of the base model Qwen3 will disappear, and the hybrid inference mode is the key to the success or failure of the project.

[0028] After training, the outputs of the Qwen3 pedestal model, the SFT model (obtained by supervised fine-tuning of SFT from the pedestal model, and used as a comparative reference model in this embodiment), and the GRPO model output (obtained from the above training) are uniformly evaluated. Responses are generated for each model, each dataset, and each sample. For example, when the number of samples is N=100, the number of models is 3, and the number of datasets is 3, approximately 900 responses are obtained in the generation stage.

[0029] The model set includes three categories: M_0 represents the Qwen3 base model, M_s represents the SFT model, and M_g represents the GRPO model. Among them, the SFT model is only used as the prediction output or control result and does not participate in the training process. This embodiment does not use each model's own tokenizer or probability distribution to score itself. Instead, it inputs the answers generated by all models into the Qwen3 original model evaluator, and the Qwen3 original model's tokenizer and the Qwen3 original model server calculate the conditional log probability logprob of the inference response.

[0030] For any prompt and its corresponding model inference response, perform the following logprob calculation steps: Concatenate the system prompt, the model input message prompt, and the inference response into a complete text full_text; Concatenate the system prompt and the model input message prompt into a prefix text prompt_prefix; The tokenizer of the Qwen3 base model is used to encode the prefix text prompt_prefix to obtain the number of prompt tokens; Call the text completion interface of the Qwen3 base model server, pass in the full text full_text, and the request parameters include: the maximum number of generated tokens is 1, the sampling temperature is 0, and the logprob of each token is returned (prompt_logprobs=1). The server returns the logprob sequence of each token in the full text file (full_text). Skip the first prompt_tokens tokens (the number of prompt tokens calculated above), and only accumulate the logprob of subsequent response tokens to obtain the total conditional log probability score for a single sample; The expression is:

[0031] In the formula, This represents the conditional probability distribution represented by the Qwen3 pedestal model; x i Indicates a sample; The unified scoring mechanism is the key to this application: all model outputs are evaluated in the same Qwen3 original distribution coordinate system, avoiding incomparability caused by the self-scoring of each model; In existing technologies, post-training methods for large language models typically focus on task accuracy, recall, F1 score, or human evaluation scores. However, they lack a quantitative system to quantify whether the trained model retains the original output distribution of the base model. After SFT or other post-training, the model may improve its performance on target business data, but it may also experience problems such as shift in the original language distribution, decreased cross-domain generalization, knowledge forgetting, output template-based approaches, and shrinking inference style. Maintaining the original model's output distribution means that catastrophic forgetting has not occurred, which is the most crucial aspect of post-training. To address this issue, this embodiment proposes an evaluation method based on the MI (Minimum Inference) metric, specifically including: In this embodiment, the MI metric is essentially a base cross-entropy distribution preservation measure function. It is not a regular accuracy rate, nor does it allow the model to evaluate itself. Instead, it puts the inference responses generated by different models back into the probability distribution of the original Qwen3 model and calculates the average log-likelihood of these inference responses under the base distribution.

[0032] From an information theory perspective, if we denote the Qwen3 pedestal model as the evaluation distribution... If we consider the set of answers generated by a model to be evaluated on a dataset as an empirical output distribution, then the inference response... The negative average logprob corresponds to a cross-entropy or negative log-likelihood cost. In this embodiment, the average logprob in the opposite direction is used as the MI score.

[0033] Therefore, the MI metric can be understood as "base distribution acceptance" or "reverse entropy cost score". It does not measure whether the answer hits the label, but rather whether the answer is natural, has low surprise, and still resembles what Qwen3 might generate in the original probability space.

[0034] Let the input sample set be:

[0035] Model M m In the sample x i The response above is:

[0036] The server-side calculation of the conditional log probability using the Qwen3 base model:

[0037] Define Model M m In the dataset D The MI indicator on the chart is:

[0038] in, V m To successfully obtain a sample index set with valid logprob, the higher the MI, the more natural the model output appears to the original Qwen3 model, and the closer it is to the original Qwen3 output distribution.

[0039] The key function of this function is to eliminate the blind spot of "good task metrics but bad distribution". Traditional accuracy only judges whether the final answer is correct. However, the MI metric proposed in this application judges whether the way the correct answer is generated is still recognized by the original Qwen3 model. For example, if two models answer the same question correctly, if the output of the SFT model has a low logprob under the Qwen3 distribution, while the output of the GRPO model has a higher logprob under the Qwen3 distribution, it means that the GRPO model not only answers correctly, but also answers correctly in a way that is closer to the base model.

[0040] This function also solves the problem of incomparable self-scoring scores of different models. If the SFT model, GRPO model, and Qwen3 base model score each model using their own tokenizer and probability distribution, each model will naturally be biased towards its own output, making it impossible to determine which is closer to the original Qwen3 distribution. This application uses the Qwen3 base model as the only evaluator, which is equivalent to fixing the same "entropy ruler". All inference responses are compared in the same probability coordinate system.

[0041] It can be equivalently understood as:

[0042] In the formula, Representation Model M m The output is relative to the base cross-entropy cost of the original Qwen3 distribution. The smaller the MI, the less surprised Qwen3 is by the model's output; conversely, the larger the MI, the more the model's output retains the original distribution characteristics of Qwen3. Calculate the Delta distribution offset index: Define the distribution offset of the trained model relative to the Qwen3 pedestal model:

[0043] For the SFT model and the GRPO model, calculate separately:

[0044]

[0045] The interpretation rules are as follows: The closer Delta is to 0, the closer the model is to the Qwen3 pedestal model distribution; A large negative Delta value indicates that the model output has shifted significantly relative to the original Qwen3 distribution; when Compare When the GRPO model's performance metrics are closer to 0 and reach those of the SFT model, it indicates that the GRPO model achieves both task capability and better base distribution preservation.

[0046] Delta transforms the MI from an absolute score into a relative offset. Different datasets, different prompt lengths, and different answer lengths all affect the absolute size of the original logprob. Therefore, this embodiment does not consider the MI of a single model, but uses the MI of the original Qwen3 model on the same dataset as the zero-point reference. This indicates how much more cross-entropy cost or how much less pedestal acceptance the GRPO model output has compared to the Qwen3 output itself.

[0047] From the perspective of entropy cost, it can also be written as:

[0048] therefore, The closer it is to 0, the closer the entropy cost increment is to 0. Superior This is equivalent to the GRPO model having a smaller additional cross-entropy cost to the original Qwen3 distribution compared to the SFT model. This design makes the question of "whether the model after training destroys the pedestal distribution" a computable function instead of an empirical judgment. The system outputs task metrics, MI metrics, Delta metrics, low confidence markers, and Markdown reports. If the number of valid logprob samples in a certain group is less than half of the total number of samples, that is, the number of valid samples is less than 0.5 × N, a low confidence warning is output. Ultimately, the deployment model can be selected based on the following rules: The GRPO model achieves the same performance metrics as the SFT model. Delta_GRPO is superior to Delta_SFT; The no_think inference output is stable; Business tag output conforms <answer>< / answer> structure; The distribution maintains an assessment with no significant risk of low confidence.

[0049] To provide a clearer understanding of the solution presented in this application, the following experimental data comparison is also provided in this embodiment: In one embodiment, the Qwen3 base model, SFT model, and GRPO model are used to evaluate on five datasets: MMLU, MMLU-Pro, Chinese Logic MC, Chinese Math, and CMMLU. Each model randomly samples 100 samples from each dataset. The Qwen3 base model tokenizer and the server uniformly calculate logprob, and then calculate the MI and Delta metrics.

[0050] The experimental results are shown in the table below:

[0051] Among them, 30 and 32 samples from the Chinese Math and CMMLU datasets, respectively, were excluded due to abnormal logprob calculations, resulting in valid sample numbers of 70 / 100 and 68 / 100, respectively. The evaluation conclusions were given with low confidence, but the overall trend can still be observed.

[0052] The above results indicate that: On all 5 datasets, All are much smaller than The distribution preservation performance of the GRPO model is superior to that of the SFT model across the board; The output of the GRPO model appears more natural than that of the SFT model in the Qwen3 pedestal model. On the Chinese Logic MC dataset, The value is only -0.0058, almost zero, indicating that its output distribution is very close to that of the Qwen3 pedestal model. On the Chinese Math dataset, With a value of +0.2442, MI slightly exceeds Qwen3's own level; The MI / Delta formula in this embodiment can transform "whether the model maintains the base distribution" into a calculable and comparable numerical value. Furthermore, in the ride-hailing customer service complaint risk classification task (taking the first 1400 test samples, covering 40+ risk categories), the task accuracy comparison between the SFT model and the GRPO model is as follows:

[0053] The SFT model outputs mostly long natural language descriptions with a low label hit rate (especially in multi-label scenarios, only 26.21%); while the GRPO model, through structured... <answer>The output and rule-based reward training significantly outperformed the SFT model on both single-label and multi-label tasks, with an overall accuracy improvement of 24.35 percentage points. This result shows that the proposed method can enable the GRPO model to directly acquire and surpass the business task capabilities of the SFT model without undergoing a cold start with SFT or sacrificing the pedestal distribution. At the same time, the MI / Delta index proves that its output distribution is closer to the original distribution of the Qwen3 pedestal.

[0054] Example 2: Figure 3 This is a system schematic diagram of a large language model distribution-preserving training apparatus according to another exemplary embodiment, the apparatus comprising: Sample Acquisition Module 1: Used to acquire multiple business task samples and base models. Each business task sample includes text input and correct answer labels. Training set construction module 2: used to construct the business task samples into structured data that can be recognized by the group relative strategy optimization GRPO training framework, to obtain structured business task samples, and to obtain a GRPO training set by combining multiple structured business task samples. Iterative Training Module 3: Used to iteratively train the base model using the GRPO training set to obtain the GRPO model; the training process includes: The prompt field of the model input message in each of the structured business task samples is used to generate multiple candidate responses according to preset sampling parameters; The reward score for each candidate response is calculated based on the reward_model field configured in the structured business task sample. The estimated intra-group relative advantage of each candidate response is calculated based on the reward score of each candidate response; Using the estimated intra-group relative advantage of each candidate response as a weight, all parameters of the base model are updated through the policy gradient algorithm to complete one round of iterative update; The base model is iteratively updated multiple times until a preset stopping condition is met, thus obtaining the GRPO model.

[0055] Example 3: This embodiment provides a storage medium storing a computer program, which, when executed by a host controller, implements the various steps in the above method. It is understood that the storage medium mentioned above can be a read-only memory, a hard disk, or an optical disk, etc.

[0056] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0057] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0058] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0059] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0060] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0061] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0062] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0063] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0064] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.< / answer> < / answer> < / answer>

Claims

1. A distribution-preserving training method for large language models, characterized in that, The method includes: Obtain multiple business task samples and a base model. Each business task sample includes text input and a correct answer label. The business task samples are constructed into structured data that can be recognized by the group relative strategy optimization GRPO training framework to obtain structured business task samples. Multiple structured business task samples are combined to obtain the GRPO training set. The pedestal model is iteratively trained using the GRPO training set to obtain the GRPO model; the training process includes: The prompt field of the model input message in each of the structured business task samples is used to generate multiple candidate responses according to preset sampling parameters; The reward score for each candidate response is calculated based on the reward_model field configured in the structured business task sample. The estimated intra-group relative advantage of each candidate response is calculated based on the reward score of each candidate response; Using the estimated intra-group relative advantage of each candidate response as a weight, all parameters of the base model are updated through the policy gradient algorithm to complete one round of iterative update; The base model is iteratively updated multiple times until a preset stopping condition is met, thus obtaining the GRPO model.

2. The method for maintaining the distribution of a large language model according to claim 1, characterized in that, The structured business task sample includes: a data source identifier field (data_source), a model input message field (prompt), a capability label field (ability), a reward model configuration field (reward_model), and an additional information field (extra_info); The model input message field prompt is a list of messages containing individual user messages. The content field of the user message contains the model role description, business input text, tag system, and output format requirements. The model input message field prompt replaces the system prompt during the model training phase.

3. The method for maintaining the distribution of a large language model according to claim 1, characterized in that, Also includes: After obtaining the structured business task sample, length detection is performed on the structured business task sample, and multiple structured business task samples after length detection are combined to obtain the GRPO training set; The length detection performed on the structured business task sample includes: Set a maximum length threshold for the model input message field prompt, and perform an error reporting and filtering operation on model input message fields prompt that exceed the maximum length threshold.

4. The method for maintaining the distribution of a large language model according to claim 3, characterized in that, Also includes: Perform a response evaluation on the GRPO model, including: Supervised fine-tuning of the SFT is performed on the base model to obtain the SFT model, and the GRPO model, base model and SFT model are obtained; The business task samples from the same batch are respectively constructed into structured business task samples that can be recognized by the GRPO model, the base model, and the SFT model, and the GRPO verification dataset, the base verification dataset, and the SFT verification dataset are obtained based on the structured business task samples. The GRPO validation dataset, the base validation dataset, and the SFT validation dataset are input into the corresponding GRPO model, base model, and SFT model, respectively, to obtain the inference response of each of the GRPO model, base model, and SFT model. MI metric verification is performed based on the inference responses of the GRPO model, the base model, and the SFT model, respectively. The GRPO model, base model, and SFT model employ a dual-message structure during the inference phase, consisting of the system prompt and the model input message field prompt, including: Write the business input text into the content field of the user message; write the model role description, tag system, and output format requirements into the system prompt word systemprompt.

5. The method for maintaining the distribution of a large language model according to claim 4, characterized in that, The step of performing MI metric verification based on the inference responses of the GRPO model, the base model, and the SFT model includes: The inference responses of the GRPO model, the base model, and the SFT model are input into the server of the base model, and the conditional log probability score of a single business task sample is generated through the server of the base model. The server also outputs the MI index of the GRPO model, the base model, and the SFT model based on the conditional log probability score of the single business task sample; the MI index is the average conditional log probability of the GRPO model, the base model, and the SFT model on the corresponding GRPO validation dataset, the base validation dataset, and the SFT validation dataset.

6. The method for maintaining the distribution of a large language model according to claim 5, characterized in that, The server also outputs the Delta distribution offset index of the GRPO model relative to the base model based on the MI index of the GRPO model and the base model; the Delta distribution offset index is the difference between the MI index of the GRPO model and the base model.

7. The method for maintaining the distribution of a large language model according to claim 5, characterized in that, The step of inputting the inference responses of the GRPO model, the base model, and the SFT model into the server of the base model, and generating the conditional log probability score of a single business task sample through the server of the base model, includes: For any inference response output by the GRPO model, the base model, and the SFT model, the model input message field prompt, the system prompt word, and the corresponding inference response are concatenated into a complete text full_text. The full text is input into the server of the base model, and the server of the base model generates a sequence of conditional log probability scores for each token in the full text. Filter out the portion of the prompt token in the conditional log probability score sequence, and sum the conditional log probability scores of the remaining tokens to obtain the conditional log probability score of the single business task sample.

8. The method for maintaining the distribution of a large language model according to claim 7, characterized in that, The step of filtering the portion of the prompt token in the conditional log probability score sequence, and summing the conditional log probability scores of the remaining tokens to obtain the conditional log probability score of the single business task sample includes: Concatenate the model input message field prompt and the system prompt word into a prefix text prompt_prefix; The prefix text prompt_prefix is ​​encoded by the tokenizer of the base model to obtain the number of prompt tokens; Based on the number of prompt tokens, skip the portion of the preceding prompt tokens in the conditional log probability score sequence, and sum the conditional log probability scores of the remaining tokens in the conditional log probability score sequence to obtain the conditional log probability score of the single business task sample.

9. A large language model distribution-preserving training device, characterized in that, The device includes: Sample acquisition module: used to acquire multiple business task samples and base models. Each business task sample includes text input and correct answer labels; Training set construction module: used to construct the business task samples into structured data that can be recognized by the group relative strategy optimization GRPO training framework, to obtain structured business task samples, and to obtain a GRPO training set by combining multiple structured business task samples. Iterative training module: used to iteratively train the base model using the GRPO training set to obtain the GRPO model; the training process includes: The prompt field of the model input message in each of the structured business task samples is used to generate multiple candidate responses according to preset sampling parameters; The reward score for each candidate response is calculated based on the reward_model field configured in the structured business task sample. The estimated intra-group relative advantage of each candidate response is calculated based on the reward score of each candidate response; Using the estimated intra-group relative advantage of each candidate response as a weight, all parameters of the base model are updated through the policy gradient algorithm to complete one round of iterative update; The base model is iteratively updated multiple times until a preset stopping condition is met, thus obtaining the GRPO model.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by the main controller, implements the various steps of the large language model distribution-preserving training method as described in any one of claims 1-8.

Images