Model training method, model inference method, and computing device

Abstract

The embodiment of the specification provides a model training method, a model inference method and a computing device, wherein the model training method comprises: obtaining a sampling group of a current policy model, wherein the sampling group comprises a sample instruction and at least one candidate output generated by the current policy model according to the sample instruction, and any candidate output is composed of multiple word pieces; determining a to-be-truncated candidate output from the at least one candidate output based on the number of word pieces of the at least one candidate output; truncating the multiple word pieces of the to-be-truncated candidate output to obtain a truncated candidate output; constructing an effective sampling group based on the sample instruction and the truncated candidate output; and performing reinforcement learning on the current policy model based on the effective sampling group to obtain a target policy model. The length drift phenomenon on the problem answer is inhibited, the gradient disappearance problem caused by the increase of the proportion of fully correct samples is avoided, the sampling efficiency is significantly improved, and the target policy model with high accuracy and low inference cost is obtained.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large language model technology, and in particular to a model training method, a model inference method, and a computing device. Background Technology

[0002] As large language models continue to improve their capabilities in reasoning tasks, reinforcement learning based on verifiable rewards has become a core paradigm for enhancing the logical rigor and mathematical / symbolic reasoning abilities of these models. Current mainstream methods rely on external validators to provide fine-grained, reliable reward signals, driving the model to generate more reliable and traceable thought processes. However, models exhibit irrational and redundant behavior with a large number of already mastered question answers, such as repeatedly restating premises, cyclically verifying intermediate conclusions, or generating reasoning paths far exceeding necessary length. This phenomenon severely restricts the engineering usability of the models.

[0003] Currently, the main problems with the model are: First, there is a significant length drift phenomenon. Second, directly using length as a component of the reward will cause gradient conflicts, thereby suppressing correct but slightly longer high-quality samples and impairing the policy exploration space. Third, as training progresses, the proportion of all correct candidates in the sampling group increases rapidly, leading to a sharp decline in the variance of the reward within the group, resulting in a deterioration of the policy gradient signal-to-noise ratio, a collapse in sampling efficiency, and the formation of a negative feedback loop.

[0004] Therefore, there is an urgent need for a new training paradigm that decouples length control from semantic optimization, does not modify the reward structure, and can adaptively maintain gradient effectiveness. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a model training method. One or more embodiments of this specification also relate to a model inference method, a question-answering model training method, a question-answering method, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a model training method is provided, comprising: Obtain the sampling group of the current policy model, wherein the sampling group includes sample instructions and at least one candidate output generated by the current policy model according to the sample instructions, and any candidate output consists of multiple lexical units; Based on the number of lexical units in at least one candidate output, determine the candidate output to be truncated from at least one candidate output; The multiple words in the candidate output to be truncated are truncated to obtain the truncated candidate output; Construct an effective sampling group based on sample instructions and truncated candidate outputs; Based on the effective sampling set, reinforcement learning is performed on the current policy model to obtain the target policy model.

[0007] According to a second aspect of the embodiments of this specification, a model reasoning method is provided, comprising: Obtain model instructions input by the user; Input the model instructions into the target policy model to obtain the target policy model's instruction output in response to the model instructions, wherein the target policy model is trained according to any model training method.

[0008] According to a third aspect of the embodiments of this specification, a question-answering model training method is provided, comprising: Obtain question-answer sample pairs from the current question-answering model, wherein each question-answer sample pair includes a sample question and at least one sample answer generated by the current question-answering model based on the sample question, and any sample answer consists of multiple word units; Based on the number of lexical units in at least one sample answer, determine the sample answer to be truncated from at least one sample answer; The multiple words in the truncated sample answer are truncated to obtain the truncated sample answer; Construct effective question-answer sample pairs based on sample questions and truncated sample answers; Based on valid question-answer sample pairs, reinforcement learning is performed on the current question-answering model to obtain the target question-answering model.

[0009] According to a fourth aspect of the embodiments of this specification, a question-and-answer method is provided, including: Get the question text entered by the user; Input the question text into the target question answering model to obtain the answer output by the target question answering model for the question text. The target policy model is trained according to the question answering model training method.

[0010] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of any one of the above-mentioned model training method, a model inference method, a question answering model training method, and a question answering method.

[0011] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of any one of the above-described model training method, a model inference method, a question-answering model training method, and a question-answering method.

[0012] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any one of the above-described model training method, a model inference method, a question-answering model training method, and a question-answering method.

[0013] The model training method provided in one or more embodiments of this specification includes: obtaining a sampling group of the current policy model, wherein the sampling group includes sample instructions and at least one candidate output generated by the current policy model according to the sample instructions, and any candidate output consists of multiple lexical units; determining a candidate output to be truncated from the at least one candidate output based on the number of lexical units in the at least one candidate output; truncating the multiple lexical units of the candidate output to be truncated to obtain a truncated candidate output; constructing an effective sampling group based on the sample instructions and the truncated candidate output; and performing reinforcement learning on the current policy model based on the effective sampling group to obtain a target policy model. By acquiring a sampling set containing sample instructions and multi-lexical candidate outputs, the original generation diversity and validation foundation of the model are preserved. Candidate outputs to be truncated are directly filtered and truncated based on the number of lexical terms, avoiding gradient conflicts between shortening length and maintaining accuracy from the data source without introducing additional length penalties or modifying the reward formula, thus preventing training instability caused by reward coupling. An effective sampling set is constructed using sample instructions and truncated candidate outputs, forcing the samples input to the reinforcement learning stage to converge to a shorter length distribution while retaining the correct inference logic, suppressing length drift in question answers. Reinforcement learning of the current policy model based on this effective sampling set avoids the gradient vanishing problem caused by an increased proportion of all-correct samples, significantly improving sampling efficiency and obtaining a target policy model that combines high accuracy and low inference cost. Attached Figure Description

[0014] Figure 1 A flowchart illustrating a model training method provided in one embodiment of this specification; Figure 2 A flowchart illustrating a model reasoning method provided in one embodiment of this specification; Figure 3 A flowchart illustrating a question-answering model training method provided in one embodiment of this specification; Figure 4 A flowchart illustrating a question-and-answer method provided in one embodiment of this specification; Figure 5 This is a schematic diagram of the structure of a model training device provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a model reasoning device provided in one embodiment of this specification; Figure 7This is a structural block diagram of a computing device provided for one embodiment of this specification. Detailed Implementation

[0015] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0016] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0017] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0018] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include large-scale language models (LLMs) and multi-modal pre-training models.

[0020] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0021] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0022] Length drift: refers to the phenomenon that during training, the average output length of a model continues to increase even when it can correctly solve simple inputs.

[0023] Group Relative Policy Optimization (GRPO) is a policy optimization method that does not explicitly fit the value function. Instead, it uses within-group relative comparisons to replace the value function estimation, normalizes rewards within groups, and estimates advantages to achieve efficient optimization for training large language models.

[0024] Proximal Policy Optimization (PPO) is a mainstream, stable, and efficient policy gradient-based reinforcement learning algorithm. It is widely used in practice due to its stable training, robust hyperparameters, and ease of implementation and tuning, making it a practical choice in the field.

[0025] Verifiable Rewards Reinforcement Learning (RLVR) is a new paradigm of reinforcement learning. Its core difference lies in that the reward signal originates from automatically verifiable objective standards, rather than human subjective preferences (RLHF). This reward can be directly determined through procedural rules or by using automatically verifiable task rewards (such as math answer validators), for example, whether a math answer is correct, whether code passes a test, or whether output conforms to format constraints, thus enabling reinforcement learning training.

[0026] Token: The basic discrete unit processed by a language model. It usually refers to the smallest unit of text data after word segmentation, such as a word, punctuation mark or number. The output length is usually measured in terms of the number of tokens.

[0027] KL divergence (also known as relative entropy or information divergence) measures the degree of difference between two probability distributions. The greater the difference, the greater the KL divergence; the smaller the difference, the smaller the KL divergence. When both are equal, the KL divergence is 0.

[0028] KL-Cov regularization: A targeted KL regularization strategy for high covariance lexical units, used to suppress the rapid collapse of policy entropy and maintain exploratory capabilities.

[0029] This specification provides a model training method. One or more embodiments of this specification also relate to a model inference method, a question-answering model training method, and a question-answering method, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0030] See Figure 1 , Figure 1 A flowchart of a model training method provided in one embodiment of this specification includes the following steps: Step 102: Obtain the sampling group of the current policy model, wherein the sampling group includes sample instructions and at least one candidate output generated by the current policy model according to the sample instructions, and any candidate output consists of multiple lexical units.

[0031] The current policy model is a language model instance used for reinforcement learning. In this specification, the current policy model can be in a specific training stage and carry the current parameter state during the reinforcement learning iteration process. It can be represented as a neural network architecture after pre-training initialization, before or during a certain round of reinforcement learning update. The role of the current policy model is to generate candidate output sequences containing multiple lexical units based on the input sample instructions, providing the original data foundation for subsequent truncation processing based on the number of lexical units and the construction of effective sampling groups.

[0032] A sampling set is a dataset consisting of a specific sample instruction and at least one candidate output generated by the current policy model according to that instruction. It provides an initial data source containing the diversity of original generation and lexical distribution characteristics, enabling subsequent selection of truncated objects based on lexical counts and construction of effective samples for reinforcement learning. This set can be represented as one or more sequences containing thought chains of varying lengths for mathematical problems, logical reasoning tasks, or code generation instructions, or as a candidate set encompassing a single correct solution and multiple redundant variations. Its specific composition dynamically changes with the sampling strategy or instruction, serving as the basic input unit for performing truncation operations and gradient updates.

[0033] Sample instructions are natural language prompts or task descriptions input to the current policy model to trigger the generation of candidate outputs. Sample instructions can serve as the starting conditions for the reasoning process, guiding the model to produce a response sequence composed of multiple lexical units for subsequent truncation and reinforcement learning evaluation. They can be presented as one or more forms of mathematical calculation problems, logical deduction propositions, code writing requirements, or common sense questions and answers, such as requiring the solution of a system of equations, generating travel guides, generating scripts with specific functions, etc.

[0034] Candidate output is the response sequence generated by the current policy model in response to sample instructions. It may include one or more lexical units, which serve as the basic data objects for subsequent filtering, truncation, and construction of effective sampling groups based on the number of lexical units. In other words, it provides raw material for implementing length control and policy optimization. Candidate output can take various forms, such as a thought chain containing a step-by-step reasoning process, a directly given final answer, or a long text with redundant restates and loop checks. Its specific length and content structure change dynamically with the model's generation state.

[0035] There are several ways to obtain the sampling group of the current policy model. One possible way is to obtain the sampling group of the current policy model from the already constructed sample library.

[0036] In this approach, a fixed-multiplier oversampling method can be used: for each sample instruction, a fixed number of samples are generated, and the current policy model is used to sample and generate the corresponding number of candidate outputs in parallel, directly forming the initial sampling group. Then, a reward-based selection process is used to retain valid samples. A dynamic adaptive sampling method dynamically adjusts the oversampling multiplier based on the proportion of valid groups in historical training rounds; if the historical effectiveness rate is low, the sampling number is automatically increased, and vice versa, to maintain a stable number of valid candidate outputs in each group. A reward threshold filtering method first performs large-scale coarse sampling to generate a large number of candidate outputs, then uses a preset reward function to score all candidates, retaining only candidate outputs with reward scores higher than a specific threshold and their corresponding instructions to form a sampling group, eliminating low-quality or erroneous samples. A diversity-constrained sampling method introduces random control strategies such as kernel sampling or temperature regulation when generating candidate outputs to ensure that the generated candidate outputs have significant differences in word distribution, inference path length, or logical structure, avoiding homogenization of samples within the sampling group. The progressive sampling method for courses involves classifying tasks by difficulty. Simple tasks are sampled in small quantities, while high-difficulty tasks undergo multiple rounds of iterative sampling or the number of samples per sample is increased. Sampling groups of different sizes are dynamically constructed based on the model's current capability level. This method may combine one or more of these sampling methods, depending on the specific circumstances. The embodiments in this specification do not limit this approach.

[0037] Another possible approach is to generate and construct a sampling group for the current policy model based on prompt words during the training process, according to the requirements.

[0038] This step obtains a sampling group containing sample instructions and corresponding multi-word candidate outputs, providing a data source with original generation diversity and distribution characteristics for subsequent filtering and truncation operations based on the number of words. This enables the model to construct effective training samples based on inference paths or response forms of different lengths, thereby supporting gradient updates and length control optimization of the policy model during reinforcement learning and ensuring that the training process can handle various forms of generated results, from concise answers to lengthy thought chains.

[0039] Step 104: Based on the number of lexical units in at least one candidate output, determine the candidate output to be truncated from at least one candidate output.

[0040] The candidate output to be truncated is the original object used to perform the truncation operation to generate valid training samples that meet the length limit. It can be a response sequence selected from the sampling group based on the number of lexical units and whose length exceeds a preset threshold. The candidate output to be truncated can be presented as a thought chain containing lengthy reasoning steps, a paragraph with repeated verification steps, or a code snippet with redundant comments and explanations, etc. For example, long texts with a lot of intermediate derivation processes generated for complex mathematical problems, detailed descriptions with repeated self-correction generated for open-ended problems, or program code with documentation output when responding to programming instructions. Its specific characteristic is that the total number of lexical units exceeds the preset range and needs to be processed before it can be included in the subsequent reinforcement learning process.

[0041] There are several ways to determine candidate outputs to be truncated. One possible way is to set a preset length threshold. If the number of words in the candidate output is greater than the preset threshold, the candidate output is determined to be a candidate output to be truncated.

[0042] Another possible approach is to determine a dynamic length threshold based on the number of tokens in at least one candidate output, and then determine the candidate output to be truncated from at least one candidate output according to the dynamic length threshold.

[0043] Another possible approach is to use a large model to determine the number of redundant words in the candidate output. When the number of redundant words is greater than a preset threshold, or when the number of redundant words and the number of candidate output words reach a certain ratio, the candidate output is determined as the candidate output to be truncated.

[0044] This step accurately identifies the target to be truncated from the candidate output based on the number of lexical units, enabling automatic location and differentiation of excessively long response sequences or texts containing redundant content. This allows subsequent processing to perform targeted length truncation operations, thereby eliminating resource waste and training noise caused by excessively long outputs while preserving key semantic information. This lays the foundation for building high-quality and effective sampling groups that conform to preset length specifications, thereby improving the policy model's ability to control the generated length and training efficiency during reinforcement learning.

[0045] Step 106: Truncate multiple words in the candidate output to obtain the truncated candidate output.

[0046] Truncation is a processing operation that removes or shortens the ends of text sequences that exceed a preset length limit. It is used to adjust overly long candidate outputs into effective samples that meet training specifications, thereby optimizing resource utilization and model convergence. This operation can manifest as cutting off the tail of lengthy inference steps, deleting paragraphs with repeated verifications, or removing redundant comment code, etc. For example, it can perform end truncation on mathematical solutions containing a large number of intermediate derivations, compress the length of detailed descriptions that repeatedly self-correct, or remove suffixes from program code with accompanying documentation. The specific execution method changes dynamically with the word distribution and threshold setting.

[0047] Truncated candidate outputs are text sequences that meet length limits, generated after performing lexical removal or shortening on the candidate outputs to be truncated. They are used as valid samples that meet training specifications in the reinforcement learning process to support gradient updates and optimization of the policy model. Truncated candidate outputs can be presented as thought chains that retain the core reasoning logic, concise statements after removing duplicate checks, or code snippets with redundant comments removed, etc.

[0048] There are several ways to truncate multiple words in the candidate output to obtain the truncated candidate output. One possible way is to truncate multiple words at the end of the candidate output to obtain the truncated candidate output.

[0049] Another possible approach is to identify multiple redundant words, truncate the redundant words in the candidate output to be truncated, and obtain the truncated candidate output.

[0050] This step transforms the ultra-long sequence into a truncated candidate output that conforms to the preset length specification by performing tail removal or redundant word removal operations on the identified candidate output to be truncated. This allows samples that could not be directly used for training due to exceeding the limit to retain their core semantics and be included in the effective sampling group. In this way, while avoiding resource waste and gradient update anomalies, the diversity and integrity of training data in the reinforcement learning process are maintained, and the adaptability and convergence efficiency of the policy model under different generation length scenarios are improved.

[0051] Step 108: Construct an effective sampling group based on the sample instructions and truncated candidate outputs.

[0052] The effective sample set is a filtered collection of samples that meets training requirements. It provides high-quality, long-stable question-answer pairs during reinforcement learning to support stable calculation of policy gradients and model parameter updates. The effective sample set may include truncated thought chains of inference samples, a streamlined dialogue library with redundant arguments removed, or training packages with unnecessary annotations removed, etc. The effective sample set can include sample instructions, truncated candidate outputs, and other candidate outputs from the sample set besides the truncated candidate outputs.

[0053] Based on sample instructions and truncated candidate outputs, an effective sampling group can be constructed. One possible approach is to construct the effective sampling group using sample instructions and truncated candidate outputs as question-answer pairs.

[0054] Another possible approach is to construct an effective sampling group using sample instructions, truncated candidate outputs, and untruncated candidate outputs from the candidate outputs.

[0055] This step assembles sample instructions and truncated candidate outputs into a structurally unified effective sampling group, transforming the generated data that was originally difficult to use directly due to excessive length into high-quality question-answer pairs that conform to training specifications. This provides a continuous and consistent optimization signal for policy gradient calculation while ensuring the stability of input and output lengths, effectively avoiding length drift caused by sequence length fluctuations and improving the convergence speed and generalization ability of the model when dealing with long text generation tasks.

[0056] Step 110: Based on the effective sampling group, perform reinforcement learning on the current policy model to obtain the target policy model.

[0057] Reinforcement learning is a machine learning paradigm that maximizes cumulative rewards by interacting with the environment and adjusting policies based on feedback signals. It guides models to autonomously explore optimal behavioral sequences and achieve adaptive optimization for complex tasks in scenarios lacking explicit supervised labels. Reinforcement learning can manifest as alignment training based on human preference ranking, decision evolution dependent on game outcomes, or a generative process that self-corrects based on code execution success rates. Examples include updating weights for multiple versions of responses output by a large language model based on human scoring, and adjusting operation probabilities through backpropagation based on game outcomes. Reinforcement learning paradigms can include GRPO and PPO, among others.

[0058] The target policy model is the final policy network formed through reinforcement learning training iterations. It typically possesses specific task execution capabilities and optimization characteristics. During the deployment phase, it receives the environmental state and outputs the optimal action sequence to solve complex decision-making problems or generate high-quality content. The target policy model can take the form of a dialogue assistant aligned with human feedback, a game agent evolved based on game win-loss rewards, or a programming generator that self-corrects based on code test pass rates. Examples include reasoning models that can output logically rigorous and length-compliant answer sequences in long text reasoning tasks, dialogue models that generate concise, accurate, and non-redundant responses in open-domain question-answering scenarios, or code analysis models that produce runnable program snippets that pass test cases in automated coding environments.

[0059] Based on the effective sampling set, reinforcement learning is performed on the current policy model to obtain the target policy model. One possible approach is to perform group-relative policy optimization on the current policy model based on the effective sampling set to obtain the target policy model.

[0060] Another possible approach is to perform near-end policy optimization on the current policy model based on the effective sample set to obtain the target policy model.

[0061] In practical applications, one possible implementation is as follows: The training cue set is denoted as:

[0062] in, The question answer represents a single training cue word or question.

[0063] The training step index is denoted as the question answer. The number of valid target groups is recorded as the number of question answers. This indicates the number of valid cue groups expected to be used for each parameter update. This is the set of sampling prompts for the u-th step. (Answer to the question) The number of prompts for the actual sampling step is recorded as the number of questions answered. The corresponding set of hints is denoted as:

[0064] Each prompt word corresponds to a sampled question answer. Candidate outputs, The question-answer combination is denoted as the group size. The current policy model is denoted as the question-answer combination. The old strategy or sampling strategy is denoted as the question answer. .

[0065] Answering questions from the training cue set Mid-sampling yields the set of prompts for the current training step and answers the questions. .

[0066] Answer each prompt question Answer the question based on the current strategy. Sample generation of question answers The candidate outputs are divided into a sampling group:

[0067] Calculate the task reward for each candidate output, where, For the first Candidate output corresponding prompts The reward value, ( ) represents the preset reward function:

[0068] The sampled groups are judged to be all correct, and the length mean and standard deviation of the groups that meet the conditions are calculated.

[0069] Dynamic truncation is performed only on the length of outlier candidate outputs in the all-correct group to obtain the truncated output question answer. .

[0070] Answer to the question about recalculating rewards after truncation. , and filter valid groups based on whether the variance of the reward within the group is non-zero.

[0071] Based on the effective groups, a reinforcement learning policy update is performed once. The preferred method is Question Response GRPO, but it can also be replaced by Question Response PPO, GSPO, GMPO, or other group relative policy optimization methods.

[0072] For any prompt word Current strategy The within-group response was obtained through sampling;

[0073] The relative advantage of a group is defined as:

[0074] in, It is a numerically stable term; it can also be taken as a term when the standard deviation of the within-group reward is strictly greater than zero. .

[0075] Define the lexical importance ratio:

[0076] The objective of the relative strategy optimization can then be written as:

[0077] in, Indicates will Cut off to interval Inside.

[0078] This specification's embodiments preserve the original generative diversity and validation foundation of the model by acquiring a sampling group containing sample instructions and multi-term candidate outputs. Based on the number of terms in the candidate outputs, the truncated candidate outputs are directly selected and truncated. Without introducing additional length penalties or modifying the reward formula, this avoids gradient conflicts between shortening length and maintaining accuracy from the data source, preventing training instability caused by reward coupling. An effective sampling group is constructed using sample instructions and truncated candidate outputs, forcing the samples input to the reinforcement learning stage to converge to a shorter length distribution while retaining the correct inference logic, suppressing length drift in question answers. Reinforcement learning of the current policy model based on this effective sampling group avoids the gradient vanishing problem caused by an increased proportion of all-correct samples, significantly improving sampling efficiency and obtaining a target policy model that combines high accuracy and low inference cost.

[0079] In one possible embodiment of this specification, before determining the candidate output to be truncated from at least one candidate output based on the number of lexical terms in at least one candidate output, the method further includes: The number of lexical units in at least one candidate output is counted to obtain the lexical unit count statistics; Based on the number of lexical terms in at least one candidate output, determine the candidate output to be truncated from at least one candidate output, including: Based on the lexical number statistics, determine the candidate output to be truncated from at least one candidate output.

[0080] Statistics is the process of quantifying the numerical features of the number of lexical units in candidate outputs. It is used to evaluate the length distribution patterns and fluctuation range of generated content to assist in screening or optimization strategies. Statistics may include counting lexical units and calculating statistical values ​​such as mean, variance, and standard deviation. This process can be presented as measuring the number of lexical units and statistical values ​​of multiple candidate responses generated in a single inference, or comparing and analyzing the number of lexical units or statistical values ​​of candidate outputs under different sampling strategies. For example, it can be an operation to calculate the average number of lexical units and the length difference of multiple response schemes generated by a large language model.

[0081] The lexical count is a numerical result obtained by counting and quantifying the number of lexical units in the candidate output. It is used to measure the content scale, evaluate the consumption of computing resources, and optimize the generation strategy. This value can be presented as the total number of lexical units generated in a single batch, the mean and variance among multiple candidate schemes, standard deviation statistics, etc.

[0082] There are several ways to count the number of words in at least one candidate output. One possible way is to determine the number of words in at least one candidate output, and then calculate the mean, variance, and standard deviation of the number of words in at least one candidate output to obtain the count of words.

[0083] Based on the lexical count statistics, one possible way to determine the candidate output to be truncated from at least one candidate output is as follows: Outlier candidate outputs are determined based on the mean, and these outlier candidate outputs are used as candidate outputs to be truncated.

[0084] Another possible approach is to determine the dynamic truncation threshold based on the mean and variance, and then determine the candidate output to be truncated based on the dynamic truncation threshold and the number of tokens.

[0085] This specification's embodiments introduce statistical analysis of the number of candidate output tokens, using quantitative indicators such as mean, variance, and standard deviation to dynamically identify length distribution characteristics. This enables adaptive filtering of outlier candidates that deviate from the normal length range or precise location of objects to be truncated based on dynamic thresholds. This effectively avoids the problems of incomplete semantic truncation or resource waste caused by fixed truncation strategies, improves the flexibility and accuracy of output content length control in variable generation scenarios, and ensures that the final generated results maintain high logical coherence and information integrity while meeting system resource constraints.

[0086] One possible embodiment of this specification involves truncating multiple words in the candidate output to obtain truncated candidate output, including: Based on the lexical count statistics, multiple lexical units of the candidate output to be truncated are truncated to obtain the truncated candidate output.

[0087] The process involves truncating multiple words in the candidate output and removing several words that exceed a preset length limit. This operation can be presented as deleting tail segments based on the word count, dynamically adjusting the word removal range based on semantic boundaries, or eliminating redundant words based on the remaining budget ratio. For example, when generating long responses using a large language model, it can force the deletion of words that exceed the context window at the end, or in a code auto-completion scenario, it can prune words that exceed the number of lines displayed in the editor.

[0088] Based on the lexical count statistics, multiple lexical units in the candidate output to be truncated are truncated to obtain truncated candidate output. One possible approach is to determine the number of truncated lexical units based on the lexical count statistics, truncate lexical units at preset positions in the candidate output to be truncated, and obtain truncated candidate output.

[0089] Another possible approach is to determine the number of words to be retained based on the word count statistics, truncate the candidate output to be truncated to obtain the retained words, and use the retained words as the candidate output to be truncated.

[0090] In practical applications, one possible implementation is as follows: Perform outlier detection and truncation on the all-correct group. If a candidate output satisfies:

[0091] Then cut it into the first part One word element; otherwise, it remains unchanged. Therefore:

[0092] This specification's embodiments quantify the length distribution characteristics of candidate outputs to be truncated by utilizing the statistical value of the number of lexical units, and then dynamically truncate multiple lexical units of the candidate outputs to be truncated. This achieves the removal of redundant lexical units, effectively avoiding semantic breaks or uneven resource allocation caused by fixed-length truncation, and improving the adaptive capability and accuracy of output content scale control in different generation scenarios. It ensures that the final truncated candidate outputs strictly comply with system capacity constraints while maintaining the logical coherence and information integrity of the text to the greatest extent.

[0093] In one possible embodiment of this specification, the lexical number statistics include the mean lexical number and the standard deviation of the lexical number; Before truncating multiple words in the candidate output based on the word count statistics to obtain the truncated candidate output, the process also includes: The truncation threshold for candidate outputs to be truncated is determined based on the mean number of lexical units and the standard deviation of the number of lexical units. Based on the lexical count statistics, multiple lexical units in the candidate output to be truncated are truncated to obtain truncated candidate output, including: Based on the lexical truncation threshold, multiple lexical units of the candidate output to be truncated are truncated to obtain the truncated candidate output.

[0094] The mean number of lexical units is the arithmetic mean obtained by summing the number of lexical units contained in each sequence of at least one candidate output and dividing it by the total number of candidate outputs. It is used to measure the central tendency of the length of generated content and to provide a benchmark for assessing the overall size or calculating outliers.

[0095] The standard deviation of the number of tokens is the square root of the average of the squares of the differences between the number of tokens in each sequence of at least one candidate output and their arithmetic mean. It is used to quantify the dispersion of the length distribution of generated content and to help identify the length fluctuation range or determine outlier candidate outputs.

[0096] The lexical truncation threshold is a length limit for performing lexical removal operations on candidate outputs. It can be used to determine which candidate outputs can be truncated, and the length to retain after truncating the candidate output. The lexical truncation threshold can control the maximum size of the generated content.

[0097] There are several ways to determine the truncation threshold of candidate outputs to be truncated based on the mean and standard deviation of the number of lexical units. One possible way is: The truncation threshold for candidate outputs to be truncated is determined by linearly weighting the mean number of lexical units and the standard deviation of the number of lexical units.

[0098] Another possible approach is to calculate the ratio based on the mean number of lexical units and the standard deviation of the number of lexical units, and then map the obtained lexical unit ratio to a preset truncation length to determine the truncation threshold of the candidate output to be truncated.

[0099] Based on the truncation threshold, multiple words in the candidate output to be truncated are truncated. There are several ways to obtain the truncated candidate output. One possible way is: The number of truncated words is determined based on the word truncation threshold. Based on the number of truncated words, multiple words in the candidate output to be truncated are truncated to obtain the candidate output.

[0100] Another possible approach is: The number of words to be retained is determined based on the word truncation threshold. Based on the number of words to be retained, multiple words in the candidate output to be truncated are truncated to obtain the truncated candidate output.

[0101] In practical applications, in one possible scenario, the dynamic truncation threshold is defined as:

[0102] in, Reflects the average length of the current group. Reflects the degree of dispersion in length within a group. These are pre-defined hyperparameters. The threshold value changes with the difficulty of the problem, the policy state, and the training phase. It changes automatically, eliminating the need to pre-set a uniform fixed length limit for all tasks.

[0103] Minimum cutoff amplitude mechanism: In one possible scenario, to avoid frequent and minimal perturbations in the later stages of training when the length has converged, the candidate output can satisfy:

[0104] Only then is the truncation executed, among which, It is the candidate output length. It is a dynamic truncation threshold. This is the minimum truncation range, used to avoid the truncated candidate output being too short.

[0105] The embodiments in this specification dynamically construct a truncation threshold based on the mean and standard deviation of the number of tokens, accurately identify and truncate extremely long outputs in the sample group, and use the processed truncated candidate outputs as high-quality training samples to directly participate in reinforcement learning training. This effectively suppresses the length drift phenomenon generated by the model during training, significantly reduces inference costs while retaining the necessary long-range inference capabilities, thereby greatly improving the stability of the training process and the efficiency of sample utilization.

[0106] One possible embodiment of this specification, based on a lexical number statistical value, determines candidate outputs to be truncated from at least one candidate output, including: Based on the number of tokens in at least one candidate output and the token truncation threshold, determine the truncation amount of at least one candidate output; Based on the truncation amount of at least one candidate output and the preset minimum number of truncation terms, determine the candidate output to be truncated from at least one candidate output.

[0107] The truncation amount is the number of words removed from the candidate output. It can be the portion of the candidate output that exceeds the word truncation threshold and meets the minimum truncation magnitude constraint. It is used to determine the number of words to be truncated during the truncation process. The determination method may include one or more of the following: directly calculating the difference between the number of words and the word truncation threshold, scaling the truncation length based on a preset ratio, or combining the nonlinear mapping result of the dynamic adjustment coefficient.

[0108] The preset minimum number of truncated tokens is the lowest limit of the number of tokens whose candidate output length exceeds a dynamic threshold before triggering the truncation operation. This limit is used to filter out minor length deviations caused by statistical fluctuations, ensuring that pruning is performed only on significantly redundant outliers and maintaining the semantic value of the truncated content. The preset minimum number of truncated tokens can be an empirically fixed constant, a variable derived from the statistical values ​​and standard deviations of candidate output tokens, or an adaptive parameter that is dynamically adjusted during the training phase.

[0109] There are several ways to determine the truncation amount of at least one candidate output based on the number of lexical units in at least one candidate output and the lexical truncation threshold. One possible way is to directly calculate the difference between the total number of lexical units in the candidate output and the lexical truncation threshold, and determine the difference as the truncation amount.

[0110] Another possible approach is to construct a nonlinear function based on the ratio of the total sequence length to the word truncation threshold, and then generate the truncation amount based on the nonlinear function.

[0111] There are several ways to determine the candidate output to be truncated from at least one candidate output based on the truncation amount of at least one candidate output and the preset minimum number of truncation words. One possible way is to compare the truncation amount of the candidate output with the preset minimum number of truncation words to determine the candidate output to be truncated from at least one candidate output.

[0112] Another possible approach is to compare the truncation amount of the candidate outputs with the preset minimum number of truncation terms, and determine the candidate output with a truncation amount greater than the preset minimum number of truncation terms from at least one candidate output as the candidate output to be truncated.

[0113] The embodiments in this specification, by combining a dual judgment mechanism of word truncation threshold and preset minimum number of truncation words, can accurately identify and filter out significantly long candidate output sequences. While effectively filtering out minor length deviations caused by statistical fluctuations, it ensures that pruning is only performed on outlier samples that truly need processing. Thus, while suppressing model-generated length drift and improving the quality of training samples, it also takes into account the preservation of semantic integrity and the reasonable allocation of computing resources, avoiding the damage to content value caused by invalid truncation.

[0114] One possible embodiment of this specification involves obtaining a sample group for the current policy model, including: Get sample instructions; Using the current policy model, generate at least one candidate output based on the sample instructions; A reward score is assigned to at least one candidate output based on a preset reward function to obtain the reward function for at least one candidate output; If the reward function of at least one candidate output is greater than the preset reward threshold, the sample instruction and at least one candidate output generated by the current policy model according to the sample instruction are determined as the sampling group of the current policy model.

[0115] A pre-defined reward function is a mathematical mapping rule used to quantitatively evaluate the quality of an agent's behavior or output. It is typically pre-set during reinforcement learning or optimization to guide the model towards the desired goal and adjust the policy update direction through numerical feedback signals to optimize long-term cumulative returns. Its construction can take various forms, including linear scoring based on task completion, weighted summation combining multiple objective weights, phased judgments incorporating sparse rewards, or complex nonlinear models integrating human preference ranking, etc. These can flexibly drive agents to learn expected behavioral patterns in different scenarios.

[0116] Reward scoring is the process of numerically measuring agent behavior based on pre-defined evaluation rules. It can include scoring of behavior, generated content, or state transitions, and is used to transform abstract quality standards into computable feedback signals to guide policy optimization or model parameter updates. Implementation paths for reward scoring can include precise counting based on rule matching, semantic similarity inference using pre-trained models, relative scoring combined with human preference ranking, or comprehensive judgment incorporating multi-dimensional weighting factors, etc.

[0117] The reward function for the candidate output is a quantitative score obtained by scoring the candidate output, which is used to measure the quality of different candidate solutions and serve as the basis for subsequent strategy optimization or sequence selection decisions.

[0118] The preset reward threshold is a numerical limit used in the evaluation process to determine whether candidate outputs meet the criteria. It is used to convert continuous reward scores into a screening decision for candidate outputs, ensuring the correct execution of subsequent processing logic. The preset reward threshold can be set in one or more ways, such as quantile truncation based on historical data statistics, adaptive boundaries dynamically adjusted according to task difficulty, weighted critical values ​​combining multiple objectives, or fixed benchmarks referencing human expert scoring standards. For example, it can be set so that the corresponding sampling set is only used as a sampling group when all candidate output scores are higher than 0.85.

[0119] There are several ways to obtain the reward function for at least one candidate output by scoring at least one candidate output based on a preset reward function. One possible way is to score at least one candidate output based on a preset answer standard to obtain the reward function for at least one candidate output.

[0120] Another possible approach is to score at least one candidate output based on a pre-defined reward model to obtain a reward function for at least one candidate output.

[0121] One possible implementation in practical applications is as follows: For each prompt word The group type is determined for the corresponding sampling group. If:

[0122] or:

[0123] If the group is completely correct, it is considered a group with no completely correct answers; otherwise, it is considered a group with no completely correct answers. This refers to the reward threshold, which can also be understood as a manually set qualification line, such as a reward ≥ This value is acceptable.

[0124] Zero advantage property of all correct sets: When the reward is binary and all responses are correct, the within-group reward variance is zero, therefore the group relative advantage satisfies:

[0125] This means that all-correct sets no longer provide an effective gradient for "correctness improvement" under the original GRPO update. Therefore, significantly excessively long responses are more likely to be considered redundant rather than exploratory actions necessary to solve the puzzle. This rule avoids imposing length constraints on puzzle sets that are still being explored, thus preserving the ability to reason along long chains.

[0126] The embodiments in this specification utilize a preset reward function to perform multi-dimensional quantitative scoring on the candidate outputs generated by the policy model, and introduce a preset reward threshold as a strict quality filtering gate. Only those combinations of instructions and outputs whose scores exceed the threshold are established as sampling groups, thereby effectively eliminating low-quality or unexpected sample noise and avoiding constraints on the exploration of unsolved problems. This ensures that the data entering the subsequent training or optimization stages has high consistency and reliability, thereby improving the convergence efficiency, stability, and final task performance of the policy model in the reinforcement learning process.

[0127] In one possible embodiment of this specification, before constructing a valid sample set based on sample instructions and truncated candidate outputs, the following is also included: The truncated candidate outputs are scored based on a preset reward function to obtain the reward function for the truncated candidate outputs; Calculate the standard deviation of the reward function for the truncated candidate outputs to determine the reward standard deviation corresponding to the reward function for the truncated candidate outputs; Based on the reward standard deviation and the preset reward standard deviation threshold, truncated candidate outputs are determined from the truncated candidate outputs to construct effective sampling groups.

[0128] In the embodiments of this specification, before constructing an effective sampling group based on sample instructions and truncated candidate outputs, it is necessary to re-score the truncated candidate outputs using a preset reward function to obtain the reward function of the truncated candidate outputs. This reward function of the truncated candidate outputs is used together with the rewards of other candidate outputs obtained previously to determine the reward standard deviation, so as to determine the reward distribution of each candidate output and the truncated candidate output in the sampling group.

[0129] The preset reward standard deviation threshold is a numerical limit pre-set in the data screening process to measure the dispersion of a set of candidate output reward scores. Possible scenarios include setting it to consider a data set as valid only if the recalculated reward function score fluctuates within a range exceeding 0.1, or retaining the data set if the reward standard deviation is greater than a specific value. This ensures both the quality and the richness and robustness of the model generation.

[0130] In one possible scenario, the logic of the above embodiment is to recalculate the reward for the truncated candidate output after truncation processing to obtain the corresponding recalculated reward function, calculate the standard deviation based on the reward function, and select effective groups with non-zero reward standard deviations to eliminate homogeneous sample sets that, although the mean meets the standard, lack diversity or discriminativeness, so as to ensure that subsequent policy optimization is based on data distribution with sufficient exploration space.

[0131] Based on the reward standard deviation and a preset reward standard deviation threshold, truncated candidate outputs for constructing effective sampling groups are determined from the truncated candidate outputs. One possible approach is to compare the reward standard deviation with the preset reward standard deviation threshold and determine the truncated candidate outputs whose reward standard deviation is greater than the preset reward standard deviation threshold from the truncated candidate outputs.

[0132] Another possible approach is to compare the reward standard deviation with a preset reward standard deviation threshold, and then determine the truncated candidate outputs from the truncated candidate outputs whose reward standard deviation is greater than the preset reward standard deviation threshold and is non-zero.

[0133] In practical applications, one possible implementation is as follows: Recalculate the reward for the truncated output:

[0134] Valid groups are selected based on the recalculated within-group reward variance. The set of valid groups is defined as follows:

[0135] in, To calculate the standard deviation of the object.

[0136] The embodiments in this specification introduce a reward recalculation and standard deviation verification mechanism for truncated candidate outputs before constructing an effective sampling group. By using a preset reward standard deviation threshold to perform discrete quantitative evaluation on the recalculated reward distribution, sample combinations with reward standard deviations greater than the threshold are accurately selected. This effectively eliminates homogeneous data that, although the mean meets the standard, lacks internal diversity or discrimination, ensuring that the final constructed sampling group has both high quality and sufficient exploration space. This, in turn, improves the convergence stability, generalization ability, and learning efficiency of the policy model in subsequent training.

[0137] One possible embodiment of this specification, in obtaining a target policy model by performing reinforcement learning on the current policy model based on an effective sampling set, further includes: The mean of the reward function for the truncated candidate output is calculated to determine the mean reward corresponding to the reward function for the truncated candidate output. Based on the effective sample set, reinforcement learning is performed on the current policy model to obtain the target policy model, including: Based on the reward function and mean reward of the truncated candidate outputs in the effective sampling group, determine the offset of the reward function of the truncated candidate outputs in the effective sampling group; Based on the reward function offset and the reward standard deviation, determine the dominant function for truncating candidate outputs in the effective sampling group; Based on the advantage function, the model parameters of the current policy model are adjusted to obtain the target policy model.

[0138] Mean calculation is the process of summing a set of numerical data and dividing by the number of data points to obtain the arithmetic mean. It is used to quantify the central tendency or overall level of a dataset, so as to quickly grasp the overall characteristics of a sample without having to pay attention to individual differences.

[0139] The reward mean is the arithmetic mean of a set of candidate outputs, used to characterize the overall performance level of the sampled group and determine the difference between each sample and the average level. For example, the average reward level of the group can be obtained by summing the validation scores of the five truncated inference paths under a sample instruction and dividing by 5, thus providing a numerical anchor for calculating the advantage value of each path and providing gradient update direction.

[0140] The reward function offset is the difference between the reward score of a single truncated candidate output and the mean reward of the sampled group. It quantifies the relative deviation of a particular output from the group average, eliminating baseline interference and highlighting individual differences when calculating the dominance function. The reward function offset can be understood as follows: for example, if an inference path scores 0.85 while the group average score is 0.70 in a given iteration, its offset is 0.15, indicating that the path outperforms the average, thus providing a positive guiding signal for subsequent parameter updates.

[0141] The advantage function measures the relative superiority of a particular output compared to the average level of the current sampled group. It can be obtained by subtracting the mean reward from the reward score of a single candidate output, then dividing by the sum of the standard deviation of the reward within the group and the numerical stability term. This helps eliminate baseline bias and reduce the variance of gradient estimation during policy optimization. Its applications include determining the direction of policy gradients in reinforcement learning, evaluating the value of actions in multi-armed slot machines, and ranking the quality of different responses to the same prompt word in generative models. For example, if an inference path scores 0.9 in a sample while the average score within the group is 0.7 and the standard deviation is 0.1, its advantage function value is 2.0, indicating that this path is significantly better than the average level, thus guiding the model parameters to adjust towards generating such high-quality responses.

[0142] In practical applications, the above process can be described as follows: Reward Calculation Module: Calculates the task reward for each candidate output.

[0143] For any prompt word Current strategy The sampling yielded the within-group responses of the sampled group:

[0144] The relative advantage of a group is defined as:

[0145] in, It is a numerically stable term; it can also be taken as a term when the standard deviation of the within-group reward is strictly greater than zero. .

[0146] There are several ways to obtain the target policy model by adjusting the model parameters of the current policy model based on the advantage function. One possible way is to construct a policy optimization objective based on the advantage function, and then adjust the model parameters of the current policy model based on the policy optimization objective to obtain the target policy model.

[0147] Another possible approach is to use the advantage function as a weighting coefficient to weight the log probability gradient corresponding to the candidate output generated by the current policy model, thereby obtaining the policy gradient estimate. Then, the model parameters of the current policy model are directly updated using the stochastic gradient ascent algorithm based on this gradient estimate until convergence is achieved to obtain the target policy model.

[0148] The embodiments in this specification effectively eliminate intra-group baseline bias and maintain the variance stability of gradient estimation by constructing a normalized advantage function calculation link based on the reward mean, offset, and standard deviation. This suppresses length drift in simple tasks while avoiding the gradient vanishing problem caused by all-correct samples, ultimately obtaining a target policy model that combines high inference accuracy with low computational cost.

[0149] In one possible embodiment of this specification, before adjusting the model parameters of the current strategy model to obtain the target strategy model, the following steps are also included: Determine the lexical covariance among multiple lexical units of the truncated candidate output in the valid sampling group; Based on the lexical covariance and a preset lexical covariance threshold, determine whether there are over-coupled lexical units among multiple lexical units; Based on the advantage function, the model parameters of the current policy model are adjusted to obtain the target policy model, including: If so, construct an over-coupling penalty based on the model parameters of the current strategy model and the adjusted model parameters of the current strategy model; Under the constraint of over-coupling penalty, the model parameters of the current policy model are adjusted based on the advantage function to obtain the target policy model.

[0150] Lexical covariance is a statistical measure of the degree of coordinated fluctuation in the probability distribution or feature representation of lexical units at different positions in a sequence. It is used to quantify the dependence and coupling tightness between adjacent or spaced lexical units to identify potential redundancy, repetition, or logical rigidity in the generation process. For example, in a generated text, if the t-th lexical unit "because" appears, the (t+1)-th lexical unit has a very high probability of being fixed as "therefore," and their fluctuations are highly synchronized, then the covariance value between these two lexical units is significantly high, indicating a strong over-coupling relationship between them. Regularization may be needed to break this mechanical association to improve the diversity of the generated text.

[0151] The preset lexical covariance threshold is a pre-defined numerical limit used to determine whether the degree of covariance between lexical terms exceeds the allowable range. It is used to distinguish between normal semantic dependencies and over-coupling states, so as to trigger the penalty mechanism for redundant associations in a timely manner during model optimization. The setting of the preset lexical covariance can be based on the statistical distribution of historical training data, the task's tolerance for generation diversity, and dynamically adjusted empirical coefficients, etc. For example, if the threshold is set to 0.85, when the covariance calculation result of two consecutive lexical terms is detected to be 0.92, it is determined that there is over-coupling between the two, thereby activating the constraint term to break this rigid fixed collocation pattern.

[0152] Overcoupled lexical units are combinations of multiple lexical units exhibiting abnormally strong dependencies. They are typically lexical units in a sequence whose degree of co-variation exceeds a preset limit. Overcoupled lexical units are used to identify potential mechanical repetition, logical rigidity, or lack of diversity in the generation process, triggering targeted penalty mechanisms during model training to break fixed collocation patterns. Overcoupled lexical units may appear in situations such as fixed patterns of adjacent word connections in autoregressive text generation, redundant loops of semantic units in multi-step inference paths, rigid nesting of specific grammatical structures in code generation, or unlimited reuse of response templates in dialogue systems. For example, in a generation task, if the probability of "secondly" immediately following "firstly" is abnormally close to 1 and their covariance far exceeds a threshold, then these two lexical units constitute overcoupled lexical units, indicating that the model has fallen into a deterministic transition trap lacking flexibility.

[0153] The overcoupling penalty is a constraint term calculated based on the abnormally high degree of co-variance between lexical units. It is typically introduced during model parameter updates to suppress mechanical repetition or logically rigid fixed collocation patterns in the generated sequence, thereby improving the diversity and flexibility of the output while maintaining semantic coherence. Its construction methods can include linear weighting, exponential amplification, dynamic decay adjustment, or orthogonal projection onto the gradient of the dominance function, etc. For example, when the lexical units "because" and "therefore" are detected to exhibit a strong correlation far exceeding a preset limit in multiple samplings, this penalty term will generate a significant negative gradient signal in the loss function, forcing the model parameters to adjust to reduce the probability of such deterministic shifts, thereby encouraging the exploration of more diverse causal expression paths.

[0154] Based on the lexical covariance and a preset lexical covariance threshold, one possible way to determine whether there are over-coupled lexical units among multiple lexical units is to compare the lexical covariance with the preset lexical covariance threshold and identify lexical units whose lexical covariance is greater than the preset lexical covariance threshold as over-coupled lexical units.

[0155] Another possible approach is to perform a ratio calculation between the lexical covariance and a preset lexical covariance threshold, and determine lexicals whose results are greater than the ratio threshold as over-coupled lexicals.

[0156] In a practical application, the above process can be implemented as follows: Policy entropy definition: To monitor whether the policy becomes prematurely deterministic, a policy entropy metric is introduced. Given a training set... Policy entropy is defined as the average word-level entropy on the training set:

[0157] in, The representation strategy model for prompt words The generated complete output sequence, Indicates the first Each word element.

[0158] In engineering implementation, the entropy of the entire vocabulary can also be calculated:

[0159] The average of the time dimension and the batch dimension is calculated and used as a real-time monitoring indicator.

[0160] Entropy change relationship under soft maximum strategy: Treat the language model as a soft maximum policy. For any state With action ,definition:

[0161] in, Representing state Next action Logit.

[0162] Under the first-order approximation, the entropy change between two adjacent policy steps satisfies:

[0163] in, Indicates the first State distribution under the step strategy.

[0164] If we use a normal policy gradient, then we have:

[0165] Therefore, we get:

[0166] If we use the natural policy gradient approximation, it can be written as:

[0167] The above relationship shows that when high-probability actions also have high advantages, the policy entropy will decrease faster, so it is necessary to impose directional constraints on a few high covariance positions.

[0168] Let the first Step by step, we collected Each output word, according to Number. The conditional prefix states of each lexical unit are denoted as follows: The corresponding word element is denoted as .

[0169] Define the log probability of this term as

[0170] Since the advantage of GRPO is primarily defined at the response level, therefore if the first The word element comes from the first For each response, its lexical-level advantage inherits the relative advantage of the group of responses:

[0171] Define the within-batch mean:

[0172] Accordingly, the first is defined The empirical covariance scores of each word are:

[0173] in, This is used to characterize the potential contribution of the terminology to the reduction of policy entropy in the current training batch.

[0174] High covariance lexicon set: For all Empirical covariance score of each word Sort by largest to smallest. Define the set of high covariance lexics as:

[0175] in, express Rank the scores in descending order of all scores. To select a proportion, and it usually satisfies:

[0176] Targeted KL regularization objective: Preferred only for sets KL penalty is applied to high covariance lexical units within the scope, instead of applying a uniform strengthening constraint to all lexical units.

[0177] Define the overall objective function as:

[0178] in, This represents the KL penalty coefficient.

[0179] The objective function means that: for ordinary words, the original GRPO objective is still updated; only for words with high covariance, the deviation between the current policy and the old policy is limited to suppress the policy from becoming too deterministic at a few key positions.

[0180] The embodiments in this specification calculate the lexical covariance of lexical units in the effective sampling group and select a set of lexical units with high covariance. This can accurately identify potential redundant dependency positions that cause the policy entropy to drop too quickly. Then, targeted KL regularization constraints are applied only to these specific lexical units instead of uniformly restricting the entire sequence. This fine-grained control mechanism retains the model's ability to explore and update ordinary lexical units while effectively suppressing the premature determination of policies caused by excessive coupling in a few key positions. As a result, the diversity and flexibility of the output are significantly improved while maintaining the semantic coherence of the generated content, and the model is prevented from falling into the trap of mechanical repetition or logical rigidity of local optima.

[0181] In one possible embodiment of this specification, before obtaining the sampling group of the current policy model, the method further includes: The proportion of historical valid groups is determined based on the number of historical sampling groups and the number of valid groups in the historical sampling groups; Based on the proportion of historical effective groups, determine the oversampling factor of the sampling group; Based on the oversampling factor, the sampling sample instruction and at least one candidate output generated by the current policy model according to the sample instruction are used to obtain the sampling group of the current policy model.

[0182] Historical sampling sets are datasets that have been collected in previous training iterations or evaluation cycles, containing sample instructions and their corresponding model-generated candidate outputs. They are used to statistically analyze historical effective sets to dynamically deduce the oversampling factor required for the current stage, thereby optimizing resource allocation and ensuring sufficient high-quality training data. For example, in a reinforcement learning training cycle, if the system records complete pairing data of all input prompts and model outputs from the previous 100 iterations, these accumulated data packets constitute historical sampling sets, which are used by the algorithm to analyze and determine how many times the sampling scale needs to be expanded in the next round to compensate for the expected invalid generation loss.

[0183] The historical effective set is the collection of sample instructions and candidate outputs from the historical sampling set that can be incorporated into the dominance function calculation or parameter update process.

[0184] The historical effective group ratio is the ratio of the number of effective sampled groups to the total number of historical sampled groups within past training cycles or evaluation windows. It quantifies the usability of the model-generated content and dynamically derives the required oversampling factor for subsequent stages, establishing an adaptive balance between resource consumption and data quality. For example, if 500 candidate output batches were collected in the past ten iterations, and only 350 passed the length limit and logical consistency checks, the calculated O7 value is the historical effective group ratio. This metric directly guides the algorithm in determining how many times to increase the sampling scale in the next round to compensate for expected invalid losses.

[0185] The oversampling factor is a numerical coefficient derived from the proportion of historical effective sets. It is used to amplify the initial sample command acquisition scale. It is used to pre-increase the sampling base in case some invalid data is expected, ensuring that a sufficient number of effective data sets are obtained to meet training requirements. For example, if statistics show that only half of the outputs in the past meet the quality standard, the system sets the oversampling factor to 2. This means that for each target command, the model needs to be pre-driven to generate twice the number of candidate results, so that after filtering, the desired number of effective samples can still be retained for subsequent model optimization.

[0186] In practical applications, one possible implementation of the above process is as follows: Definition of effective group proportion: No. The set of effective steps is defined as:

[0187] The corresponding effective group ratio is defined as:

[0188] Historical window statistics: Define the historical effective group ratio window as follows:

[0189] Its mean and standard deviation are defined as follows:

[0190] Oversampling factor prediction formula: To ensure that the number of valid groups obtained in the next step is as close as possible to the target value. The prediction oversampling factor is defined as:

[0191] Therefore, the first The actual number of samples collected was:

[0192] In this formula, Used to compensate for batch reduction caused by historical average efficiency. Used to provide a safety margin for fluctuations, in order to reduce the shortage of effective groups due to fluctuations in efficiency.

[0193] Window update rules: When the After the first step is completed, calculate the current proportion of valid groups.

[0194] Update the history window again:

[0195] in, This indicates appending an element to the end of the sequence. This indicates that only the most recent ones will be retained. Each element.

[0196] Engineering constraints: To avoid excessive sampling in extreme cases, an upper limit can be set for the oversampling factor:

[0197] in, This is a preset constant.

[0198] The embodiments in this specification dynamically derive the oversampling factor by utilizing the proportion of effective groups in historical sampling data, and pre-amplify the sampling scale of the sample command before obtaining the sampling group of the current strategy model. This achieves automatic compensation for data loss in the event of expected invalid generation, thereby ensuring that the final candidate output set contains a sufficient amount of high-quality training data. This effectively avoids the problem of resource waste or insufficient effective samples caused by blindly fixing the sampling quantity, and improves the resource allocation efficiency and data acquisition stability during the model training process.

[0199] One possible embodiment of this specification includes a method that further comprises: Obtain intermediate model training data, which includes at least one of the following: truncation ratio, average truncation magnitude, average number of tokens, policy entropy, average reward function, and oversampling factor. Model training logs are generated based on intermediate data from model training.

[0200] The truncation ratio is the ratio of the number of candidate outputs that are forcibly terminated by the system or exceed the length limit to the total number of candidate outputs. It is used to quantify the number of candidate outputs that are truncated due to excessive word count during the model generation process. For example, if a batch processing generates a total of one thousand candidate answers, of which one hundred and fifty are truncated by the system because they reach the preset length threshold, then the calculated value of 0.15 is the truncation ratio for that step. This metric helps monitor whether the model tends to generate excessively long content, thereby affecting training efficiency.

[0201] The average truncation magnitude is the average of the candidate outputs of the truncated responses. It is used to assess the average severity of the model-generated content exceeding the limits and to help adjust the length penalty strategy or buffer size. For example, if fifty truncated output samples are recorded in a training session, and it is statistically shown that each of these samples was truncated by an average of twenty tokens, then this value is the average truncation magnitude.

[0202] The average number of tokens is the average of the total number of tokens in the generated response divided by the candidate outputs. It is used to measure the average length of a single generation by the model and to help monitor for length drift or issues with generation that is too short or too long. For example, if two hundred model outputs are collected in a single sampling, and the total number of tokens after word segmentation is eight thousand, then the calculated number of tokens is forty.

[0203] Policy entropy is a measure of uncertainty calculated based on the probability distribution of the current policy model's output. It reflects the degree of exploration by the model during the decision-making process and helps determine whether policy collapse or overconvergence has occurred. For example, if a model chooses only one specific output with a very high probability and ignores other reasonable options when faced with a certain input, its policy entropy value will decrease significantly, indicating a potential risk of loss of diversity. This requires adjusting the temperature parameter or adding a regularization term to restore the exploration capability.

[0204] The average reward function is the arithmetic mean of the rewards obtained from all generated responses. It is used to comprehensively evaluate the overall performance of the current policy model and help determine whether the training direction is correct. For example, in a reinforcement learning update, the system scores 300 model outputs, and the total reward is 450. The calculated 15 is the average reward function value. This value can be used as an indicator to guide the direction and magnitude of policy gradient updates, ensuring that the model continuously moves towards high-reward behaviors.

[0205] One possible way to obtain intermediate training data for the model is to embed a lightweight counter and accumulator to calculate and obtain the intermediate training data.

[0206] Another possible approach is to temporarily store the complete metadata of all samples in the current batch in a high-speed buffer or in-memory database, and then process and obtain intermediate data for model training.

[0207] In practical applications, one possible implementation of the above process is as follows: To support training stability monitoring and anomaly backtracking, this invention records the following metrics: Cutoff ratio:

[0208] in, For the first The number of responses that were truncated. For the first Total number of responses per step.

[0209] Average response length:

[0210] Average reward:

[0211] Average cutoff amplitude:

[0212] Strategy Entropy:

[0213] in, It can be the current monitoring batch or a fixed set of verification prompts.

[0214] Valid group ratio:

[0215] Oversampling factor:

[0216] Through joint monitoring , , , , and It can identify problems such as length drift, over-truncation, entropy collapse, or abnormal sampling efficiency.

[0217] This specification's embodiments construct a comprehensive monitoring system that fully reflects generation quality, resource consumption, exploration level, and sampling efficiency by real-time collection and integration of multi-dimensional intermediate model training data, including truncation ratio, average truncation amplitude, average number of tokens, policy entropy, average reward function, and oversampling factor. This enables the system to not only accurately quantify the degree of data loss and the risk of model output diversity caused by length limitations, but also dynamically adjust the sampling strategy based on the proportion of historical effective groups to compensate for invalid losses. Thus, while generating structured training logs, it provides intuitive data support for identifying key issues such as length drift, policy collapse, or abnormal sampling efficiency, effectively improving the transparency of the model training process, fault backtracking capability, and adaptive optimization level.

[0218] In one possible embodiment of this specification, after generating model training logs based on intermediate model training data, the method further includes: Monitor the model training logs to determine the current output status of the policy model; If the output state of the current policy model is abnormal, backtrack the model training log, adjust the current policy model based on the backtracking results, and obtain the adjusted current policy model. And / or, if the output state of the current policy model is abnormal, obtain the rollback node of the current policy model, roll back the current policy model based on the rollback node, and obtain the rolled-back current policy model, where the rollback node is the previous version of the current policy model recorded by the checkpoint.

[0219] The model output state is a comprehensive set of results generated by the generative model after a specific inference or training step, which includes text sequences, termination markers, probability distributions, and length information. It is used to characterize the model's response to the current input instruction, content compliance, and confidence level, and to assist in subsequent data filtering, reward calculation, or policy updates.

[0220] Output state anomalies refer to response results that deviate from the expected distribution, violate constraints, or exhibit atypical statistical characteristics during the model generation process. They are used to identify potential risks such as policy collapse, resource overflow, logic failure, and loss of diversity during the training or inference phases, and to trigger corresponding intervention mechanisms or parameter adjustments. Output state anomalies may manifest as single-mode output due to policy entropy values ​​approaching zero, frequent truncation events triggered by exceeding the maximum word limit, low-quality sampling batches reflected by reward function scores consistently below the threshold, or garbled character combinations caused by unstable gradient updates, etc.

[0221] Backtracking is the process of reversing the flow of training logs to retrieve and extract historical intermediate data, parameter snapshots, and environmental feedback records. Backtracking can be performed based on time indexes or event markers. It is used to pinpoint the root causes of policy performance degradation or decreased generation quality and provides data support for model parameter correction, hyperparameter reconfiguration, or training path switching. For example, when the system detects that the current model is frequently outputting repetitive content, it can backtrack to the checkpoint before the entropy value began to decrease significantly, read the temperature parameter settings and optimizer state at that time, and thus specifically restore exploration capabilities, ensuring that the training process can quickly recover from abnormal states and maintain the expected convergence direction.

[0222] Rollback is the process of restoring model parameters, optimizer state, and associated configurations to a previously saved stable version based on preset checkpoint records. It's used to quickly eliminate negative impacts such as policy collapse, reward hacking, or gradient explosion introduced by recent training iterations, and to bring the model back to a reliable performance baseline. For example, when monitoring data shows a sharp drop in the average reward value across multiple consecutive batches and the policy entropy approaches zero, the system can automatically load the model weight file from the previous checkpoint record, discarding the most recently updated gradient information. This allows the model to skip unstable training intervals and continue subsequent fine-tuning or exploration based on its previous good state, ensuring the robustness and continuity of the training process.

[0223] This specification's embodiments accurately identify abnormal output states by monitoring model training logs in real time. When risks such as policy collapse, loss of diversity, or logic failure are detected, a version rollback mechanism based on historical data backtracking or by directly loading stable checkpoints is flexibly adopted. This effectively solves the performance degradation problems caused by gradient instability, reward function spoofing, or overfitting during training. It can not only quickly restore the model to a reliable performance baseline without interrupting the overall process, but also enhance the model's adaptability and exploration ability to complex environments by dynamically adjusting hyperparameters or resetting the optimizer state, thereby significantly improving the robustness of the training process, convergence efficiency, and the quality and compliance of the final generated content.

[0224] In practical applications, the above solution can be implemented as follows: 1. Obtain the training cue set And set the hyperparameters:

[0225] in, Provide batch size information for the target. For group size, For maximum generation length, Outlier coefficient To minimize the cutoff amplitude, The correctness threshold, This represents the clipping factor in the strategic objectives. The KL penalty coefficient can be denoted as: >0, The proportion of high covariance lexical selection, This is the length of the history window.

[0226] 2. In the In each training step, the actual number of sampling prompts is determined based on the predictive dynamic sampling module. and from The sampling yielded:

[0227] 3. For each prompt word Using the old strategy sampling Candidate outputs;

[0228] And calculate the original reward:

[0229] 4. For each prompt word The group type is determined for the corresponding sampling group. If:

[0230] If the result is correct, it is considered a fully correct group; otherwise, it is considered a partially correct group.

[0231] 5. Calculate the within-group length statistic only for groups with all correct answers. Let:

[0232] but:

[0233] Define a dynamic truncation threshold:

[0234] 6. Perform outlier detection and truncation on all correct groups. If a candidate output satisfies:

[0235] Then cut it into the first part One word element; otherwise, it remains unchanged. Therefore:

[0236] 7. Recalculate the reward for the truncated output:

[0237] 8. Select effective groups based on the recalculated within-group reward variance. Define the set of effective groups:

[0238] 9. From The middle assembly is just right Each training set constitutes an update set. .when:

[0239] A prompt-level discard can be performed at certain times; when:

[0240] Masked padding, delayed batching, or step-by-step accumulation can be used to fill in the gaps.

[0241] 10. Recalculate the group relative advantage using the recalculated rewards of the effective groups and perform a policy update. For any ,definition:

[0242] Again Replace the original Enter the strategy target to perform an update.

[0243] 11. Optionally, perform directed KL regularization and predictive dynamic sampling window update, and then proceed to the next training step.

[0244] The following is in conjunction with the appendix Figure 2 Taking the application of the target policy model obtained by the model training method provided in this manual in the model inference method as an example, the model training method will be further explained. Figure 2 A flowchart of a model inference method provided in one embodiment of this specification specifically includes the following steps: Step 202: Obtain the model instructions input by the user.

[0245] Model instructions are sets of request information containing natural language text, structured code snippets, multimodal prompts, or specific control tags. They are typically initiated by end-users or upstream systems to the AI ​​model, triggering the model's internal inference mechanisms to perform operations such as content generation, logical deduction, data analysis, or task planning. These can be open-ended questions, creative writing requirements with constraints, complex programming tasks requiring step-by-step execution, context-based multi-turn dialogues, or commands to parse and transform data in specific formats. Examples include user-inputted requests such as "Please summarize the core ideas of this long document," "Generate a code function that can sort an array," or "Draw a trend chart based on the provided sales data."

[0246] Step 204: Input the model instructions into the target policy model to obtain the instruction output of the target policy model in response to the model instructions, wherein the target policy model is trained according to any model training method.

[0247] The target policy model is a neural network instance trained using the aforementioned model training methods. It possesses fixed parameter weights and convergent policy distribution characteristics. It receives external input instructions and generates response results that meet expected quality, are logically coherent, and satisfy safety constraints based on learned probability distribution patterns. It can be applied to one or more scenarios, such as performing text summarization and translation tasks in natural language processing, assisting in automatic code generation and debugging in software engineering, providing multi-step reasoning solutions in decision support systems, or completing image-text matching and content creation in multimodal interactive environments. For example, it can be used to create a dialogue engine that effectively avoids repetitive output and maintains high-diversity response capabilities through reinforcement learning.

[0248] The instruction output is the set of response results generated by the target strategy model after receiving and processing model instructions. It can include text sequences, code blocks, structured data, logical conclusions, or multimodal content, used to directly respond to user queries, execute specified automated tasks, or provide information support for decision-making. Instruction output can be an accurate answer to a factual question, an original article written to meet creative needs, complete program code that conforms to grammatical rules, a step-by-step solution based on logical deduction, or a standard format report after cleaning and transforming input data, etc. For example, when a user inquires about the weather, the model returns a natural language description containing temperature and precipitation probability; when a script is requested, a code snippet that can be run directly is generated; or when data analysis is requested, a structured report containing chart links and key indicators is output. These are all concrete manifestations of the model transforming internal knowledge representations into externally usable information.

[0249] This specification's embodiments, by acquiring model instructions and inputting them into a target policy model optimized through model training methods, fully leverage the length control mechanism and logic preservation capabilities established during the model's training phase. This allows the generated instruction output to accurately respond to complex user needs while effectively avoiding length drift and redundant repetition issues common in traditional models. This process not only ensures a significant reduction in inference costs while maintaining high accuracy and a complete inference logic front-end, but also, because the target policy model has internalized the characteristic of resolving gradient conflicts from the data source, it can directly deliver concise, compliant, and high-quality responses in actual inference without relying on additional post-processing truncation or length penalties, greatly improving user interaction experience and system resource utilization efficiency.

[0250] See Figure 3 , Figure 3 A flowchart of a question-answering model training method provided in one embodiment of this specification includes the following steps: Step 302: Obtain question-answer sample pairs of the current question-answering model, wherein the question-answer sample pair includes a sample question and at least one sample answer generated by the current question-answering model based on the sample question, and any sample answer consists of multiple word units.

[0251] The current question-answering model is an instance of a dialogue or question-answering system undergoing reinforcement learning. In this specification, the current question-answering model can be in a specific training stage and carry the current parameter state during the reinforcement learning iteration process. It can be represented as a neural network architecture after pre-training initialization and before a certain round of reinforcement learning update. The role of the current question-answering model is to generate a sequence of sample answers containing multiple lexical units based on the input sample question, providing the original data foundation for subsequent truncation processing based on the number of lexical units and the construction of effective question-answer sample pairs.

[0252] A question-answering sample pair is a dataset consisting of a specific sample question and at least one sample answer generated by the current question-answering model based on that question. It provides an initial data source containing the diversity of original generation and lexical distribution characteristics, enabling subsequent filtering of truncated objects based on lexical counts and the construction of effective samples for reinforcement learning. This set may include one or more sequences containing explanations of varying lengths from the model output for factual queries, logical reasoning tasks, or open-ended questions, or a candidate set encompassing a single standard answer and multiple redundant variations, etc.

[0253] The sample question is a natural language question or task description input into the current question-answering model to trigger the generation of sample answers. The sample question can serve as the starting condition for the reasoning process, guiding the model to produce a response sequence composed of multiple lexical units for subsequent truncation and reinforcement learning evaluation. It can be presented as one or more forms of combination, such as knowledge retrieval questions, logical deduction propositions, life advice requests, or technical troubleshooting, for example, asking for explanations of physical concepts, providing travel planning suggestions, diagnosing code errors, etc.

[0254] Sample answers are response sequences generated by the current question-answering model in response to sample questions. They may include one or more lexical units and serve as the basic data objects for subsequent filtering, truncation, and construction of valid question-answer sample pairs based on the number of lexical units. In other words, they provide raw material for implementing length control and strategy optimization. Sample answers can take various forms, such as detailed explanations with step-by-step derivations, direct conclusive statements, or long texts with redundant restates and polite greetings. Their specific length and content structure change dynamically with the model's generation state.

[0255] There are several ways to obtain question-answer sample pairs for the current question-answering model. One possible way is to obtain the question-answer sample pairs for the current question-answering model from the already constructed question database.

[0256] In this approach, one or more of the following sampling methods can be used in combination: fixed-multiple oversampling, dynamic adaptive sampling, reward threshold filtering, diversity constraint sampling, and curriculum progressive sampling. The specific method depends on the actual situation, and the embodiments in this specification do not limit this.

[0257] Another possible approach is to generate and construct question-answer sample pairs for the current question-answering model based on prompt words during the training process, according to the requirements.

[0258] This step obtains question-answer sample pairs containing sample questions and corresponding multi-lexical sample answers, providing a data source with original generation diversity and distribution characteristics for subsequent filtering and truncation operations based on the number of lexical units. This enables the model to construct effective training samples based on inference paths or response forms of different lengths, thereby supporting gradient updates and length control optimization of the question-answering model during reinforcement learning and ensuring that the training process can handle various forms of generated results, from concise answers to lengthy explanations.

[0259] Step 304: Based on the number of lexical units in at least one sample answer, determine the sample answer to be truncated from at least one sample answer.

[0260] The truncated sample answers are used as sample answers to perform the truncation operation, generating valid training samples that meet the length limit. They can be response sequences selected from question-answer sample pairs based on the number of lexical units, exceeding a preset threshold in length. The truncated sample answers can be presented as detailed analyses containing lengthy reasoning steps, argumentative paragraphs with repeated verification steps, or long answers with redundant pleasantries and background information, etc. For example, long texts with a large number of intermediate derivations generated for complex scientific questions, detailed descriptions with repeated self-corrections generated for open-ended questions, or responses with excessive preamble when responding to consultation instructions. Their specific characteristics are that the total number of lexical units exceeds the preset range, and they need to be processed before being included in the subsequent reinforcement learning process.

[0261] There are several ways to determine the sample answer to be truncated. One possible way is to set a preset length threshold. If the number of words in the sample answer is greater than the preset threshold, the sample answer is determined to be the sample answer to be truncated.

[0262] Another possible approach is to determine a dynamic length threshold based on the number of lexical units in at least one sample answer, and then determine the sample answer to be truncated from at least one sample answer according to the dynamic length threshold.

[0263] Another possible approach is to use a large model to determine the number of redundant words in the sample answer. When the number of redundant words exceeds a preset threshold, or when the number of redundant words and the number of words in the sample answer reach a certain ratio, the sample answer is determined to be a sample answer to be truncated.

[0264] This step accurately identifies the objects to be truncated from the sample answers based on the number of lexical units, enabling automatic location and differentiation of excessively long response sequences or texts containing redundant content. This allows subsequent processing to perform targeted length truncation operations, thereby eliminating resource waste and training noise caused by excessively long generated texts while preserving key semantic information. This lays the foundation for constructing high-quality and effective question-answer sample pairs that conform to the preset length specifications, thereby improving the question-answering model's ability to control the generated length and its training efficiency during reinforcement learning.

[0265] Step 306: Truncate multiple words in the truncated sample answer to obtain the truncated sample answer.

[0266] Truncation is a process that removes or shortens the ends of text sequences that exceed a preset length limit. It is used to adjust overly long sample answers into effective samples that meet training specifications, thereby optimizing resource utilization and model convergence. This operation can manifest as cutting off the tail of lengthy reasoning steps, deleting paragraphs with repeated verifications, or removing redundant pleasantries and background information. For example, it can perform end truncation on scientific answers containing a large number of intermediate derivations, compress the length of detailed descriptions that repeatedly self-correct, or remove suffixes from responses with excessive preamble. The specific execution method changes dynamically with word distribution and threshold settings.

[0267] Truncated sample answers are text sequences that meet length limits and are generated after performing word removal or shortening on the truncated sample answers. They are used as valid samples that meet training specifications in the reinforcement learning process to support gradient updates and optimization of the question-answering model. Truncated sample answers can be presented as detailed explanations that retain the core reasoning logic, concise statements after removing duplicate checks, or reply fragments that have removed unnecessary pleasantries, etc.

[0268] There are several ways to truncate multiple words in a truncated sample answer. One possible way is to truncate multiple words at the end of the truncated sample answer to obtain the truncated sample answer.

[0269] Another possible approach is to identify multiple redundant words, truncate the redundant words in the answer to be truncated, and obtain the truncated answer. This step transforms the excessively long sequences into truncated sample answers that conform to the preset length specification by performing tail removal or redundant word removal operations on the identified sample answers to be truncated. This allows samples that could not be directly used for training due to exceeding the limit to retain their core semantics and be included in effective question-answer sample pairs. In this way, while avoiding resource waste and gradient update anomalies, the diversity and integrity of training data in the reinforcement learning process are maintained, and the adaptability and convergence efficiency of the question-answering model under different generation length scenarios are improved.

[0270] Step 308: Construct effective question-answer sample pairs based on sample questions and truncated sample answers.

[0271] Valid question-answer pairs are a set of samples that meet training requirements after screening. They are used to provide high-quality, long-stable question-answer pairs during reinforcement learning to support stable calculation of policy gradients and updates to model parameters. Valid question-answer pairs may include truncated and detailed inference sample sets, concise dialogue libraries with redundant arguments removed, or training packages with unnecessary preamble removed, etc. Valid question-answer pairs may include sample questions, truncated sample answers, and other sample answers in the question-answer pair excluding the truncated sample answers.

[0272] One possible way to construct effective question-answer pairs is to use sample questions and truncated sample answers as question-answer pairs.

[0273] Another possible approach is to construct effective question-answer sample pairs using sample questions, truncated sample answers, and untruncated sample answers from the sample answers.

[0274] This step assembles sample questions and truncated sample answers into structurally consistent and effective question-answer pairs. This transforms the generated data, which was originally difficult to use directly due to its excessive length, into high-quality question-answer pairs that meet the training specifications. Thus, while ensuring the stability of input and output lengths, it provides a continuous and consistent optimization signal for policy gradient calculation, effectively avoiding length drift caused by sequence length fluctuations, and improving the convergence speed and generalization ability of the model when dealing with long text answering tasks.

[0275] Step 310: Based on the effective question-answer sample pairs, perform reinforcement learning on the current question-answering model to obtain the target question-answering model.

[0276] Reinforcement learning is a machine learning paradigm that maximizes cumulative rewards by interacting with the environment and adjusting policies based on feedback signals. It guides models to autonomously explore optimal behavioral sequences and achieve adaptive optimization of complex tasks in scenarios lacking explicit supervision labels. Reinforcement learning can manifest as alignment training based on human preference ranking, decision evolution dependent on user satisfaction results, or a self-correcting generative process based on answer accuracy scores. Examples include updating weights for multiple versions of answers output by a large language model based on human scores, and adjusting answer probabilities through backpropagation based on user feedback in consultation scenarios. Reinforcement learning paradigms can include GRPO and PPO, among others.

[0277] A goal-oriented question-answering model is the final policy network formed through reinforcement learning training iterations. It typically possesses specific task execution capabilities and optimization characteristics. During the deployment phase, it receives user questions and outputs the optimal sequence of answers to solve complex inquiries or generate high-quality content. Goal-oriented question-answering models can take the form of intelligent customer service assistants aligned with human feedback, consultation agents evolving based on user satisfaction rewards, or knowledge-based question-answering engines that self-correct based on answer accuracy. Examples include inference models that can output logically rigorous and length-compliant answer sequences in long-text inference tasks, dialogue models that generate concise, accurate, and non-redundant responses in open-domain question-answering scenarios, or analytical models that produce directly resolving answer fragments in automated consultation environments.

[0278] This specification's embodiments obtain question-and-answer sample pairs containing sample questions and multi-lexical sample answers, preserving the original generation diversity and validation foundation of the current question-and-answer model. Based on the number of lexical units in the sample answers, it directly filters and determines the sample answers to be truncated. Without introducing additional length penalties or modifying the reward formula, it avoids the gradient conflict between shortening the answer length and maintaining the answer accuracy from the data source, preventing training instability caused by reward coupling. It constructs effective question-and-answer sample pairs using sample questions and truncated sample answers, forcing the samples input to the reinforcement learning stage to converge to a shorter length distribution while retaining the core reasoning logic, suppressing the length drift phenomenon in question answers. Based on these effective question-and-answer sample pairs, it performs reinforcement learning on the current question-and-answer model, avoiding the gradient vanishing problem caused by an increase in the proportion of all correct samples, significantly improving sampling efficiency and obtaining a target question-and-answer model that combines high accuracy and low inference cost.

[0279] The following is in conjunction with the appendix Figure 4 Taking the application of the target question-answering model obtained by the question-answering model training method provided in this manual in the model inference method as an example, the question-answering model training method will be further explained. Figure 4 A flowchart of a question-and-answer method provided in one embodiment of this specification specifically includes the following steps: Step 402: Obtain the question text input by the user.

[0280] The question text is a natural language query request initiated by the end user or upstream system to the question answering system. It is used to trigger the target question answering model to perform response behaviors such as information retrieval, logical deduction, knowledge integration, or fact verification. Its forms can cover one or more cases, such as factual questions, such as "How high is Mount Everest?", explanatory questions, such as "Why does iron rust?", operation guidance requests, such as "How to freeze the first row in Excel?", multi-hop reasoning questions, such as "How much difference is there between Einstein's age when he won the Nobel Prize and his age when he published the theory of special relativity?", or conversational follow-up questions that require contextual understanding. The text serves as the input condition for the model's reasoning and can drive the model to generate semantically coherent, length-appropriate, and content-compliant answers.

[0281] Step 404: Input the question text into the target question answering model to obtain the answer output by the target question answering model for the question text. The target policy model is trained according to the question answering model training method.

[0282] The target question-answering model is a neural network instance optimized using the aforementioned question-answering model training method. Its parameter weights have converged during the reinforcement learning process, and its policy distribution possesses the ability to stably model the semantics of the question and the ability to intrinsically control the length of the answer. After receiving the question text, the model can autonomously generate an answer that retains the core reasoning chain or key facts while avoiding redundant preambles, repetitive statements, and ineffective small talk. Its output is naturally adapted to the response latency, bandwidth limitations, and user experience expectations of actual deployment scenarios, making it suitable for application environments with clear requirements for response quality and efficiency, such as intelligent customer service, educational tutoring, technical document Q&A, and medical consultation assistance.

[0283] The answer is a complete response sequence generated by the target question-answering model based on the question text. It consists of multiple lexical units, and its content covers the facts, logical deduction process or operation steps involved in the question. It also exhibits a controlled distribution in the number of lexical units—neither missing necessary information due to being too short, nor introducing irrelevant details due to being too long. The answer can be expressed in various forms such as concise conclusions, step-by-step explanations, explanations with supporting evidence, or structured summaries. Its generation quality and length stability are derived from the truncation and selection of question-answer sample pairs, effective construction, and reinforcement learning optimization during the training phase, rather than heuristic rules or manual template intervention during the deployment phase.

[0284] This specification's embodiments acquire user-inputted question text, providing natural language input consistent with real-world interaction scenarios for subsequent reasoning. This question text is then input into a target question-answering model optimized using a question-answering model training method. This model, during the training phase, has internalized the ability to autonomously control the length of responses and stably retain core semantics through truncation and filtering based on word count, construction of effective question-answer sample pairs, and reinforcement learning updates. Therefore, the model can directly output logically complete, information-rich, and word-count-controlled responses. While maintaining high accuracy and the integrity of key reasoning chains, this response significantly reduces the proportion of redundant words and generation latency, avoiding response delays or information dilution caused by length drift. Thus, in practical application scenarios such as intelligent customer service, educational tutoring, and technical Q&A, it simultaneously improves response quality, response efficiency, and system resource utilization.

[0285] Corresponding to the above method embodiments, this specification also provides embodiments of a model training device. Figure 5 This is a schematic diagram of a model training device provided in one embodiment of this specification. Figure 5 As shown, the device includes: The sampling acquisition module 502 is configured to acquire a sampling group of the current policy model, wherein the sampling group includes a sample instruction and at least one candidate output generated by the current policy model according to the sample instruction, and any candidate output consists of multiple lexical units; The determination module 504 is configured to determine the candidate output to be truncated from at least one candidate output based on the number of lexical units in at least one candidate output; The truncation module 506 is configured to truncate multiple words in the candidate output to obtain the truncation candidate output; Module 508 is configured to construct a valid sample group based on sample instructions and truncated candidate outputs; The reinforcement learning module 510 is configured to perform reinforcement learning on the current policy model based on the effective sample group to obtain the target policy model.

[0286] The model training device also includes a statistics module, configured to count the number of lexical units in the at least one candidate output to obtain a lexical unit count value; The model training apparatus also includes a statistical threshold determination module, configured to determine candidate outputs to be truncated from the at least one candidate output based on the statistical value of the number of lexical units.

[0287] The truncation module 506 is further configured to truncate multiple words in the candidate output to be truncated based on the word number statistics to obtain truncated candidate output; The model training device also includes a threshold generation module, configured to determine the truncation threshold of the candidate output to be truncated based on the mean number of lexical units and the standard deviation of the number of lexical units. The truncation module 506 is further configured to truncate multiple words in the candidate output to be truncated based on the word truncation threshold to obtain truncated candidate output.

[0288] The determination module is further configured to determine the truncation amount of at least one candidate output based on the number of lexical units in at least one candidate output and a lexical truncation threshold; and to determine the candidate output to be truncated from at least one candidate output based on the truncation amount of at least one candidate output and a preset minimum number of truncation lexical units.

[0289] The sampling acquisition module 502 is further configured to acquire sample instructions; generate at least one candidate output based on the sample instructions using the current policy model; score the at least one candidate output for reward based on a preset reward function to obtain the reward function of at least one candidate output; if the reward function of at least one candidate output is greater than a preset reward threshold, determine the sample instructions and at least one candidate output generated by the current policy model based on the sample instructions as the sampling group of the current policy model.

[0290] The model training device also includes an effective sampling determination module, which is configured to score the truncated candidate outputs based on a preset reward function to obtain the reward function of the truncated candidate outputs; calculate the standard deviation of the reward function of the truncated candidate outputs to determine the reward standard deviation corresponding to the reward function of the truncated candidate outputs; and determine the truncated candidate outputs used to construct the effective sampling group from the truncated candidate outputs according to the reward standard deviation and the preset reward standard deviation threshold.

[0291] The model training device also includes a mean calculation module, which is configured to calculate the mean of the reward function of the truncated candidate outputs to determine the mean reward corresponding to the reward function of the truncated candidate outputs.

[0292] The reinforcement learning module 510 is further configured to determine the reward function offset of the truncated candidate outputs in the effective sampling group based on the reward function and the mean reward of the truncated candidate outputs in the effective sampling group; determine the dominance function of the truncated candidate outputs in the effective sampling group based on the reward function offset and the standard deviation of the reward; and adjust the model parameters of the current policy model based on the dominance function to obtain the target policy model.

[0293] The model training device also includes an overcoupling determination module, which is configured to determine the lexical covariance among multiple lexical units of truncated candidate outputs in the effective sampling group; and to determine whether there are overcoupling lexical units among the multiple lexical units based on the lexical covariance and a preset lexical covariance threshold.

[0294] The reinforcement learning module 510 is further configured to, if so, construct an overcoupling penalty based on the model parameters of the current policy model and the adjusted model parameters of the current policy model; under the constraint of the overcoupling penalty, adjust the model parameters of the current policy model based on the advantage function to obtain the target policy model.

[0295] The model training device also includes an oversampling determination module, configured to determine the proportion of historical effective groups based on the number of historical sampling groups and the number of effective groups in the historical sampling groups; determine the oversampling factor of the sampling groups based on the proportion of historical effective groups; and obtain the sampling groups of the current policy model based on the sampling sample instructions and at least one candidate output generated by the current policy model according to the sample instructions.

[0296] The model training device also includes a training log determination module, configured to acquire intermediate model training data, wherein the intermediate model training data includes at least one of the following: truncation ratio, average truncation magnitude, average number of tokens, policy entropy, average reward function, and oversampling factor; and generate model training logs based on the intermediate model training data.

[0297] The model training device also includes a monitoring module configured to monitor the model training log, determine the output state of the current policy model, and, in the event of an abnormal output state of the current policy model, backtrack the model training log, adjust the current policy model based on the backtracking results, and obtain the adjusted current policy model; and / or, in the event of an abnormal output state of the current policy model, obtain the rollback node of the current policy model, roll back the current policy model based on the rollback node, and obtain the rolled-back current policy model, wherein the rollback node is the previous version of the current policy model recorded by the checkpoint.

[0298] The model training device provided in this specification achieves autonomous control over response length from the sampling group at the training source through a closed-loop process of sampling, truncation, construction, and reinforcement learning. By leveraging statistical analysis of the number of lexical units and dynamic threshold setting, redundant long responses are accurately identified and truncated, preserving key semantics. Combined with reward selection and advantage function modeling, the quality of training samples and gradient stability are ensured. Over-coupling detection and penalty mechanisms are introduced to improve the decoupling and consistency of generated content. Oversampling scheduling driven by historical efficiency is used to improve sampling efficiency. Furthermore, training log monitoring, backtracking, and rollback capabilities enable rapid location and recovery of output anomalies. Overall, while maintaining high accuracy and logical integrity, this device significantly reduces response length, decreases invalid generation, and enhances training stability, ultimately obtaining a high-quality, low-latency, and robust target policy model.

[0299] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0300] Corresponding to the above method embodiments, this specification also provides embodiments of a model inference device. Figure 6 This is a schematic diagram of a model inference device provided in one embodiment of this specification. Figure 6 As shown, the device includes: The instruction acquisition module 602 is configured to acquire model instructions input by the user.

[0301] The output acquisition module 604 is configured to input the model instruction into the target policy model and obtain the instruction output of the target policy model in response to the model instruction, wherein the target policy model is trained according to the above-described model training method.

[0302] The instruction acquisition module and output acquisition module provided in the embodiments of this specification directly receive user natural language instructions and input them into the target policy model optimized by the aforementioned training method. This allows the model to autonomously output semantically accurate, logically coherent, and appropriately sized instructions during the inference phase without additional truncation, reordering, or rule post-processing. This process fully utilizes the length control capabilities and diversity preservation mechanisms internalized during the training phase, effectively avoiding problems such as redundancy, overexpansion, and length drift. While ensuring response quality, it significantly reduces computational latency and resource consumption, thereby improving the real-time performance, consistency, and user experience of terminal interaction.

[0303] The above is a schematic scheme of a model reasoning device according to this embodiment. It should be noted that the technical solution of this model reasoning device and the technical solution of the model reasoning method described above belong to the same concept. For details not described in detail in the technical solution of the model reasoning device, please refer to the description of the technical solution of the model reasoning method described above.

[0304] Figure 7 This is a structural block diagram of a computing device provided according to one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0305] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 80211 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0306] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0307] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0308] The processor 720 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned model training method, model inference method, question answering model training method, or question answering method.

[0309] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the above-mentioned model training method, model inference method, question-answering model training method or question-answering method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the above-mentioned model training method, model inference method, question-answering model training method or question-answering method.

[0310] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described model training method, model inference method, question-answering model training method, or question-answering method.

[0311] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-described model training method, model inference method, question-answering model training method, or question-answering method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the above-described model training method, model inference method, question-answering model training method, or question-answering method.

[0312] An embodiment of this specification also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described model training method, model inference method, question-answering model training method, or question-answering method.

[0313] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned model training method, model inference method, question-answering model training method or question-answering method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned model training method, model inference method, question-answering model training method or question-answering method.

[0314] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0315] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0316] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0317] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0318] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A model training method, characterized in that, include: Obtain a sampling group of the current policy model, wherein the sampling group includes a sample instruction and at least one candidate output generated by the current policy model according to the sample instruction, and any candidate output consists of multiple lexical units; Based on the number of lexical units in the at least one candidate output, determine the candidate output to be truncated from the at least one candidate output; The multiple words in the candidate output to be truncated are truncated to obtain truncated candidate output; Based on the sample instructions and the truncated candidate outputs, construct an effective sampling group; Based on the effective sampling group, reinforcement learning is performed on the current policy model to obtain the target policy model.

2. The method according to claim 1, characterized in that, Before determining the candidate output to be truncated from the at least one candidate output based on the number of lexical units of the at least one candidate output, the method further includes: The number of lexical units in the at least one candidate output is counted to obtain a lexical unit count value; The step of determining the candidate output to be truncated from the at least one candidate output based on the number of lexical units in the at least one candidate output includes: Based on the statistical value of the number of lexical units, candidate outputs to be truncated are determined from the at least one candidate output.

3. The method according to claim 2, characterized in that, The step of truncating multiple words in the candidate output to be truncated to obtain truncated candidate output includes: Based on the statistical value of the number of lexical units, multiple lexical units of the candidate output to be truncated are truncated to obtain truncated candidate output.

4. The method according to claim 3, characterized in that, The lexical count statistics include the mean lexical count and the standard deviation of the lexical count; Before truncating multiple words in the candidate output to be truncated based on the word count statistics to obtain the truncated candidate output, the method further includes: The truncation threshold of the candidate output to be truncated is determined based on the mean number of lexical units and the standard deviation of the number of lexical units. The step of truncating multiple words in the candidate output to be truncated based on the statistical value of the number of words to obtain the truncated candidate output includes: Based on the lexical truncation threshold, multiple lexical units of the candidate output to be truncated are truncated to obtain truncated candidate output.

5. The method according to claim 4, characterized in that, The step of determining the candidate output to be truncated from the at least one candidate output based on the statistical value of the number of lexical units includes: Based on the number of lexical units in the at least one candidate output and the lexical unit truncation threshold, determine the truncation amount of the at least one candidate output; Based on the truncation amount of the at least one candidate output and the preset minimum number of truncation terms, the candidate output to be truncated is determined from the at least one candidate output.

6. The method according to claim 1, characterized in that, The step of obtaining the sampling group of the current policy model includes: Get sample instructions; Using the current policy model, at least one candidate output is generated based on the sample instructions; The reward function for the at least one candidate output is obtained by scoring the at least one candidate output based on a preset reward function. If the reward function of at least one candidate output is greater than a preset reward threshold, the sample instruction and at least one candidate output generated by the current policy model according to the sample instruction are determined as a sampling group of the current policy model.

7. The method according to claim 1, characterized in that, Before constructing the valid sampling group based on the sample instructions and the truncated candidate output, the method further includes: The truncated candidate output is scored based on a preset reward function to obtain the reward function of the truncated candidate output; The standard deviation of the reward function of the truncated candidate output is calculated to determine the reward standard deviation corresponding to the reward function of the truncated candidate output. Based on the reward standard deviation and the preset reward standard deviation threshold, truncated candidate outputs for constructing the effective sampling group are determined from the truncated candidate outputs.

8. The method according to claim 7, characterized in that, The process of performing reinforcement learning on the current policy model based on the effective sampling group to obtain the target policy model further includes: The mean of the reward function of the truncated candidate output is calculated to determine the mean reward corresponding to the reward function of the truncated candidate output; The step of performing reinforcement learning on the current policy model based on the effective sampling group to obtain the target policy model includes: Based on the reward function and the mean reward of the truncated candidate outputs in the effective sampling group, the offset of the reward function of the truncated candidate outputs in the effective sampling group is determined; Based on the reward function offset and the reward standard deviation, determine the dominant function for truncating candidate outputs in the effective sampling group; Based on the aforementioned advantage function, the model parameters of the current strategy model are adjusted to obtain the target strategy model.

9. The method according to claim 8, characterized in that, Before adjusting the model parameters of the current strategy model based on the advantage function to obtain the target strategy model, the method further includes: Determine the lexical covariance among multiple lexical units of the truncated candidate output in the effective sampling group; Based on the lexical covariance and the preset lexical covariance threshold, determine whether there are over-coupled lexical units among the plurality of lexical units; The step of adjusting the model parameters of the current strategy model based on the advantage function to obtain the target strategy model includes: If so, construct an over-coupling penalty based on the model parameters of the current strategy model and the adjusted model parameters of the current strategy model; Under the constraint of the over-coupling penalty, the model parameters of the current policy model are adjusted based on the advantage function to obtain the target policy model.

10. The method according to claim 1, characterized in that, Before obtaining the sampling group of the current policy model, the method further includes: The proportion of historical valid groups is determined based on the number of historical sampling groups and the number of valid groups in the historical sampling groups; Based on the aforementioned proportion of historical valid groups, determine the oversampling factor for sampling the sampled groups; Based on the oversampling factor, the sample instruction and at least one candidate output generated by the current policy model according to the sample instruction are sampled to obtain the sample group of the current policy model.

11. The method according to claim 1, characterized in that, The method further includes: Obtain intermediate training data for the model, wherein the intermediate training data includes at least one of the following: truncation ratio, average truncation magnitude, average number of tokens, policy entropy, average reward function, and oversampling factor; Based on the intermediate data from the model training, a model training log is generated.

12. The method according to claim 11, characterized in that, After generating the model training log based on the intermediate data from the model training, the method further includes: Monitor the model training logs to determine the output state of the current policy model; If the output state of the current policy model is abnormal, the model training log is backtracked, and the current policy model is adjusted based on the backtracking results to obtain the adjusted current policy model. And / or, if the output state of the current policy model is abnormal, obtain the rollback node of the current policy model, roll back the current policy model based on the rollback node, and obtain the rolled-back current policy model, wherein the rollback node is the previous version of the current policy model recorded by the checkpoint.

13. A model reasoning method, characterized in that, include: Obtain model instructions input by the user; The model instruction is input into the target policy model to obtain the instruction output of the target policy model in response to the model instruction, wherein the target policy model is trained by the model training method according to any one of claims 1-12.

14. A question-answering model training method, characterized in that, include: Obtain question-answer sample pairs of the current question-answering model, wherein the question-answer sample pair includes a sample question and at least one sample answer generated by the current question-answering model based on the sample question, and any sample answer consists of multiple word units; Based on the number of lexical units in the at least one sample answer, determine the sample answer to be truncated from the at least one sample answer; The multiple words of the sample answer to be truncated are truncated to obtain the truncated sample answer; Based on the sample questions and the truncated sample answers, construct effective question-answer sample pairs; Based on the effective question-answer sample pairs, reinforcement learning is performed on the current question-answering model to obtain the target question-answering model.

15. A question-and-answer method, characterized in that, include: Get the question text entered by the user; The question text is input into the target question-answering model to obtain the answer output by the target question-answering model for the question text, wherein the target question-answering model is trained according to the question-answering model training method of claim 14.

16. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that, The device stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

18. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 15.

Images