A large language model alignment method based on semantic segmentation and length penalty

Abstract

The application discloses a large language model alignment method based on semantic segmentation and length punishment, which introduces a dynamic semantic segmentation mechanism based on information entropy, uses the information entropy mutation point in the autoregressive output process as the natural segmentation boundary, and dynamically divides the continuous long text into semantic segments with independent logic. On this basis, by training a segment-level reward model (Segment-level RM), high-density, low-variance fine-grained feedback signals are provided for PPO training, fundamentally solving the sparse credit allocation problem of long sequence generation. The application integrates an adaptive length punishment mechanism in the generalized advantage estimation (GAE) link. According to the non-linear adaptive adjustment of the reference reply length and the current generation state, the mechanism actively removes the false preference score caused by redundant output from the source of gradient update, guiding the policy network to evolve towards the goal of pursuing high unit word information density.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for aligning large language models based on semantic segmentation and length penalty. Background Technology

[0002] The growth in computing power and high-quality data has given rise to the remarkable logical reasoning and instruction-following capabilities of Large Language Models (LLMs). However, the pre-training of basic models relies on autoregressive next-token prediction, which is essentially a statistical fit to massive corpora. Its output is often unpredictable and struggles to directly align with genuine human intentions, ethics, and complex values. To bridge this gap, "AI Alignment" has become a core focus of current large-scale model research, and reinforcement learning based on human feedback (RLHF) has established itself as the mainstream paradigm for achieving this goal.

[0003] The RLHF framework has achieved significant results in improving the quality of model interactions. However, with the increasing complexity of instruction scenarios, the limitations of traditional alignment paradigms in underlying optimization mechanisms become increasingly apparent. In Markov Decision Processes (MDPs), traditional reinforcement learning typically relies solely on the reward model to provide a global scalar score after generating a complete long text. Faced with long text responses, this coarse-grained, text-level evaluation struggles to accurately attribute rewards and penalties to specific words. This phenomenon of "sparse credit assignment" introduces significant gradient noise, severely limiting the convergence efficiency and alignment accuracy of proximal policy optimization (PPO) algorithms. Another hidden danger arising from sparse feedback is that human annotators subconsciously tend to assign high scores to detailed and lengthy responses. The reward model amplifies this spurious statistical association during fitting, inevitably giving rise to "length bias." During policy fine-tuning, the model is highly susceptible to "normative games," attempting to cheat for high scores through meaningless word stuffing. This reward hacking not only leads to sluggish content but also causes inference delays and unnecessary waste of computing power.

[0004] Current techniques often attempt to balance the ratio of long and short responses at the data level or introduce static length regularization penalties into the loss function. However, lacking adaptive awareness of the dynamic generation process, these static patching methods are prone to excessively weakening model performance, and may even lead to excessive truncation when handling complex instructions requiring lengthy, in-depth parsing, thus compromising logical integrity. How to fundamentally eliminate the length advantage while ensuring generation quality and logical depth, and achieve a dynamic balance between semantics and redundancy, has become a critical technical bottleneck that urgently needs to be addressed in the field of large model alignment. Summary of the Invention

[0005] The purpose of this invention is to provide a method for aligning large language models based on semantic segmentation and length penalty, which provides a highly robust technical path for building large language model systems with low inference latency, high information density and deep alignment with real human values.

[0006] To achieve the above objectives, the present invention provides the following solution: A method for aligning large language models based on semantic segmentation and length penalty includes the following steps: S1. Construct a Markov decision process, and based on information entropy, perform dynamic semantic segmentation on the response sequence generated by the large language model, dividing the response sequence into several non-overlapping segments with independent semantic integrity, i.e., semantic segments. S2. Based on a pre-trained fragment-level reward model, each semantic fragment is given an independent quality score, i.e., a fragment reward, and the sequence baseline score is obtained by aggregating the fragment rewards. S3. Construct an adaptive length penalty function, and calculate the length penalty based on the relative difference between the actual length of the currently generated sequence and the baseline reference length for the current prompt word; S4. The fragment reward and length penalty are integrated to reconstruct the step reward function in the near-end policy optimization algorithm. The length penalty at the end is backpropagated to the previous generation state through generalized advantage estimation to update the parameters of the large language model, improve the semantic information density of a single word in the model, and achieve alignment of the large language model.

[0007] In the preferred S1, dynamic semantic segmentation is performed on the response sequence generated by the large language model, specifically including: Computational large language model in generating the i-th lexical Local Predicted Distribution of Shannon Entropy :

[0008] in, For vocabulary Chinese word elements Given prompt words and above The predicted probability is as follows; Indicator function that triggers segmentation The definition is as follows:

[0009] in, This is a backup rule set that includes strong termination symbols specific to natural language. The empirical information entropy threshold is used; through the above dynamic algorithm, the total length is... The original word sequence was transformed A sequence of non-overlapping semantic segments .

[0010] In the preferred S1, the empirical information entropy threshold Setting it to 1.75 stabilizes the length of the segmented semantic fragments between 10 and 40 lexical units.

[0011] In the preferred S2, the sequence benchmark scores are aggregated using average pooling, as shown in the following formula:

[0012] in, For the sequence baseline score, For the fragment-level reward model, the k-th fragment The rating.

[0013] In the preferred S3, the adaptive length penalty function is constructed as follows:

[0014] in, This serves as a baseline reference length for the current suggested word. The total number of actual lexical units generated for a large language model; when When the penalty value is zero, it ensures the normal expression space of the model; when At that time, the penalty value increases linearly with the excess ratio.

[0015] In the preferred S4, the step reward function in the near-end policy optimization algorithm is reconstructed, as shown in the following formula:

[0016] in, The KL divergence constraint coefficient is... For the indicator function activated at the end node, For the current large language model, For reference to large language models, This is the length penalty coefficient. Based on this, the fragment-level dominance function is calculated using generalized dominance estimation. The formula is as follows:

[0017]

[0018] in, For time step Single-step timing difference error, This is a value estimate of the expected return of a state by the Critic value network. As a discount factor, This is the GAE smoothing parameter, used to balance the bias and variance of the advantage estimate. This represents the state at time step k.

[0019] Preferred length penalty coefficient The range of values ​​is This is used to adjust the game-theoretic balance between alignment quality and information density; for general dialogue tasks and complex logical reasoning tasks, it is used to adjust... Achieve Pareto optimality for both generation quality and length.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a large language model alignment method based on semantic segmentation and length penalty as described above.

[0021] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: (1) This invention proposes a joint optimization framework that integrates semantic segmentation and length penalty. This framework abandons the traditional global delayed scalar reward and innovatively introduces a dynamic segmentation strategy based on the information entropy of local prediction distribution, which uses semantic mutation points to adaptively segment long text sequences into independent semantic segments. This mechanism not only realizes the dimensionality increase and density of credit allocation granularity, but also significantly reduces the gradient variance of the value network during the training of proximal policy optimization (PPO), providing a more discriminative optimization signal for the policy model.

[0022] (2) This invention deeply injects a nonlinear adaptive length penalty term into the underlying link of Generalized Advantage Estimation (GAE). This mechanism, through backpropagation of the dynamic penalty terminal and the advantage function of the preceding state, blocks the model's speculative dependence on redundant lexical units from the gradient source, guiding the generation tendency to shift from expanding the text size to increasing the information density of unit lexical units. Systematic empirical results on multidimensional core benchmarks show that this framework successfully curbs the uncontrolled length drift in the unconstrained state. While significantly reducing the length of invalid generation and reducing the system inference overhead, it achieves a significant leap in absolute win rate and length control win rate, providing a highly robust technical path for building large-scale language model systems with low inference latency, high information density, and deep alignment with real human values. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A diagram illustrating the joint optimization framework for integrating semantic segmentation and length penalty provided by this invention; Figure 2 This is a comparison diagram of the Markov decision process of the credit allocation mechanism used in this invention; Figure 3 This is a mathematical curve of the adaptive length penalty term of the present invention and a schematic diagram of its gradient influence in the backpropagation of the GAE advantage function; Among them, (a) is the mathematical curve of the adaptive length penalty term, and (b) is a schematic diagram of the gradient effect in the backpropagation of the GAE advantage function; Figure 4 The figures show the PPO training dynamic index curves of the Phi-3 model under different length penalty coefficients in this embodiment of the invention. Among them, (a) is the value network loss curve, (b) is the advantage function curve, (c) is the total length curve, (d) is the response length curve, (e) is the segment reward curve, and (f) is the comprehensive reward curve after deducting KL divergence. Figure 5 The above are the dynamic index curves of PPO training for the Qwen2.5-3B model under different length penalty coefficients in this embodiment of the invention. Among them, (a) is the value network loss curve, (b) is the advantage function curve, and (c) is the comprehensive reward curve of the total length; Figure 6 This is a graph showing the relationship between the information entropy threshold and the average semantic segment length in an embodiment of the present invention. Figure 7 This is a diagram showing the dynamic adjustment effect of different λ values ​​on the model win rate and generation length in an embodiment of the present invention. Among them, (a) represents the dynamic regulatory effect on Phi-3-Mini on AlpacaEval 2.0, (b) represents the dynamic regulatory effect on Phi-3-Mini on Arena-Hard, (c) represents the dynamic regulatory effect on Qwen2.5-3B on AlpacaEval 2.0, and (d) represents the dynamic regulatory effect on Qwen2.5-3B on Arena-Hard. Figure 8 This is a system resource monitoring curve during the model training process in an embodiment of the present invention; Among them, (a) is the GPU memory access time percentage curve, (b) is the GPU core temperature curve, and (c) is the GPU computing power utilization curve. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, the present invention provides a method for aligning large language models based on semantic segmentation and length penalty, comprising the following steps: S1. Construct a Markov decision process, and based on information entropy, perform dynamic semantic segmentation on the response sequence generated by the large language model, dividing the response sequence into several non-overlapping segments with independent semantic integrity, i.e., semantic segments. S2. Based on a pre-trained fragment-level reward model, each semantic fragment is given an independent quality score, i.e., a fragment reward, and the sequence baseline score is obtained by aggregating the fragment rewards. S3. Construct an adaptive length penalty function, and calculate the length penalty based on the relative difference between the actual length of the currently generated sequence and the baseline reference length for the current prompt word; S4. The fragment reward and length penalty are integrated to reconstruct the step reward function in the near-end policy optimization algorithm. The length penalty at the end is backpropagated to the previous generation state through generalized advantage estimation to update the parameters of the large language model, improve the semantic information density of a single word in the model, and achieve alignment of the large language model.

[0028] Specifically, the method of the present invention includes: In value alignment and preference learning within large language models, reinforcement learning based on human feedback is the current mainstream technological paradigm. However, in practical applications and underlying algorithmic logic, the classic RLHF framework suffers from two deeply coupled bottlenecks: sparsity of credit allocation and length bias. Traditional sequence-level reward mechanisms typically only provide a global scalar score after the model generates a complete response. For long text generation tasks involving hundreds or even thousands of words, this delayed scalar feedback not only results in extremely large policy gradient variance in the Markov decision process but also easily induces the model to discover false statistical associations such as "more words, higher rewards," leading to reward cheating. This invention proposes a joint optimization method integrating semantic segmentation and length penalty, reconstructing the language generation process into a Markov decision process with dense feedback, and explicitly removing the false bonuses from redundant outputs in the advantage function of policy updates. The overall method architecture covers the entire process from pre-training fine-tuning model initialization, segment-level preference reward model fitting, to iterative updates of proximal policy optimization combined with dynamic length constraints. The overall method's process evolution and module interaction relationships are as follows: Figure 1 As shown.

[0029] 1. Joint Optimization Framework To address the credit allocation problem, a rigorous Markov decision process formal reconstruction is performed on the autoregressive generative process of a large language model. The value alignment process of the large model is defined as a tuple. In traditional token-level reinforcement learning, the action space... Usually a vocabulary list single word in Individual tokens generated by modern word segmenters such as byte-pair encoding often lack independent and complete semantic information. For example, forcibly splitting a technical term into meaningless subwords and then requiring the reward model to score each subword not only lacks linguistic basis but also introduces a large amount of high-frequency gradient noise into the PPO value network. The joint optimization framework semantically upgrades the action space. At the time step... ,state Includes user-input suggestions And all currently generated contexts and actions It is no longer a single word unit, but a series of consecutive word units. In this architecture, the state transition probability It still maintains the deterministic nature of autoregressive generation, but the reward function It is reconstructed into a dense stream of feedback signals continuously provided during the generation process. Credit allocation mechanisms with different alignment granularities, for example... Figure 2As shown, the joint optimization framework's operational pipeline is mainly divided into two stages. In the fragment-level preference fitting stage, a fragment-level reward model capable of accurately scoring local semantic quality is trained using a preference dataset containing pairwise labels (Chosen / Rejected) and a dynamic segmentation mechanism. In the constrained policy optimization stage, the policy model receives dense rewards from fragments from the interaction sampling of PPOs, while being subject to strict auditing of the global generation length. By modifying the generalized advantage estimation, an adaptive length penalty is seamlessly injected into the gradient backpropagation link, forcing the model to achieve Pareto optimality between exploring high-quality semantics and suppressing redundant stacking.

[0030] 2. Semantic segmentation based on information entropy Static heuristic rules often exhibit poor robustness when dealing with complex instructions, code generation, or multilingual collation. To achieve a balance between feedback density and semantic integrity, this invention introduces a dynamic segmentation strategy based on the predictive distribution information entropy to segment the continuous word stream, using it as the foundational credit allocation module of this joint optimization framework.

[0031] 2.1 Shannon Entropy Measurement of Local Predicted Distributions Within a complete semantic block, the transition probabilities between lexical units are highly deterministic, strongly constrained by the context. However, when a semantic block ends and the model prepares to generate a completely new concept or make a logical transition, the uncertainty in predicting the next generation direction increases dramatically. Supervised fine-tuning models or initial policy models on the vocabulary... The predicted probability distribution of the next word given above is as follows: This invention uses Shannon entropy to quantify the degree of prediction uncertainty. For position... lexical elements Its predictive information entropy The definition is given in formula (1):

[0032] It characterizes the "hesitation" of the language model regarding future direction in the current context. The smaller the entropy value, the more confident the model is in the continuity of the current phrase and the stronger the semantic coherence; the appearance of an entropy peak indicates that the current position is very likely a transition point of syntactic boundary or higher-order semantics.

[0033] 2.2 Dynamic Segmentation Criteria and Algorithm Implementation Based on information entropy measurement, this invention employs a dynamic threshold segmentation criterion. An empirical information entropy threshold is preset. In the response sequence of the large model When performing real-time processing or offline preprocessing, the entropy value of each generated word is calculated sequentially. , Indicator function that triggers segmentation The definition is given in formula (2):

[0034] It is a set containing strong termination symbols for a specific natural language, such as [".", "!", "?", " "]. Introduction As a heuristic backup rule, it prevents excessively long fragments from occurring in certain low-order expressions where the entropy value does not show a significant jump, thus avoiding memory overflow or feedback degradation. When the indicator function... At that time, the current word element Its subsequent lexical elements will be assigned to new segments. The preceding words were closed into old fragments. Through this dynamic algorithm, the total length is The original word sequence was losslessly and adaptively transformed into A sequence of non-overlapping semantic segments Combining ablation experiments and prior settings, core parameters Setting the threshold to 1.75 ensures that the length of the segmented fragments is mainly between 10 and 20 lexical units, preserving the logical coherence of the language while achieving a highly dense reward distribution.

[0035] 3. Policy optimization incorporating adaptive length penalty The segmented reward model significantly improves the accuracy of credit allocation, but large models still have the potential risk of maximizing overall returns by piecing together a large number of "correct but useless" correct fragments. To avoid this canonical game loophole, this invention designs an adaptive length penalty mechanism in the policy optimization objective and reconstructs the advantage function estimation of the PPO algorithm.

[0036] 3.1 Fragment Preference Fitting and Benchmark Reward Aggregation Given sequence-level labels in a preference dataset, the reward model first needs to map local feedback to a global score using an aggregation function. Average pooling is then used to evaluate the baseline score of the sequence. For aggregation, see formula (3):

[0037] Average aggregation effectively eliminates the cumulative score advantage naturally carried by long texts due to the increased number of fragments. Under the constraint of average aggregation, RM is forced to focus on the "unit semantic quality" of each local fragment. By minimizing the Bradley-Terry (BT) negative log-likelihood loss, the reward model is trained to have the ability to independently evaluate the quality of fragments.

[0038] 3.2 Mathematical Construction of the Adaptive Length Penalty Term By stripping away the benefits of lengthy outputs, a nonlinear and asymmetric length penalty function is constructed. , .make The total number of actual tokens generated by the strategy model. This serves as the baseline reference length for the current prompt word. The adaptive length penalty is defined in formula (4):

[0039] When the model's response length Located within the reasonable expression range ( When the generated length exceeds the threshold, the derivative is zero, and the penalty mechanism is dormant, ensuring that the normal space for deep reasoning or detailed explanations by the model is not violated. However, once the generated length exceeds the threshold, the penalty term increases monotonically and linearly with the excess ratio. This relative penalty design based on ratio rather than a fixed constant enhances the model's general robustness in handling heterogeneous tasks such as short dialogues and long text question answering.

[0040] 3.3 PPO Objective Function Reconstruction and Advantage Estimation By fusing the fragment reward with the adaptive penalty term, the final alignment reward of the sequence is given by formula (5):

[0041] This is the length penalty coefficient, used to adjust the game-theoretic balance between alignment quality and information density. In the actual interaction sampling of the PPO algorithm, the policy model at each time step, in addition to fitting the value network, also needs to minimize the approximation of the reference policy. The KL divergence is used to avoid catastrophic forgetting. An adaptive length penalty is treated as a global cost of delay and settled at the terminal state of sequence generation. For each time step... Its revised step reward The form is shown in formula (6):

[0042] The KL divergence constraint coefficient is... This is an indicator function that is activated only at the terminal nodes. Based on this, generalized dominance estimation (GAE) is used to calculate the fragment-level dominance function. See formulas (7) and (8):

[0043]

[0044] For time step Single-step timing difference error; For the Critic value network, the value estimate of the expected return of a state is provided. Discount factor; This is the GAE smoothing parameter, used to balance the bias and variance of the advantage estimate. The state at time step k (including cue words and context).

[0045] Because the GAE algorithm has the characteristic of temporal differential forward propagation, the huge penalty of the terminal node will be reduced by the decay parameter. The backpropagation is performed on the intermediate states that generated these redundant fragments. (Policy Network) The objective function for the trimming substitution is shown in formula (9):

[0046] Fragment generation probability ratio between the old and new strategies The definition is given in formula (10):

[0047] By reconstructing step reward In conjunction with GAE calculations, the adaptive length penalty term alters the gradient direction of the Actor network. When the model attempts to pile up correct but redundant statements, the high penalty value will cause the advantage function of the terminal and preceding states to become negative. Driven by gradient descent, the policy model is forced to streamline its output path, retaining only the core semantic segments with the highest information gain, thereby ensuring accurate alignment of values ​​while mitigating the length bias crisis during the alignment process. A gradient diagram is shown below. Figure 3 As shown.

[0048] 4. Experimental Setup To verify the effectiveness of the dual optimization framework of value alignment in large models that integrates semantic segmentation and length penalty, this chapter details the experimental scheme. The main contents include: the construction of the dataset used for model training and evaluation, the selection criteria for the baseline, and the hyperparameter settings and system hardware configuration at the underlying reinforcement learning level.

[0049] 4.1 Dataset and Evaluation Benchmark In the interactive training phase of preference fitting and proximal policy optimization (PPO) for the reward model, this invention selects a refined subset derived from the large-scale open-source preference dataset UltraFeedback after deep cleaning and binarization as the core training corpus. This dataset contains approximately 60,000 to 70,000 high-quality response pairs from a large language model, stimulated by diverse and challenging instructions, along with fine-grained human or strong AI preference rating labels. To ensure the stability of the training process and the reliability of the final model, this invention performs rigorous heuristic filtering on the original data before experiments, removing low-quality samples with extremely short responses or severe formatting errors, ensuring that the proposed segmented reward model can learn accurate credit allocation strategies within high-quality, noise-free semantic boundaries.

[0050] After completing the model alignment training based on high-quality preference data, a rigorous evaluation of the policy network's ability to follow real instructions was conducted. Given the high degree of openness and creativity in the content generated by large language models, traditional static evaluation metrics based on lexical overlap (such as BLEU, ROUGE, etc.) are no longer sufficient to truly and comprehensively reflect the model's semantic logic quality and value alignment level. This invention selects two of the most challenging and widely accepted automated evaluation benchmarks for large language models in the academic community for comprehensive evaluation. One is AlpacaEval 2.0, an automated evaluation benchmark with extremely high consistency with human preferences, which is used to comprehensively test the model's performance in everyday conversations and general instructions. To strictly eliminate the interference of canonical game theory where policy models rely purely on "word count stacking" to cheat for a higher win rate, the report focuses on the highly convincing length-controlled win rate (LC %) under this benchmark, and uses the average number of characters (#char) as the core auxiliary indicator for quantifying model redundancy. Secondly, the Arena-Hard benchmark uses data deeply sampled from highly challenging real user prompts in Chatbot Arena. This benchmark places extremely high demands on the complex logical reasoning, multi-step instruction adherence, and domain knowledge extraction capabilities of the models participating in the evaluation. This invention records the model's absolute win rate (WR%) and average token generation length (#token) in detail on this benchmark. To ensure the objectivity of the evaluation results and the consistency of variables in the comparative experiments, GPT-4 Turbo was uniformly used as the judge model in all baseline and ablation experiments. Large language models generally exhibit length bias when acting as judges. Suppressing the evaluation bias of the judge model during the evaluation process ensures that the reported win rate improvement is entirely attributable to the actual improvement in semantic density and logical quality of the model, rather than evaluation errors caused by increased word count.

[0051] 4.2 Model and Baseline In the initialization phase of the policy network, this experiment selected mainstream and representative open-source large-scale models as the basic weights. Considering the diversity of architectures, two different types of models were included. First, the Phi-3-Mini-4K-Instruct model with approximately 3.8 billion parameters, representing a lightweight large-scale language model, exhibits extremely superior logical reasoning and instruction compliance capabilities at the same parameter scale. This model primarily serves as the core carrier for infrastructure exploration and ablation experiments, used for rapid iteration and observation of performance evolution under different alignment mechanisms. Second, the Qwen2.5-3B model with approximately 3 billion parameters, an advanced medium-scale open-source language model, possesses richer world knowledge representation and stronger complex instruction processing capabilities. The purpose of introducing this model is to deeply examine the scalability and robustness of the "segmented reward and length penalty" mechanism in a larger parameter space and more complex network distribution.

[0052] In setting the baseline for comparison, a horizontal comparison system covering multiple alignment paradigms is constructed to deeply analyze the impact of the granularity of the credit allocation mechanism on model generation quality and length bias. The experiment first establishes a supervised fine-tuning baseline (SFT Baseline) as the performance bottom line, i.e., the initial state after fine-tuning with only high-quality instructions without any reinforcement learning intervention. Then, a sequence-level reinforcement learning baseline (Sequence-level PPO), representing a common industry approach, is introduced, utilizing its delayed global scalar score to reveal the inherent defects of sparse credit allocation and inducing reward cheating. Simultaneously, to explore the extreme impact of feedback density, the experiment further sets up an extremely fine-grained token-level reinforcement learning baseline (Token-level RLHF PPO) to reveal the potential damage of dense per-token feedback lacking semantic constraints to gradient stability and text coherence. The experiment focuses on demonstrating the proposed semantic segmentation reinforcement learning optimization framework (Segment-level RLHF) and, through the adaptive length penalty coefficient... Perform multigradation ( The sensitivity analysis aims to systematically explore the Pareto optimality between precise alignment and redundancy suppression, thereby completely resolving the length bias crisis in the alignment process.

[0053] 4.3 Experimental Parameters and System Configuration Reinforcement learning proximal policy optimization (PPO) requires the simultaneous loading of four deep neural networks: Actor, Critic, Reference, and Reward, placing extremely demanding requirements on hardware memory and parallel computing capabilities. In this invention, all training, fine-tuning, and inference evaluation tasks are completed on a high-performance computing workstation, with the underlying computing power supported by two NVIDIA RTX 5880 AdaGeneration GPUs with 48GB of VRAM (total cluster VRAM 96GB). The software architecture is based on the Linux operating system and the PyTorch deep learning framework, and deeply integrates the DeepSpeed ​​distributed training engine. By enabling the DeepSpeed ​​ZeRO-2 redundancy elimination optimizer and combining it with bf16 mixed-precision training, the system achieves efficient sharding of model parameters, gradients, and optimizer states, significantly reducing the peak VRAM consumption of long-sequence reinforcement learning and ensuring training stability throughout the entire lifecycle.

[0054] At the level of training strategy and core hyperparameter settings, this invention implements differentiated optimization for base models of different sizes. During the PPO optimization phase, the initial learning rate of the policy network (Actor) is uniformly set to... The learning rate of the Critic network is set to... To ensure rapid convergence of value estimation and provide a stable advantage signal, the global sampling batch size (Rollout Batch Size) was set to 1024 and the global training batch size (Train Batch Size) to 512. For differences in model size, a hierarchical iterative approach was adopted: the lightweight Phi-3-Mini model had 7 episodes to ensure sufficient alignment; while for the Qwen2.5-3B model with a larger number of parameters, given its large vocabulary dimension leading to a significant increase in computational cost per training round, the number of episodes was strictly limited to 2 rounds, considering both training cost and convergence characteristics. This setting not only effectively avoided the disorderly expansion of training time but also prevented catastrophic forgetting caused by reward cheating or a surge in KL divergence in the later stages of alignment.

[0055] For the core fragment-level credit allocation mechanism of this invention, the smoothing parameter of generalized advantage estimation (GAE) is used. Set to 0.95, discount factor A value of 1.0 is used to ensure a reasonable allocation of weights between remote rewards and near-end feedback. To prevent policy updates from exceeding limits, a PPO pruning threshold is set. Both the value clipping threshold and the KL divergence constraint coefficient are strictly limited to 0.2. To prevent the model from deviating from the original language distribution during alignment, the KL divergence constraint coefficient is... Set to 0.01. Dynamic segmentation core parameter information entropy threshold. The global value is fixed at 1.75. Through a rigorous method of controlling variables, this invention ensures the objectivity and reproducibility of the experimental verification to the greatest extent. Detailed reinforcement learning hyperparameters are shown in Table 1.

[0056] Table 1

[0057] 5. Experimental Results and Analysis 5.1 Dynamic Analysis of Reward Model and Strategy Optimization Training In the alignment training of large language models, monitoring the dynamic evolution of training metrics is a core means of evaluating the robustness and convergence quality of the algorithm. This section summarizes and analyzes the training performance of Phi-3-Mini (3.8B) and Qwen2.5-3B pedestal models under different credit allocation mechanisms. The fitting quality of the reward model (RM) and the key statistics of the proximal policy optimization (PPO) stage are shown in Table 2.

[0058] Table 2

[0059] In the Phi-3 model group, after introducing a dynamic semantic segmentation strategy based on information entropy, fragment-level reinforcement learning achieved an accuracy (Acc) of 0.875 and a significant decrease in pre-reward loss (RM), demonstrating that fragment-level feedback can provide more discriminative gradient signals in lightweight models. In the Qwen2.5-3B experimental group, the data exhibited clear game-theoretic characteristics. The traditional token-level method achieved the highest discriminative accuracy, but its reward difference (RewardDiff) amplified sharply to 1.63. Although the Acc of the SEAL-PPO method in this invention fell back to 0.650, it successfully compressed the reward difference to an extremely low level of 0.30. The phenomenon of decreased discriminative accuracy and convergence of the reward difference reveals the successful removal of redundant preference bonuses by the method in this invention. The artificially high accuracy of the token-level method often stems from the model capturing a false statistical correlation between length and preference, i.e., reward hacking. The framework of this invention corrects the gradient direction of the Actor network by injecting an adaptive length penalty into the generalized advantage estimation (GAE) stage, forcing the model to shift from simply pursuing output length to pursuing semantic information density per unit word.

[0060] While static final metrics can reflect the converged model results, they fail to capture the real-time game state between the Actor and Critic networks under complex alignment constraints. To verify the training stability and convergence quality of the proposed method in suppressing length bias, this invention extracts key dynamic monitoring metrics during the PPO iteration process, such as... Figure 4 (a)-(f) and Figure 5 As shown in (a)-(f).

[0061] Comparing the two sets of charts reveals that different pedestal models exhibit differentiated length evolution dynamics on baselines without length penalties. For example... Figure 4 As shown in (c), the Phi-3 model exhibits an explosive increase in generated length with increasing training steps when length constraints are lacking, proving that policy networks can easily exploit loopholes in the reward function to accumulate redundant words to cheat for high scores; while Figure 5 In (c), the Qwen2.5-3B model, benefiting from stronger base characteristics, did not experience length runaway, but the length of its unpenalized baseline declined very slowly, remaining at a high level and failing to actively remove redundant data. The optimal SEAL-PPO configuration of this invention (solid red line) is introduced. Afterwards, the surge in Phi-3's generation length was effectively contained in the early stages of training, while the response length of Qwen2.5-3B was further significantly compressed on the original basis. This result shows that the adaptive length penalty mechanism can not only effectively defend against the length collapse crisis, but also continuously squeeze out redundancy when the model does not drift, ultimately causing the outputs of both models to converge and remain in a healthy range of conciseness and efficiency.

[0062] From the perspective of training stability, Figure 4 (a) and Figure 5 (a) shows the changing trends of the Critic loss for the two models. Benefiting from the segment-level credit allocation mechanism, the loss function of the value network exhibits significantly smaller oscillations than the baseline throughout the training process, and the variance of the advantage function remains at a low level. This demonstrates that dynamic segmented feedback based on information entropy can resolve the problem of sparse credit allocation in traditional alignment, providing a more discriminative gradient signal for the policy network. Finally, in Figure 4 (f) and Figure 5 (f) On the Reward-KL curve, the method of this invention shows a robust and continuous upward trend. In summary, whether on the Phi-3 or Qwen architecture, the joint optimization framework of segment-level reward and adaptive length penalty can achieve higher-density semantic alignment while ensuring training convergence stability, verifying the versatility and robustness of the scheme under different scale models.

[0063] 5.2 Multidimensional Benchmark Evaluation To comprehensively evaluate the instruction adherence quality and generation efficiency under different alignment strategies, this section conducts multidimensional quantitative evaluations on two authoritative benchmarks: AlpacaEval2.0 (general dialogue) and Arena-Hard (complex logic reasoning), and performs qualitative case analysis in conjunction with typical instructions. All quantitative evaluations uniformly use GPT-4 Turbo as the referee model, and a length-balanced cue word strategy is introduced to minimize the evaluation bias of the referee. The main experiment selects the optimal configuration version of each baseline and the method of this invention for horizontal comparison. The results are shown in Table 3 (Multidimensional performance comparison of different alignment strategies on dual automated evaluation benchmarks).

[0064] Table 3

[0065] In the general dialogue instruction compliance evaluation (AlpacaEval 2.0), quantitative data confirms the "length bias" crisis encountered by traditional micro-alignment mechanisms. In the Phi-3-Mini experimental group, the average response length of token-level PPOs ballooned uncontrollably to 1839 characters, exhibiting severe redundancy drift. On the Qwen2.5-3B model, due to the lack of higher-order semantic constraints, the length-controlled win rate (LC %) of token-level PPOs degraded to 13.39% due to control noise, even lower than the unenhanced SFT baseline. In contrast, the method of this invention demonstrates robust constraint capabilities across model architectures. After introducing an adaptive length penalty term, the Phi-3 policy network still achieved a high length control win rate of 22.28% while efficiently controlling the length to 1543 characters. Qwen2.5-3B, while resisting length explosion and reducing the number of characters to 1930, also boosted both the length control win rate and absolute win rate to the best in the group at 16.18% and 16.79%, respectively.

[0066] The Arena-Hard benchmark, which focuses on complex logical reasoning, further highlights the core advantage of this invention's framework in pursuing high semantic density per unit word. Complex reasoning often has extremely low fault tolerance, and lengthy expressions are prone to introducing factual illusions or logical jumps. Data shows that traditional Token-level PPO still tends to output lengthy and inefficient text in long sequence reasoning. In the Phi-3 experimental group, the method of this invention compressed the number of generated words to the lowest level of 631.33 words (a reduction of 25.3% compared to Token-level), while the absolute win rate surged to a global peak of 43.6%. In the Qwen2.5-3B experimental group, the generated length was strictly limited to 738.70 words while maintaining a top win rate of 52.4%. This strongly supports the underlying argument of this invention: the adaptive length penalty mechanism is not merely a superficial implementation of hard word truncation, but rather, through precise end-node gradient backpropagation, it forces the policy network to prioritize the organization of high-order semantic segments with the highest information density using limited carrying space.

[0067] To more intuitively reveal the micro-logic reshaping mechanism behind the differences in macro win rates, this section selects a complex cooking instruction containing clear steps and pitfall avoidance requirements for qualitative analysis. Specific generation comparisons under different alignment strategies are shown in Table 4.

[0068] Table 4

[0069] Qualitative cases reveal the destructive or protective impact of credit allocation granularity on linguistic coherence. Token-level PPOs, in their attempt to manipulate length feedback, lose control of core logic when generating long texts, directly triggering common-sense illusions. In contrast, the method of this invention, under the adaptive constraint of the penalty mechanism, not only eliminates pathological redundancy but also achieves significant enhancement at the macro-logical planning level. This verifies the effectiveness of the proposed framework in mitigating alignment vulnerabilities at the micro-semantic level.

[0070] 5.3 Adaptive Length Penalty and Parameter Sensitivity Analysis In the joint optimization framework, the dynamic piecewise information entropy threshold ( ) and adaptive length penalty coefficient ( These two parameters together constitute the core parameter space for credit allocation and gradient constraints. They are not isolated but jointly determine the density of the underlying credit allocation and the strength of the gradient backpropagation constraint. By comparing the performance of different scaled base models under various hyperparameter configurations, the dynamic control mechanism in the policy network optimization process can be deeply revealed.

[0071] Regarding the granularity boundary of credit allocation, the information entropy threshold It plays a crucial valve role. Setting the threshold too low leads to overly fragmented segments, disrupting the coherence of local phrases; setting the threshold too high degenerates into traditional sparse rewards, reintroducing the credit allocation problem. The evolution trend of average segment length for different pedestal models under various information entropy thresholds is as follows: Figure 6 As shown.

[0072] like Figure 6 As shown, with As the length of the semantic segment gradually increases, the conditions for triggering dynamic segmentation become more stringent, leading to a significant non-linear exponential increase in the average semantic segment length. At times, the model feedback signals are too dense, which in engineering can easily disrupt normal local syntactic structures and introduce high-frequency gradient noise; while when At that point, the fragment length rapidly increased to over 80 units, causing dense feedback to degenerate into traditional sparse rewards, ultimately establishing... As a globally optimal threshold benchmark, under this configuration, the average segment lengths of Phi-3-Mini and Qwen2.5-3B stabilize at around 38.64 and 34.89 tokens, respectively, achieving an optimal trade-off between local semantic integrity and feedback density.

[0073] Based on establishing the optimal semantic segmentation boundary, the length penalty coefficient... As a core weight in the backpropagation of Generalized Advantage Estimation (GAE), it determines the policy network's tolerance for redundant text. To systematically evaluate the dynamic adjustment effect of this parameter, experiments were conducted on a bipedal model with discrete gradients. The SEAL-PPO performance under different conditions and the Segment-PPO baseline without penalty were used for full ablation calculations. Detailed ablation experimental data are shown in Table 5 (different...). Ablation experiment results on the quality and length of model generation.

[0074] Table 5

[0075] As shown by the full ablation dataset, introducing a length penalty is not simply a physical, hard truncation of the text, but a dynamic game process deeply coupled with the advantage function. Without penalty, both models exhibit strong logical coherence in their fragment-level reinforcement learning. However, in complex inference scenarios such as Arena-Hard, the unpenalized version of Phi-3 generates a length as high as 1066.77 words, indicating that the model still exhibits probing behavior by attempting to circumvent the penalty through detailed parsing. To more intuitively observe the dynamic evolution of alignment win rate and response length as the penalty intensity increases, the trends of its multidimensional indicators are shown below. Figure 7 As shown in (a)-(d).

[0076] like Figure 7 As shown, with the penalty coefficient With the smooth introduction of [unclear text], the average generation length of both Phi-3-Mini and Qwen2.5-3B on the dual evaluation benchmarks was robustly and monotonically suppressed. With the elimination of redundancy, the win rate did not show a linear contraction, but rather exhibited a Pareto optimal state with a "V-shaped" rebound and local transitions. In the AlpacaEval general dialogue task, Phi-3 [unclear text]... The win rate peaked at 22.28% with Qwen2.5-3B. It reached its peak win rate (16.18%). In the Arena-Hard task, which highly focuses on core inference steps, Phi-3 achieved... It achieved the highest absolute win rate of 43.6% with 631.33 tokens; Qwen2.5-3B also... Even when the length was compressed to the lowest of the entire game at 738.70 words, the absolute win rate remained high at 52.4%. The adaptive length penalty mechanism successfully truncated the false benefits of canonical games by dynamically applying relative load costs to terminal nodes, forcing the policy network to prioritize organizing high-order semantic segments with the highest information density using limited load space.

[0077] 5.4 Training and System Resource Analysis This section evaluates the system resource utilization efficiency of the fragment-level dense feedback mechanism in actual training from the perspective of engineering overhead. The experiment is based on a hardware and software platform with two NVIDIA RTX 5880 Ada GPUs with 48GB of video memory, and deploys the DeepSpeed ​​Stage-2 distributed optimization strategy. To verify the system robustness of the algorithm under high load, this section uses the Qwen2.5-3B model with a larger parameter scale as an example, recording its underlying video memory allocation, computing power utilization, and temperature monitoring indicators throughout the distributed training process, such as... Figure 8 As shown in (a)-(c).

[0078] Thanks to the segment-level feedback mechanism's structured segmentation of the sequence generation cycle, the model exhibits excellent stability in memory usage during the training phase. Compared to the frequent memory overflow (OOM) risk easily caused by long-sequence global gradient backpropagation in traditional token-level PPO training, the joint optimization method proposed in this invention, while maintaining the integrity of the policy gradient, makes the memory allocation curve exhibit a highly consistent step-like periodic pattern. Various adaptive length penalty coefficients of different intensities are introduced into the underlying computation graph. The system's video memory usage remained within a stable range, without any uncontrolled surge in video memory usage due to the additional backpropagation calculations of the dominant function. Figure 8 (b) and Figure 8As shown in the hardware performance trend in (c), the underlying computing power of the model was fully and evenly allocated during distributed training, remaining at a high saturation level for most of the time. Along with the efficient release of computing power, the GPU core temperature remained stable within a safe and controllable engineering range throughout the process. Monitoring data demonstrates that this joint optimization framework not only significantly improves the alignment quality of the model at the macro-algorithm level, but also successfully controls the peak memory pressure of a single reinforcement learning round at the micro-level, greatly optimizing the hardware throughput efficiency of long sequence alignment. This demonstrates the low-cost deployment potential and high engineering practical value of this joint optimization framework in real-world large-scale model production environments.

[0079] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a large language model alignment method based on semantic segmentation and length penalty as described above.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0081] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for aligning large language models based on semantic segmentation and length penalty, characterized in that, Includes the following steps: S1. Construct a Markov decision process, and based on information entropy, perform dynamic semantic segmentation on the response sequence generated by the large language model, dividing the response sequence into several non-overlapping segments with independent semantic integrity, i.e., semantic segments. S2. Based on a pre-trained fragment-level reward model, each semantic fragment is given an independent quality score, i.e., a fragment reward, and the sequence baseline score is obtained by aggregating the fragment rewards. S3. Construct an adaptive length penalty function, and calculate the length penalty based on the relative difference between the actual length of the currently generated sequence and the baseline reference length for the current prompt word; S4. The fragment reward and length penalty are integrated to reconstruct the step reward function in the near-end policy optimization algorithm. The length penalty at the end is backpropagated to the previous generation state through generalized advantage estimation to update the parameters of the large language model, improve the semantic information density of a single word in the model, and achieve alignment of the large language model.

2. The method for aligning large language models based on semantic segmentation and length penalty according to claim 1, characterized in that, In step S1, dynamic semantic segmentation of the response sequence generated by the large language model is performed, specifically including: Computational large language model in generating the i-th lexical Local Predicted Distribution of Shannon Entropy : in, For vocabulary Chinese word elements Given prompt words and above The predicted probability is as follows; Indicator function that triggers segmentation The definition is as follows: in, This is a backup rule set that includes strong termination symbols specific to natural language. The empirical information entropy threshold is used; through the above dynamic algorithm, the total length is... The original word sequence was transformed A sequence of non-overlapping semantic segments .

3. The method for aligning large language models based on semantic segmentation and length penalty according to claim 2, characterized in that... In S1, the empirical information entropy threshold Setting it to 1.75 stabilizes the length of the segmented semantic fragments between 10 and 40 lexical units.

4. The method for aligning large language models based on semantic segmentation and length penalty according to claim 3, characterized in that... In S2, the sequence benchmark scores are aggregated using average pooling, as shown in the following formula: in, For the sequence baseline score, For the fragment-level reward model, the k-th fragment The rating.

5. The method for aligning large language models based on semantic segmentation and length penalty according to claim 4, characterized in that... In S3, the adaptive length penalty function is constructed as follows: in, This serves as a baseline reference length for the current suggested word. The total number of actual lexical units generated for a large language model; when When the penalty value is zero, it ensures the normal expression space of the model; when At that time, the penalty value increases linearly with the excess ratio.

6. The method for aligning large language models based on semantic segmentation and length penalty according to claim 5, characterized in that... In step S4, the step reward function in the near-end policy optimization algorithm is reconstructed, as shown in the following formula: in, The KL divergence constraint coefficient is... For the indicator function activated at the end node, For the current large language model, For reference to large language models, This is the length penalty coefficient. Based on this, the fragment-level dominance function is calculated using generalized dominance estimation. The formula is as follows: in, For time step Single-step timing difference error, This is a value estimate of the expected return of a state by the Critic value network. As a discount factor, This is the GAE smoothing parameter, used to balance the bias and variance of the advantage estimate. This represents the state at time step k.

7. The method for aligning large language models based on semantic segmentation and length penalty according to claim 6, characterized in that, The length penalty coefficient The range of values ​​is This is used to adjust the game-theoretic balance between alignment quality and information density; for general dialogue tasks and complex logical reasoning tasks, it is used to adjust... Achieve Pareto optimality for both generation quality and length.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a large language model alignment method based on semantic segmentation and length penalty as described in any one of claims 1-7.

Images