A large language model alignment method and system based on contrastive online policy distillation

Abstract

The application relates to an online contrast-based policy distillation-based large language model alignment method and system, and relates to the field of artificial intelligence. The application solves the problem that the training signal is invalid due to the teacher target conflict in the existing multi-teacher online policy distillation (OPD) method. The method constructs a training prompt set containing a first domain and a second domain prompt subset, and configures a first domain teacher model and a second domain teacher model corresponding to the two domains respectively. A student model generates a response sequence by performing online policy sampling on each training prompt. A top-k truncated forward KL divergence is used at a word-by-word generation position. The word-by-word KL distances of the positive and negative teacher models and the student model are calculated respectively. A word-by-word contrast distance is obtained according to the two types of KL distances. A COPD contrast distillation loss function is constructed by fusing an anchoring constraint term and a sigmoid adaptive contrast term. The student model parameters are updated only through the loss gradient, and the teacher model is frozen throughout the process, so that precise and stable model alignment optimization is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for aligning large language models based on contrastive online policy distillation. Background Technology

[0002] Large language models typically acquire specific target capabilities in the post-training phase through supervised fine-tuning (SFT) or reinforcement learning based on human feedback (RLHF). However, these training paradigms have inherent limitations: supervised fine-tuning uses static training targets, and the training performance deteriorates significantly when the target distribution deviates from the actual input distribution encountered by the model during inference; while reinforcement learning-based methods can maintain distribution consistency through online sampling, each training step requires generating a complete response and calculating a reward, and the computational cost increases dramatically with the model size.

[0003] On-Policy Distillation (OPD) offers a compromise between the two approaches: the student model is trained based on its own generated trajectories, while simultaneously obtaining word-by-word distributed supervision signals from frozen teacher models, significantly reducing computational costs while maintaining policy training stability. This efficiency makes OPD a natural choice for integrating the complementary capabilities of multiple specialized teachers into a single student model. However, when there are structural conflicts in the optimization objectives of multiple teacher models, existing multi-teacher OPD methods cannot transform teacher disagreements into effective directional training signals.

[0004] Taking the typical application scenario of security alignment as an example: the security teacher can reliably reject harmful requests by fine-tuning adversarial instruction data, but it suffers from over-rejection of benign requests; the practical teacher retains strong general capabilities but lacks security safeguards. For harmful prompts, the security teacher encodes the expected rejection behavior, while the practical teacher encodes the dangerous compliance behaviors that should be suppressed; for benign prompts, their roles are reversed. Neither simple mean nor routing strategies can model this asymmetric, domain-dependent preference.

[0005] Therefore, there is an urgent need for a new multi-teacher OPD method that can transform teacher disagreements into directional training signals through domain-related role assignments, and simultaneously achieve competitive levels on two conflicting optimization objectives. Summary of the Invention

[0006] This invention aims to overcome the problem of invalid training signals caused by teacher goal conflict in existing online policy distillation (OPD) methods involving multiple teachers. To address this, this invention proposes a large language model alignment method and system based on contrastive online policy distillation. The invention achieves the above-mentioned technical problem through the following technical solution: Option 1: This invention proposes a large language model alignment method based on contrastive online policy distillation, the method comprising the following steps: Step 1: Construct the training cue set The training cue set Including the first domain hint subset Second domain hint subset And train two domain teacher models, the two domain teacher models including a first domain teacher model. Second Domain Teacher Model For each training cue ; Step 2: Student model to be trained For each training tip Perform online policy sampling to generate the corresponding model response sequence. ,satisfy For each word position t in the response sequence generation process, top-k truncated forward KL divergence is used to calculate the word-by-word probability distribution distance between the positive teacher model and the student model, and between the negative teacher model and the student model, respectively, to obtain the positive word-by-word KL distance. Negative word-by-word KL distance ; Step 3: Based on the forward word-by-word KL distance Negative word-by-word KL distance Calculate word-by-word comparison distance, fuse anchoring constraint terms and sigmoid adaptive comparison terms, and construct an end-to-end comparison distillation loss function COPD; based on the gradient of the loss function, backpropagate and update only the student model. Network parameters.

[0007] Furthermore, a preferred implementation is provided, wherein the first domain teacher model The model is obtained by supervised fine-tuning of the base model on the first domain dataset, and is used to represent the target behavior output capability in the first domain. The second domain teacher model is used. The initial basic model for the frozen state Alternatively, the model can be fine-tuned for a specific domain, with parameters frozen throughout the process and not used for training updates.

[0008] Furthermore, in a preferred embodiment, step 1 further includes dynamically assigning positive teachers based on the domain tag. and negative teachers :Right now,

[0009] when Belongs to a subset At that time, the first domain teacher model For positive teachers, second-domain teacher models For negative teachers; when Belongs to a subset At this time, roles are reversed; positive teachers encode the desired target behaviors in the current domain, while negative teachers encode behaviors that should be suppressed.

[0010] Furthermore, a preferred embodiment is provided, wherein the truncated forward KL divergence calculation in step 2 is specifically as follows: In each word position ,set up The student model with the highest probability A set consisting of 1 word elements; both teacher and student distributions are in After renormalization, the forward KL divergence is calculated; the truncation operation concentrates the supervision signal in the word region to which both models assign significant probability quality. The value of is an integer ranging from 8 to 64.

[0011] Furthermore, a preferred embodiment is provided in which the forward word-by-word KL distance is calculated in step 2. Negative word-by-word KL distance The method is as follows:

[0012]

[0013] in, Cut off the KL divergence for top-k. As a positive reference strategy, A negative reference strategy, The conditional probability distribution of the main model to be optimized is shown below. For the original input of the i-th sample, Let i be the model output sequence generated for the i-th sample before time t, where i is the sample index. This is the location index of the token generated by autoregression.

[0014] Furthermore, a preferred embodiment is provided, wherein the method for constructing the end-to-end comparative distillation loss function COPD by fusing the anchoring constraint term and the sigmoid adaptive contrast term in step 3 is as follows:

[0015] in, For the sigmoid function, To compare sensitivity parameters, This is the anchoring coefficient.

[0016] Furthermore, a preferred embodiment is provided, wherein the second domain teacher model Two optional configuration options are available: Mode 1: Directly adopt the initial model π0 of the basic model before alignment training with completely frozen parameters, and fully preserve the original general knowledge distribution of the basic model; Mode 2 adopts a professional model with minor adjustments in mathematics, programming, and translation. While maintaining the dynamic role allocation framework of dual teachers, the professional domain capabilities are transferred to the student model through comparative distillation.

[0017] Option 2: A large language model alignment system based on contrastive online strategy distillation, the system comprising: The dataset building module is used to build the training cue set. The training cue set Including the first domain hint subset Second domain hint subset And train two domain teacher models, the two domain teacher models including a first domain teacher model. Second Domain Teacher Model For each training cue ; The probability distribution distance calculation module is used for the student model to be trained. For each training tip Perform online policy sampling to generate the corresponding model response sequence. ,satisfy For each word position t in the response sequence generation process, top-k truncated forward KL divergence is used to calculate the word-by-word probability distribution distance between the positive teacher model and the student model, and between the negative teacher model and the student model, respectively, to obtain the positive word-by-word KL distance. Negative word-by-word KL distance ; Alignment module, used to align based on the forward word-by-word KL distance Negative word-by-word KL distance Calculate word-by-word comparison distance, fuse anchoring constraint terms and sigmoid adaptive comparison terms, and construct an end-to-end comparison distillation loss function COPD; based on the gradient of the loss function, backpropagate and update only the student model. Network parameters.

[0018] Option 3: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in Option 1.

[0019] Option 4: A computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in Option 1.

[0020] The advantages of this invention are: This invention presents a large language model alignment method based on comparative online policy distillation, achieving multi-domain adaptive alignment optimization of large language models while balancing domain alignment accuracy and model generalization ability. The invention dynamically switches between positive and negative teacher roles for alignment prompt samples in different domains, enabling each teacher model to output compliant alignment behavior and provide positive guidance constraints in its domain of expertise, and output suppressed behavior and provide negative correction constraints in its non-domain domain. This overcomes the limitations of fixed teacher roles, singular ranking of superiority and inferiority, and rigid supervision direction in traditional large language model alignment training. It achieves differentiated and adaptive behavior alignment correction based on domain relative preferences, accurately completing safe alignment and compliant output constraints for professional domain tasks, effectively suppressing model illusions, harmful outputs, and domain-biased outputs, while also being compatible with general knowledge retention. Furthermore, the teacher architecture can adapt to frozen native base models and specialized fine-tuned models for mathematics, programming, and translation, flexibly adapting to multi-domain large language model alignment scenarios without altering the overall alignment framework, demonstrating strong versatility and scalability.

[0021] The large language model alignment method based on comparative online policy distillation described in this invention utilizes Top-k truncated word-by-word KL online policy calculation to effectively purify the alignment training supervision signal and improve the iterative convergence stability and alignment accuracy of the large language model. Traditional large model online policy alignment methods often employ global undifferentiated KL divergence supervision, calculating gradients for all lexical units in the generated sequence. However, large language models generate a large number of low-probability long-tail lexical units, which are highly random, have limited effective supervision information, and high noise levels, easily leading to problems such as training gradient oscillations, slow convergence, and distortion of alignment supervision signals. In the word-by-word iteration process of online policy sampling and response generation, this invention selects only the top k core lexical units with the highest probability from the student model, re-normalizes them, and calculates the KL distance between the teacher and student models. This highly focuses the alignment supervision signal on the core generation decision region of the model, thoroughly filtering out invalid noise interference from long-tail lexical units, significantly purifying the word-by-word alignment gradient, and allowing model training to accurately focus on learning and correcting compliant generation behavior. This significantly improves the convergence speed, stability, and final output alignment quality of the large language model alignment training, ensuring the logic and coherence of the generated text.

[0022] The large language model alignment method based on contrastive online policy distillation described in this invention uses a COPD loss function that fuses anchor terms and Sigmoid adaptive contrast terms. Through a positive KL anchor term, it provides rigid alignment constraints to the large language model, forcing the model to learn the standard and compliant output behavior of a positive teacher, thus fundamentally solving the model degradation problem of pure contrastive training. Simultaneously, relying on the Sigmoid adaptive contrast mechanism, it dynamically adjusts the training gradient intensity based on the differences between teacher and student model outputs. It strengthens the alignment training signal at keyword positions with significant output deviations and automatically decays the gradient after the model alignment effect reaches the target, achieving word-by-word refined and adaptive alignment optimization, significantly improving the compliant output capability of the large language model in various scenarios.

[0023] The alignment training architecture of the large language model based on contrastive online policy distillation described in this invention is lightweight, has low computational overhead, and is highly practical and applicable in engineering. This invention freezes all parameters of the dual-teacher model throughout the entire training process, only iteratively updating the parameters of the student large language model to be aligned. This significantly reduces the memory usage and computational consumption of alignment training, avoiding the problem of increased training complexity caused by simultaneous optimization of multiple model parameters. Simultaneously, the domain labels of the training samples are directly obtained from the native classification annotations of the prompt set, eliminating the need for additional training of auxiliary networks such as domain classifiers and discriminators, saving additional model training and data preprocessing overhead, and simplifying the overall alignment training chain. The entire alignment framework is concise, efficient, and robust, adaptable to fine-tuning alignment, safe alignment, and specialized capability alignment scenarios of various mainstream large language models, effectively balancing alignment performance, training efficiency, and engineering implementation costs. Attached Figure Description

[0024] Figure 1 This is a block diagram illustrating the principle of the large language model alignment method based on contrastive online policy distillation as described in Implementation Method 1.

[0025] Figure 2 This is a schematic diagram of the general benchmark test results described in Implementation Method 1. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0027] Implementation Method 1, see [link] Figures 1 to 2This embodiment proposes a large language model alignment method based on contrastive online policy distillation, which adopts a phased training design. The first phase involves constructing a domain dataset and training corresponding domain teachers. The second phase assigns positive and negative teachers to each domain. In the third phase, the student model simultaneously receives supervision and scoring from both teachers on its self-generated online policy response trajectory and calculates the KL divergence. Finally, the anchored contrastive loss function is used as the target loss function to update the parameters. The method includes the following steps: Step 1: Training Data and Teacher Construction The method described in this embodiment is based on a prompt set. Training was conducted, among which For the first domain hint subset, The second domain is a subset of the hints, and the proportions of the two are balanced.

[0028] Teacher Model This model is obtained by supervised fine-tuning of the base model on data from the first domain, and it excels at target behavior in the first domain. Taking the safe alignment application as an example, harmful and benign mixed samples from the adversarial instruction dataset can be used to generate responses from a larger-scale model, which are then filtered and fine-tuned on this dataset.

[0029] Teacher Model The frozen initial model before alignment can be used directly. No additional training is required to preserve the complete knowledge distribution of the basic model; alternatively, it can be replaced with a professional teacher model that has been finely tuned for a specific field (such as a finely tuned model for mathematics, programming, translation, etc.). While keeping the teacher role allocation framework unchanged, the specific capabilities of that field are transferred to the student model through positive teacher role distillation.

[0030] Domain labels are provided directly by the domain classification annotations of the cue set itself, without the need for additional classifier training.

[0031] 2.3 Step Two: Assignment of Domain Teachers For each training cue Dynamically assign positive teachers based on their domain tags. and negative teachers :

[0032] when Belongs to a subset At that time, the teacher For positive teachers, teachers For negative teachers; when Belongs to a subset In this process, roles are reversed. Positive teachers encode the expected target behaviors in their current domain, while negative teachers encode behaviors that should be suppressed. Each teacher plays a positive guiding role in their professional domain and a negative constraining role in their non-professional domain. These roles express relative, domain-related preferences rather than fixed quality rankings.

[0033] 2.4 Step 3: Online Strategy Sampling and Successive KL Distance Calculation Student Model For the current prompt Perform online policy sampling to generate response sequences. At each word position in the generated response The word-by-word distances between positive and negative teachers and students were calculated using truncated forward KL divergence.

[0034]

[0035] The truncation operation selects the student model with the highest probability. Each term is renormalized on this subset, and the KL divergence is calculated to concentrate the supervision signal on the term regions that are assigned significant probability quality in both models, thus avoiding noise interference from long-tail terms. The value of is an integer ranging from 8 to 64.

[0036] Step 4: Anchoring Comparison Loss Calculation and Parameter Update The difference between the KL distances of positive and negative teachers As a word-by-word comparison signal, the anchor term and the sigmoid comparison term together constitute the complete objective function:

[0037] in For the sigmoid function, To compare sensitivity parameters, This is the anchoring factor. Anchoring term. It provides direct distillation pressure toward the positive teacher, preventing the student model from achieving loss reduction by moving away from the negative teacher instead of closer to the positive teacher under a purely contrastive objective, and provides stable distillation supervision when the two teachers are highly consistent and the contrastive signal is weak. The sigmoid contrastive term will... The training signal is concentrated at the word positions with the greatest teacher disagreement, treated as a word-by-word affinity score. The contrastive gradient automatically decays when the student approaches a positive teacher and increases when the student approaches a negative teacher, achieving adaptive adjustment of the training signal strength. The gradient based on this loss function only updates the student model. The parameters of the two teacher models are frozen throughout the training process.

[0038] This invention uses Qwen3-4B-Instruct as the student model and conducts experimental verification in a safe alignment scenario, encompassing two task domains: a safe refusal task and a general response task. The teacher model (safe teacher) is obtained by fine-tuning the base model three times on the WildJailbreak dataset. The training set contains 15,000 mixed harmful and benign samples with a 2:1 ratio. The teacher model (practical teacher) directly uses the frozen initial student model without additional training. The distillation training set is obtained by sampling 20,000 prompts proportionally from the remaining WildJailbreak data, with half harmful and half benign prompts. This training set is shared by all comparison methods. During training, the anchoring coefficient and contrast sensitivity parameters are set uniformly, and all methods are trained for two rounds. All models are evaluated on four benchmarks: WildJailbreak (WJ), BeaverTails (BT), AEGIS2, and CoCoNot (CCN). The results report the attack success rate (ASR↓, the lower the better) of harmful subsets and the false rejection rate (FRR↓, the lower the better) of benign subsets.

[0039] Table 1 shows the weighted average metrics of different schemes on various datasets. The results were evaluated using GPT-OSS-Safeguard-20B. ASR represents the attack success rate, FRR represents the false rejection rate, bold numbers indicate the best performance, and underlined numbers indicate the second best performance.

[0040]

[0041] As shown in Table 1, COPD is the only method that simultaneously achieves competitive safety and a low false rejection rate: its average ASR is 2.45%, surpassing all methods, including SFT, on out-of-distribution benchmarks (AEGIS2, CoCoNot, BeaverTails); its average FRR is 19.04%, the lowest among all aligned methods, nearly half that of single-teacher OPD. Furthermore, as... Figure 2 As shown, COPD maintained all metrics within 2% of the base model on the general competence benchmarks (MMLU, GPQA) and improved GPQApass@4 by 5.05%, verifying the effectiveness of the domain teacher assignment mechanism in achieving competitive safety without sacrificing benefit.

[0042] The model training dataset described in this embodiment is divided into two major sample sets: harmful instruction data and benign instruction data. The two types of data were collected and constructed from multiple business domains, with a total of 4056 and 7914 samples respectively. The sample quantity and proportion distribution of each domain are as follows: I. Distribution of Harmful Command Data Samples (Total Samples: 4056) Harmful instructions cover 17 risk areas, with significant differences in the concentration of sample distribution: High-risk areas: Hate speech and discrimination had the most samples, with 895 items, accounting for 22.07% of the total harmful data; followed by politics and ideology (429 items, 10.58%), violence and terrorism (394 items, 9.71%), and fraud, theft and economic crimes (383 items, 9.44%). These four high-risk areas accounted for more than 50% of the total, making them the core control scenarios for harmful content identification.

[0043] The following four categories of risk areas accounted for 6% to 9% of the total sample, representing secondary risk scenarios: healthcare, cybercrime and hacking, gaming and fiction, and pornography and sex-related content. Low-risk areas: The sample proportions of personal privacy leaks, drug abuse, legal compliance, computer technology, business marketing, environmental safety, finance, and education are all less than 4%, with education having the lowest proportion (1.04%). The overall number of low-risk samples is relatively small.

[0044] II. Distribution of Benign Instruction Data Samples (Total Samples: 7914) The positive instructions cover a total of 18 regular business areas, with samples concentrated on common daily needs scenarios: The core mainstream areas are: gaming and entertainment and fiction, which are the largest sample sources, totaling 2,135 items, accounting for 26.98% of the positive data; education and academia, legal compliance, healthcare, business marketing, and computer technology are the secondary core areas, accounting for 7% to 11.5% of each category, and are the main data support for training the model's positive interaction capabilities. General and routine areas: Other comprehensive categories, political ideology, environmental public safety, finance and economics, etc. account for 2% to 7%, covering basic needs for daily consultation and industry services; Areas with extremely low percentages: The number of benign samples in risk-related areas such as fraud, privacy breaches, hate speech, cyberbullying, pornography, and drugs is extremely small, with each category accounting for less than 1.3%. Among them, there are only 16 benign samples of drug abuse (0.20%), which are only used to distinguish between legitimate and compliant consultations and illegal and irregular requests.

[0045] III. Implementation and Application Instructions for Dataset Distribution Characteristics Risk differentiation training is based on the following: There are sufficient samples of high-risk speech, violence, and economic crimes in the harmful samples, which can support the model to accurately identify various illegal inducement instructions; the number of similar fields in the benign samples is extremely low, forming a clear difference between positive and negative samples, which facilitates the model to learn the boundary features between legitimate inquiries and harmful demands.

[0046] Positive capability training is based on the following: the positive samples mainly consist of everyday and professional compliance needs in entertainment, education, medical care, and business, with a larger overall sample size to ensure the basic interactive effect of the model's positive functions such as routine question answering and professional consultation.

[0047] Training ratio implementation logic: The two types of data are independently labeled and stored in separate databases. During the training phase, the positive and negative sample mixing ratio is configured according to business needs. Based on the differentiated sample ratios in various fields, the ability to intercept high-risk and harmful content is specifically strengthened, while ensuring the response quality of normal compliance instructions.

[0048] In summary, this invention achieves target performance close to the strongest domain-specific baseline with only the computational overhead of standard OPD, while maintaining or exceeding the generality of the base model. It is the only method that can simultaneously achieve two conflicting optimization objectives.

[0049] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0050] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for aligning large language models based on contrastive online policy distillation, characterized in that, The method includes the following steps: Step 1: Construct the training cue set The training cue set Including the first domain hint subset Second domain hint subset And train two domain teacher models, the two domain teacher models including a first domain teacher model. Second Domain Teacher Model For each training cue ; Step 2: Student model to be trained For each training tip Perform online policy sampling to generate the corresponding model response sequence. ,satisfy For each word position t in the response sequence generation process, top-k truncated forward KL divergence is used to calculate the word-by-word probability distribution distance between the positive teacher model and the student model, and between the negative teacher model and the student model, respectively, to obtain the positive word-by-word KL distance. Negative word-by-word KL distance ; Step 3: Based on the forward word-by-word KL distance Negative word-by-word KL distance Calculate word-by-word comparison distance, fuse anchoring constraint terms and sigmoid adaptive comparison terms, and construct an end-to-end comparison distillation loss function COPD; based on the gradient of the loss function, backpropagate and update only the student model. Network parameters.

2. The large language model alignment method based on contrastive online policy distillation according to claim 1, characterized in that, First domain teacher model The model is obtained by supervised fine-tuning of the base model on the first domain dataset, and is used to represent the target behavior output capability in the first domain. The second domain teacher model is used. The initial basic model for the frozen state Alternatively, the model can be fine-tuned for a specific domain, with parameters frozen throughout the process and not used for training updates.

3. The large language model alignment method based on contrastive online policy distillation according to claim 1, characterized in that, Step 1 also includes dynamically assigning positive teachers based on their domain tags. and negative teachers :Right now, when Belongs to a subset At that time, the first domain teacher model For positive teachers, second-domain teacher models For negative teachers; when Belongs to a subset At this time, roles are reversed; positive teachers encode the desired target behaviors in the current domain, while negative teachers encode behaviors that should be suppressed.

4. The large language model alignment method based on contrastive online policy distillation according to claim 1, characterized in that, The specific calculation of the truncated forward KL divergence in step 2 is as follows: In each word position ,set up The student model with the highest probability A set consisting of 1 word elements; both teacher and student distributions are in After renormalization, the forward KL divergence is calculated; the truncation operation concentrates the supervision signal in the word region to which both models assign significant probability quality. The value of is an integer ranging from 8 to 64.

5. The large language model alignment method based on contrastive online policy distillation according to claim 1, characterized in that, In step 2, calculate the forward word-by-word KL distance. Negative word-by-word KL distance The method is as follows: in, Cut off the KL divergence for top-k. As a positive reference strategy, A negative reference strategy, The conditional probability distribution of the main model to be optimized is shown below. For the original input of the i-th sample, Let i be the model output sequence generated for the i-th sample before time t, where i is the sample index. This is the location index of the token generated by autoregression.

6. The large language model alignment method based on contrastive online policy distillation according to claim 1, characterized in that, In step 3, the method for fusing the anchoring constraint term and the sigmoid adaptive contrast term to construct the end-to-end contrastive distillation loss function COPD is as follows: in, For the sigmoid function, To compare sensitivity parameters, This is the anchoring coefficient.

7. The method for aligning large language models based on contrastive online policy distillation according to claim 1, characterized in that, Second domain teacher model Two optional configuration options are available: Mode 1: Directly adopt the initial model π0 of the basic model before alignment training with completely frozen parameters, and fully preserve the original general knowledge distribution of the basic model; Mode 2 adopts a professional model with minor adjustments in mathematics, programming, and translation. While maintaining the dynamic role allocation framework of dual teachers, the professional domain capabilities are transferred to the student model through comparative distillation.

8. A large language model alignment system based on contrastive online strategy distillation, characterized in that, The system includes: The dataset building module is used to build the training cue set. The training cue set Including the first domain hint subset Second domain hint subset And train two domain teacher models, the two domain teacher models including a first domain teacher model. Second Domain Teacher Model For each training cue ; The probability distribution distance calculation module is used for the student model to be trained. For each training tip Perform online policy sampling to generate the corresponding model response sequence. ,satisfy For each word position t in the response sequence generation process, top-k truncated forward KL divergence is used to calculate the word-by-word probability distribution distance between the positive teacher model and the student model, and between the negative teacher model and the student model, respectively, to obtain the positive word-by-word KL distance. Negative word-by-word KL distance ; Alignment module, used to align based on the forward word-by-word KL distance Negative word-by-word KL distance Calculate word-by-word comparison distance, fuse anchoring constraint terms and sigmoid adaptive comparison terms, and construct an end-to-end comparison distillation loss function COPD; based on the gradient of the loss function, backpropagate and update only the student model. Network parameters.

9. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

10. A computer device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.

Images