Unsupervised preference alignment method, system, medium for language models based on information entropy

Abstract

The application relates to the technical field of artificial intelligence, and discloses a language model unsupervised preference alignment method and system based on information entropy and a medium, which comprises the following steps: sampling the same instruction multiple times, obtaining a candidate response set generated by a language model and a probability distribution corresponding to each Token in each candidate response; calculating the information entropy of each Token based on the probability distribution, and generating a risk evaluation index of each response segment in each candidate response by fusing the semantic deviation degree between multiple candidate responses in the candidate response set; unsupervisedly dividing positive preference samples and negative preference samples from the candidate response set based on the risk evaluation index; and using the positive preference samples and the negative preference samples as training data, using the risk index corresponding to the negative preference samples as a weight factor, and fine-tuning the preference optimization of the language model. The application realizes completely unsupervised preference data self-construction, does not require any manual or external labeling, and thus greatly reduces the alignment cost.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an unsupervised preference alignment method, system, and medium for language models based on information entropy. It specifically relates to illusion detection, unsupervised learning, uncertainty quantification, real-time inference calibration, and model preference alignment techniques for large-scale language models, belonging to the semantic processing category in natural language processing. Background Technology

[0002] Current large language model alignment primarily relies on human feedback reinforcement learning (RLHF) or direct preference optimization (DPO). However, existing techniques suffer from the following drawbacks:

[0003] First, the annotation cost is extremely high. Traditional methods require a large amount of manual or artificial intelligence to rank the model outputs, and the high cost and long cycle of data acquisition and verification limit the large-scale application of model alignment technology.

[0004] Second, knowledge boundary conflict. When forced to align with external preferences, the model is prone to factual illusions in its own knowledge blind spots in order to cater to external preferences, that is, "talking nonsense in a serious manner", which leads to a decrease in the factual fidelity of the model.

[0005] Third, static alignment. Existing methods rely on a fixed preference dataset and cannot dynamically adjust the alignment strategy based on the real-time confidence and uncertainty state during model generation, making it difficult to adapt to changes in the model's knowledge boundary.

[0006] Fourth, hallucinations are uncontrollable. Existing technologies lack a unified and quantifiable standard for judging hallucinations, and high-risk outputs cannot be intercepted in real time during the inference phase, making it difficult to effectively control hallucination problems during the deployment phase.

[0007] Although existing technologies include schemes that use information entropy for hallucination detection or entropy for preference data filtering, as well as text analysis methods based on word frequency statistics, none of these schemes have formed a closed-loop unsupervised alignment system of "entropy quantification of uncertainty → unsupervised self-construction of preferences → entropy-weighted preference alignment → real-time calibration at the inference end". They cannot achieve the technical effects of reducing hallucinations, maintaining factual accuracy, and adapting to knowledge boundaries simultaneously under the premise of zero annotation. Summary of the Invention

[0008] This invention aims to overcome the shortcomings of existing technologies, such as reliance on manual or external annotation, susceptibility to knowledge boundary illusions, and the inability of static alignment to adapt. It provides a completely unsupervised, zero-annotation-cost unsupervised preference alignment method, system, and medium based on information entropy for language models. This method automatically identifies illusions, constructs preference data, completes closed-loop preference alignment, and supports real-time illusion interception at the inference end, based on the probability distribution and information entropy features generated by the model itself.

[0009] This invention is achieved through the following technical solution:

[0010] An unsupervised preference alignment method for language models based on information entropy includes:

[0011] The same instruction is sampled multiple times to obtain a set of candidate responses generated by the language model and the probability distribution of each token in each candidate response.

[0012] The information entropy of each token is calculated based on the probability distribution, and the semantic deviation between multiple candidate responses in the candidate response set is integrated to generate a risk assessment index for each response fragment in each candidate response.

[0013] Based on the aforementioned risk assessment indicators, positive preference samples and negative preference samples are unsupervisedly divided from the candidate response set;

[0014] The language model is fine-tuned by using the positive and negative preference samples as training data and the risk index corresponding to the negative preference samples as a weighting factor.

[0015] As an optimization, the specific process for generating risk assessment indicators for each response fragment in each of the candidate responses is as follows:

[0016] The information entropy is calculated based on the probability distribution of each token in each candidate response, and the information entropy is smoothed by a sliding window to obtain the average sliding window entropy of each token position.

[0017] Calculate the semantic deviation between the response fragment at each semantic position in each candidate response and the consensus center at the corresponding semantic position, wherein the consensus center is jointly determined by all candidate responses in the candidate response set;

[0018] The risk assessment index is generated by multiplying the mean of the sliding window entropy by the degree of semantic deviation.

[0019] As an optimization, the specific process of unsupervisedly dividing positive preference samples and negative preference samples from the candidate response set based on the risk assessment index is as follows:

[0020] The candidate response with the lowest overall entropy value and located at the semantic cluster center in the candidate response set is taken as the positive preference sample;

[0021] Identify response segments in the candidate response set whose risk assessment indicators exceed a preset threshold, and construct the negative preference sample containing hallucination tendencies by performing token-level perturbation on the response segments or selecting a high-entropy sampling path.

[0022] As an optimization, the loss function used when fine-tuning the language model for preference optimization is the entropy-weighted direct preference optimization loss function, which has the following form:

[0023] ;

[0024] in, Entropy-weighted direct preference optimization loss, For expectation operator, It is the Sigmoid activation function. Divergence constraint hyperparameter, is the entropy weighting factor for negatively biased samples, used to dynamically adjust the penalty intensity based on the risk assessment index of the negatively biased samples; x is the input instruction. For positive preference samples, This is a negative preference sample. Generate the current language model under input x The probability, Generate a reference language model (a frozen copy before fine-tuning) under input x. The probability, Generate the current language model under input x The probability, Generate a reference language model based on input x. The probability of.

[0025] As an optimization, it also includes:

[0026] The information entropy threshold is dynamically adjusted based on the confusion level of the input command.

[0027] When the language model generates candidate responses token by token, the information entropy of the current token of the candidate response is calculated in real time.

[0028] When the information entropy exceeds the dynamically adjusted information entropy threshold, at least one intervention operation is triggered, including truncation generation, resampling, or returning a rejected answer.

[0029] As an optimization, the information entropy threshold is dynamically adjusted based on the perplexity of the input command, specifically as follows:

[0030] When the confusion level of the input instruction is lower than a first preset value, the information entropy threshold is reduced.

[0031] When the confusion level of the input instruction is higher than the second preset value, the information entropy threshold is increased.

[0032] This invention also discloses an unsupervised preference alignment system for language models based on information entropy, used to perform the aforementioned unsupervised preference alignment method for language models based on information entropy, comprising:

[0033] The sampling and probability capture module is used to sample the same instruction multiple times to obtain the candidate response set generated by the language model and the probability distribution of each token in each candidate response.

[0034] The entropy calculation and illusion quantification module is used to calculate the information entropy of each token based on the probability distribution, and to fuse the semantic deviation between multiple candidate responses in the candidate response set to generate a risk assessment index for each response fragment in each candidate response.

[0035] The self-preference sample construction module is used to unsupervisedly divide positive preference samples and negative preference samples from the candidate response set based on the risk assessment index.

[0036] The entropy-weighted preference alignment module is used to fine-tune the language model by using the positive preference samples and negative preference samples as training data and the risk index corresponding to the negative preference samples as a weight factor.

[0037] As an optimization, the entropy calculation and illusion quantification module includes:

[0038] The sliding window entropy calculation unit is used to calculate the information entropy based on the probability distribution of each token in each candidate response, and to smooth the information entropy through a sliding window to obtain the average sliding window entropy of each token position.

[0039] A semantic divergence calculation unit is used to calculate the degree of semantic divergence between the response fragment at each semantic position in each candidate response and the consensus center at the corresponding semantic position, wherein the consensus center is jointly determined by all candidate responses in the candidate response set;

[0040] The risk assessment unit is used to multiply the mean of the sliding window entropy by the degree of semantic deviation to generate the risk assessment index.

[0041] As an optimization, it also includes:

[0042] The inference-side real-time calibration module is used to dynamically adjust the information entropy threshold according to the perplexity of the input command, calculate the information entropy of the current token of the candidate response in real time when the language model generates a response token by token, and trigger at least one intervention operation among truncation generation, resampling, or returning a rejected answer when the information entropy exceeds the dynamically adjusted threshold.

[0043] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned unsupervised preference alignment method for a language model based on information entropy.

[0044] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0045] This invention constructs a system based on the mean of sliding window entropy and the degree of semantic deviation. The indicator automatically identifies high-entropy, high-risk segments and divides positive and negative preference samples, realizing the self-construction of completely unsupervised preference data without any manual or external annotation, thereby significantly reducing alignment costs and enabling the scalable application of model alignment technology.

[0046] The present invention is constructed in step S2 The indicator accurately identifies illusory fragments generated by the model in knowledge blind spots, and uses an entropy-weighted loss function. For high Negative samples are penalized with stronger weights, so that the alignment signal originates from the model’s own knowledge boundary rather than external forced preferences, thereby suppressing “forced illusion” at its source and maintaining the model’s factual fidelity.

[0047] This invention dynamically adjusts the information entropy threshold based on the confusion level (PPL) of the input instruction. Specifically, it lowers the threshold and adopts strict judgment when the instruction is clear, and raises the threshold and adopts lenient judgment when the instruction is ambiguous. This enables the model to adaptively adjust the illusion judgment standard according to the clarity of the current instruction, thereby achieving dynamic adaptation to changes in the model's knowledge boundary.

[0048] This invention calculates real-time entropy token-by-token during the inference phase. and in When the dynamic threshold is exceeded, the system immediately triggers operations such as truncation, resampling, or returning a rejection response. This solidifies the entropy perception capability learned during the training phase into the inference phase, thereby enabling real-time early warning and intervention for high-risk generated content and enhancing the security and controllability of the model deployment phase.

[0049] This invention provides a token-level [service / mechanism] in step S2. and fragment level Quantitative indicators explicitly quantify the "uncertainty" of the model, thereby providing a unified and quantifiable benchmark for hallucination determination and solving the problem of the lack of quantifiable standards for hallucination determination in existing technologies.

[0050] This invention designs its core framework into five standardized modules: sampling, entropy calculation, self-constructed preference, entropy-weighted alignment, and inference calibration, and the loss function... It is compatible with mainstream DPO / IPO / SimPO frameworks, making this invention independent of specific model architectures and applicable to various base-based large language models. As a result, it can be seamlessly embedded into existing training processes, exhibiting good compatibility and portability. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0052] Figure 1 This is a flowchart of an unsupervised preference alignment method for a language model based on information entropy, according to the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0054] Example 1 discloses an unsupervised preference alignment method for language models based on information entropy, including S1 to S4. The following describes the content of each step in detail.

[0055] S1. Sample the same instruction multiple times to obtain the candidate response set generated by the language model and the probability distribution of each token in each candidate response.

[0056] For a given instruction Q, set the number of samplings N and the sampling hyperparameters, and use the language model M to be optimized to perform N independent samplings to generate a candidate response set. During the generation process, the probability distribution of each token at its position in the vocabulary is recorded. ,in This represents the t-th token to be generated. This represents the sequence of all tokens generated before the current token, and Q represents the input instruction. The conditional probability represents the probability distribution of the language model predicting the next token given the generated prefix sum and input instruction. The original scores corresponding to the probability distribution are stored in the form of a Logits matrix for subsequent calculation of information entropy.

[0057] S2. Calculate the information entropy of each Token based on the probability distribution, and integrate the semantic deviation between multiple candidate responses in the candidate response set to generate a risk assessment index for each response fragment in each candidate response.

[0058] First, for each token in the response sequence, calculate its predicted information entropy. :

[0059] ;

[0060] wherein, is the prediction information entropy of the Token at the t-th position, which quantifies the uncertainty of the prediction probability distribution of the language model when generating the Token at the t-th position; V is a vocabulary (a set of all candidate Tokens supported by the language model), v is a specific Token in the vocabulary, that is, a vocabulary index, and P(v) is a conditional probability distribution, that is, the confidence probability that the model predicts that the next Token is v.

[0061] secondly, calculating the Sliding Window Entropy (SWE), setting a sliding window size to filter instantaneous entropy fluctuation caused by function words such as local logical connectives "of", "is", "in" and the like, focus on capturing continuous high-entropy regions and focus on the uncertainty of factual Tokens. For each Token position t, calculating the arithmetic mean of the information entropy of k consecutive Tokens ending at the t-th Token (that is, the average of the entropy of k Tokens from t-k+1 to t), and recording the mean as sliding window entropy :

[0062] ;

[0063] wherein, is the mean sliding window entropy: by averaging the entropy values of a local region, the instantaneous noise generated by grammatical and logical words is smoothed to identify persistent factual uncertainty; is a window size parameter, which is a preset hyperparameter that determines the text span for investigating local uncertainty; is a traversal index, which is the position of the currently calculated Token within the window, is the information entropy of the j-th Token. The mean sliding window entropy is used for filtering the instantaneous entropy fluctuation caused by local logical interference words and capturing continuous high-entropy regions.

[0064] finally, aligning N candidate responses according to semantics, and determining a consensus center at each semantic position. For each response fragment, calculating the degree of semantic deviation between the semantic embedding vector of the fragment and the consensus center at the semantic position where the fragment is located. If the mean sliding window entropy of a response fragment is relatively high and the degree of deviation between the semantic embedding vector of the fragment and the consensus center is relatively large, the fragment is marked as a high-risk region of factual hallucination.

[0065] ;

[0066] wherein, The risk assessment metric for the response segment (hallucination risk index) is as follows: the higher the value, the higher the probability that the response segment is a hallucination caused by a knowledge blind spot of the language model; SWE is the mean sliding window entropy of the response segment; Similarity is the semantic similarity function. Let be the semantic embedding vector of the response fragment at the j-th semantic position in the i-th candidate response; Let be the consensus center (centroid vector) of the response fragments of all candidate responses at the j-th semantic position.

[0067] S3. Based on the risk assessment indicators, unsupervisedly separate positive preference samples and negative preference samples from the candidate response set. The core of this step is achieving zero-labeling: utilizing the model's own "consistency" as a positive signal to transform "uncertainty" into a negative signal.

[0068] In some embodiments, the specific process of S3 is as follows:

[0069] S3.1. Select the candidate response with the lowest overall entropy value and located at the semantic cluster center from the candidate response set as the positive preference sample. This represents the model's most confident and mainstream answer based on existing knowledge.

[0070] First, for each candidate response Sum or average the information entropy of all its tokens to obtain the overall entropy value: ;in, Let be the length of the token in the i-th candidate response. Let be the information entropy of the t-th token. The lower the overall entropy value, the more "deterministic" the model is in generating this response.

[0071] Then, the semantic embedding vectors of all candidate responses are clustered (e.g., K-Means clustering), the cluster with the most samples is found, and the centroid of the cluster is calculated as the semantic cluster center.

[0072] Finally, candidate responses that simultaneously meet the following two conditions are selected from the candidate response set: the lowest overall entropy value (or below a preset threshold); and the semantic embedding vector being closest to the semantic cluster center (or less than a preset threshold). This response is the positive preference sample. This represents the model's most confident and mainstream answer based on existing knowledge.

[0073] Taking the instruction "What is the capital of China?" as an example, the following candidate responses were obtained after 5 samplings:

[0074] "The capital of China is Beijing." (Low entropy, mainstream semantics)

[0075] "Beijing is the capital of China." (Low entropy, mainstream semantics)

[0076] "The capital of China is Shanghai." (High entropy, semantic deviation)

[0077] R4: "I don't know." (Low entropy, but semantics deviates from the mainstream).

[0078] R5: "Beijing, the capital." (Low entropy, mainstream semantics);

[0079] The semantic clustering center is the semantic related to "Beijing is the capital". , The one with the lowest overall entropy in R5 (assuming) ) was selected .

[0080] S3.2 Identify response segments in the candidate response set whose risk assessment indicators exceed a preset threshold, and construct the negative preference sample containing hallucination tendencies by performing token-level perturbation on the response segments or selecting a high-entropy sampling path.

[0081] Token-level perturbations are suitable for scenarios where there are already normal responses, and the goal is to "create" an illusion based on them. The specific process is as follows:

[0082] First, iterate through all candidate responses to identify risk assessment metrics. Response fragments exceeding a preset threshold (e.g., 0.7) are defined as high-risk fragments. These fragments represent regions where the model is uncertain at that location and semantically deviates from the consensus. Then, within these high-risk fragments, the information entropy is further located. The top few tokens (usually factual words such as nouns and verbs) are defined as high-entropy tokens. For each high-entropy token, a non-Top-1 token is sampled from the probability distribution of the current tokens and used to replace it (i.e., instead of selecting the word the model considers "most likely," a "second most likely" or "third most likely" word is chosen). Sampling strategies can include: selecting the second most likely token (suboptimal selection); random sampling according to the probability distribution (the temperature coefficient can be adjusted to increase randomness); selecting the token with the greatest semantic difference from the original token in the probability distribution, thus obtaining the perturbed token. The perturbed tokens are then recombined into a complete response as negative preference samples. .

[0083] Original response excerpt: "China's capital is Beijing."

[0084] Token probability distribution (in the "Beijing" location):

[0085] "Beijing": 60%;

[0086] Shanghai: 25%;

[0087] Nanjing: 10%;

[0088] "Guangzhou": 5%;

[0089] The perturbation method is as follows: Replace "Beijing" with "Shanghai," which has the second highest probability, to obtain:

[0090] Negative sample fragment: "The capital of China is Shanghai."

[0091] Selecting a high-entropy sampling path is suitable for directly utilizing the high-entropy path already generated by the model during sampling, without the need for additional perturbation. The specific process is as follows:

[0092] During the multiple sampling processes in S1, the model records the sampling path for each token in each generation. Some paths may contain many high-entropy tokens (i.e., the model hesitates among multiple candidate words). From the candidate response set, responses that meet one of the following conditions are directly selected as negative preference samples: the candidate response with the highest overall entropy value; containing... Candidate responses for segments exceeding a preset threshold; candidate responses whose semantic embedding vectors are furthest from the cluster centers. If higher quality negative samples are desired, the selected high-entropy responses can be further processed, such as retaining high-entropy segments and replacing low-entropy segments with correct content, forming a hybrid sample of "partially correct + partially illusory". For example, for the instruction "explain quantum entanglement", the model samples as follows:

[0093] Correct answer (low entropy), selected as ; The content containing the off-topic statement "Shanghai is an economic center" (a high-entropy segment) was directly selected as... R3: Fabrication (extremely high entropy), optional .

[0094] S4. Using the positive preference samples and negative preference samples as training data, and using the risk index corresponding to the negative preference samples as a weighting factor, the language model is fine-tuned for preference optimization.

[0095] This step improves the loss function, making the model more alert to "illusions" during fine-tuning. Unlike traditional DPO, this invention introduces an entropy weighting factor, which penalizes high-entropy segments more severely than low-entropy segments, forcing the model to tend to choose the probabilistic path of "refusing to answer" or "conservative answer" when faced with uncertain knowledge.

[0096] In some embodiments, an entropy-weighted direct preference optimization loss function is used to fine-tune the language model:

[0097] ;

[0098] in, The objective of the improved loss function, which uses entropy-weighted direct preference optimization loss, is to update model parameters via gradient descent. Let be the expectation operator, and let represent the average loss over the entire self-constructed preference dataset. The Sigmoid activation function maps the log-odds difference to the (0,1) interval, and is used to construct probability preferences. The divergence constraint hyperparameter is used to control the current language model. Relative to the reference language model The degree of deviation is considered to prevent the model from collapsing during alignment. is the entropy weighting factor for negatively biased samples, used to dynamically adjust the penalty intensity based on the risk assessment index of the negatively biased samples; x is the input instruction. For positive preference samples, This is a negative preference sample. Generate the current language model under input x The probability, Generate a reference language model (a frozen copy before fine-tuning) under input x. The probability, Generate the current language model under input x The probability, Generate a reference language model based on input x. The probability, For the current language model; This serves as a reference language model.

[0099] It also includes S5, real-time calibration and interception mechanisms at the inference end, specifically:

[0100] S5.1 Dynamically adjust the information entropy threshold based on the confusion level of the input command, specifically as follows:

[0101] Calculate the perplexity (PPL) of the input instruction.

[0102] When the confusion level of the input instruction is lower than the first preset value, the instruction is determined to be clear, the information entropy threshold is reduced (tightened), and a stricter hallucination determination standard is adopted.

[0103] When the confusion level of the input instruction is higher than the second preset value, the instruction is determined to be ambiguous, the information entropy threshold is increased (relaxed), and a more lenient hallucination determination standard is adopted.

[0104] S5.2 When the language model generates candidate responses token by token, the information entropy of the current token of the candidate response is calculated in real time. ;

[0105] S5.3, when the information entropy of the current Token When the dynamically adjusted information entropy threshold is exceeded, at least one of the following intervention operations is triggered:

[0106] Truncate the response: Stop generating the response to prevent errors from spreading.

[0107] Resampling: Replace with a more conservative sampling strategy and regenerate the current token or subsequent content;

[0108] Refuse to answer: Replace the current response with a preset disclaimer or a "Cannot answer accurately" message.

[0109] In some embodiments, perplexity is a commonly used metric to measure the uncertainty of a language model in predicting text: the lower the PPL, the more familiar the model is (the more the text conforms to the model's knowledge); the higher the PPL, the more perplexed the model is (the more strange or ambiguous the text).

[0110] For an instruction sequence of length L The degree of confusion is defined as:

[0111] ;

[0112] Or an equivalent form:

[0113] ;

[0114] in, The model predicts the current token given the previous tokens. The probability of; It is the log-likelihood of the current token; the average of the log-likelihoods of all tokens is then taken as the negative exponent. In this embodiment, the PPL value of the input instruction Q is calculated using the language model M to be optimized.

[0115] Example 2 discloses an unsupervised preference alignment system for language models based on information entropy, used to execute the unsupervised preference alignment method for language models based on information entropy described in Example 1, including:

[0116] The sampling and probability capture module is used to sample the same instruction multiple times to obtain the candidate response set generated by the language model and the probability distribution of each token in each candidate response.

[0117] The entropy calculation and illusion quantification module is used to calculate the information entropy of each token based on the probability distribution, and to integrate the semantic consistency among multiple candidate responses in the candidate response set to generate a risk assessment index for each response fragment in each candidate response.

[0118] The self-preference sample construction module is used to unsupervisedly divide positive preference samples and negative preference samples from the candidate response set based on the risk assessment index.

[0119] The entropy-weighted preference alignment module is used to fine-tune the language model by using the positive preference samples and negative preference samples as training data and the risk index corresponding to the negative preference samples as a weight factor.

[0120] In some embodiments, the entropy calculation and hallucination quantification module includes:

[0121] The sliding window entropy calculation unit is used to calculate the information entropy based on the probability distribution of each token in each candidate response, and to smooth the information entropy through a sliding window to obtain the average sliding window entropy of each token position.

[0122] A semantic divergence calculation unit is used to calculate the degree of semantic divergence between the response fragment at each semantic position in each candidate response and the consensus center at the corresponding semantic position, wherein the consensus center is jointly determined by all candidate responses in the candidate response set;

[0123] The risk assessment unit is used to multiply the mean of the sliding window entropy by the degree of semantic deviation to generate the risk assessment index.

[0124] In some embodiments, it also includes:

[0125] The inference-side real-time calibration module is used to dynamically adjust the information entropy threshold according to the perplexity of the input command, calculate the information entropy of the current token of the candidate response in real time when the language model generates a response token by token, and trigger at least one intervention operation among truncation generation, resampling, or returning a rejected answer when the information entropy exceeds the dynamically adjusted threshold.

[0126] Example 3 discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the unsupervised preference alignment method for a language model based on information entropy described in Example 1. The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An unsupervised preference alignment method for language models based on information entropy, characterized in that, include: The same instruction is sampled multiple times to obtain a set of candidate responses generated by the language model and the probability distribution of each token in each candidate response. The information entropy of each token is calculated based on the probability distribution, and the semantic deviation between multiple candidate responses in the candidate response set is integrated to generate a risk assessment index for each response fragment in each candidate response. Based on the aforementioned risk assessment indicators, positive preference samples and negative preference samples are unsupervisedly divided from the candidate response set; The language model is fine-tuned by using the positive and negative preference samples as training data and the risk index corresponding to the negative preference samples as a weighting factor.

2. The unsupervised preference alignment method for language models based on information entropy according to claim 1, characterized in that, The specific process for generating risk assessment indicators for each response fragment in each of the candidate responses is as follows: The information entropy is calculated based on the probability distribution of each token in each candidate response, and the information entropy is smoothed by a sliding window to obtain the average sliding window entropy of each token position. Calculate the semantic deviation between the response fragment at each semantic position in each candidate response and the consensus center at the corresponding semantic position, wherein the consensus center is jointly determined by all candidate responses in the candidate response set; The risk assessment index is generated by multiplying the mean of the sliding window entropy by the degree of semantic deviation.

3. The unsupervised preference alignment method for language models based on information entropy according to claim 1, characterized in that, The specific process of unsupervisedly dividing positive preference samples and negative preference samples from the candidate response set based on the aforementioned risk assessment indicators is as follows: The candidate response with the lowest overall entropy value and located at the semantic cluster center in the candidate response set is taken as the positive preference sample; Identify response segments in the candidate response set whose risk assessment indicators exceed a preset threshold, and construct the negative preference sample containing hallucination tendencies by performing token-level perturbation on the response segments or selecting a high-entropy sampling path.

4. The unsupervised preference alignment method for language models based on information entropy according to claim 1, characterized in that, The loss function used for preference optimization fine-tuning of the language model is the entropy-weighted direct preference optimization loss function, which has the following form: ； in, Entropy-weighted direct preference optimization loss, For expectation operator, It is the Sigmoid activation function. Divergence constraint hyperparameter, is the entropy weighting factor for negatively biased samples, used to dynamically adjust the penalty intensity based on the risk assessment index of the negatively biased samples; x is the input instruction. For positive preference samples, This is a negative preference sample. Generate the current language model under input x The probability, Generate a reference language model (a frozen copy before fine-tuning) under input x. The probability, Generate the current language model under input x The probability, Generate a reference language model based on input x. The probability of.

5. The unsupervised preference alignment method for language models based on information entropy according to claim 1, characterized in that, Also includes: The information entropy threshold is dynamically adjusted based on the confusion level of the input command. When the language model generates candidate responses token by token, the information entropy of the current token of the candidate response is calculated in real time. When the information entropy exceeds the dynamically adjusted information entropy threshold, at least one intervention operation is triggered, including truncation generation, resampling, or returning a rejected answer.

6. The unsupervised preference alignment method for language models based on information entropy according to claim 5, characterized in that, The information entropy threshold is dynamically adjusted based on the confusion level of the input command, specifically as follows: When the confusion level of the input instruction is lower than a first preset value, the information entropy threshold is reduced. When the confusion level of the input instruction is higher than the second preset value, the information entropy threshold is increased.

7. An unsupervised preference alignment system for language models based on information entropy, used to execute the unsupervised preference alignment method for language models based on information entropy as described in any one of claims 1-6, characterized in that, include: The sampling and probability capture module is used to sample the same instruction multiple times to obtain the candidate response set generated by the language model and the probability distribution of each token in each candidate response. The entropy calculation and illusion quantification module is used to calculate the information entropy of each token based on the probability distribution, and to fuse the semantic deviation between multiple candidate responses in the candidate response set to generate a risk assessment index for each response fragment in each candidate response. The self-preference sample construction module is used to unsupervisedly divide positive preference samples and negative preference samples from the candidate response set based on the risk assessment index. The entropy-weighted preference alignment module is used to fine-tune the language model by using the positive preference samples and negative preference samples as training data and the risk index corresponding to the negative preference samples as a weight factor.

8. The unsupervised preference alignment system for a language model based on information entropy according to claim 7, characterized in that, The entropy calculation and hallucination quantification module includes: The sliding window entropy calculation unit is used to calculate the information entropy based on the probability distribution of each token in each candidate response, and to smooth the information entropy through a sliding window to obtain the average sliding window entropy of each token position. A semantic divergence calculation unit is used to calculate the degree of semantic divergence between the response fragment at each semantic position in each candidate response and the consensus center at the corresponding semantic position, wherein the consensus center is jointly determined by all candidate responses in the candidate response set; The risk assessment unit is used to multiply the mean of the sliding window entropy by the degree of semantic deviation to generate the risk assessment index.

9. The unsupervised preference alignment system for a language model based on information entropy according to claim 7, characterized in that, Also includes: The inference-side real-time calibration module is used to dynamically adjust the information entropy threshold according to the perplexity of the input command, calculate the information entropy of the current token of the candidate response in real time when the language model generates a response token by token, and trigger at least one intervention operation among truncation generation, resampling, or returning a rejected answer when the information entropy exceeds the dynamically adjusted threshold.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements an unsupervised preference alignment method for a language model based on information entropy, as described in any one of claims 1 to 6.

Images