Fine-grained debiasing method and system for large language models based on bias strength

Abstract

The present invention discloses a fine-grained debiasing method and system for a large language model based on bias intensity, which belongs to the field of generative artificial intelligence. The present invention introduces a fine-grained data annotation method for multi-model collaboration, quantifies the bias intensity of samples through a voting mechanism of multiple mainstream large language models, and solves the problem that bias intensity is unmeasurable in traditional methods; in addition, the present invention designs an improved direct preference optimization loss function, which distinguishes the optimization weights of different biases by bias intensity weighting, and introduces a probability decoupling regularization term to block the same-direction growth of the generation probability of neutral answers and biased answers, thereby preventing the abnormal increase of bias probability during the debiasing process. The present invention realizes fine-grained and differentiated debiasing optimization, which can effectively remove bias in large language models, and can also effectively improve the stability of training, and is suitable for scenarios such as dialogue systems and text generation.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence, and in particular to a fine-grained debiasing method for a large language model based on bias strength. Background Art

[0002] Prejudice is a non-objective, negative evaluation based on social culture, individual cognition, and experience. It often takes the form of stereotypes based on social attributes such as age, gender, appearance, and nationality. This type of cognitive bias not only leads to social injustice and discrimination but can also be absorbed and solidified by large language models (LLMs) through data training. With the rapid expansion of LLM application scenarios, internalized biases within the models may be systematically amplified, causing irreversible damage to public decision-making, social equity, and group diversity. Therefore, eliminating bias in LLMs is crucial.

[0003] Current LLM debiasing research is mainly advancing along two technical paths:

[0004] One approach involves real-time intervention strategies in the generation process, such as correcting the output through techniques like zero-shot self-explanation, feedback-based prompting, or thought chain guidance. While these approaches can suppress bias immediately, they suffer from two limitations: first, their interactive mechanisms rely on users proactively providing anti-bias examples, limiting their practical application scenarios; second, these approaches only superficially intervene in the LLM output and fail to fundamentally remove the model's inherent bias.

[0005] The second approach is LLM post-training based on human preference alignment, exemplified by Direct Preference Optimization (DPO). This method constructs biased-neutral answer pairs (y+, y-) and, during the training phase, forces the model to increase the probability difference between the neutral answer y+ and the biased answer y-, driving the model toward unbiased output. However, this method's optimization mechanism also has flaws. First, due to the lack of a quantitative assessment of bias intensity, it cannot accurately optimize for varying degrees of bias severity. Second, DPO's optimization objective focuses solely on maximizing the probability difference, which can lead to a simultaneous increase in the probabilities of y+ and y-. Even as the difference between the two probabilities increases, the probability of the biased answer may still increase in tandem with the probability of the neutral answer. This characteristic makes it difficult for the model to consistently achieve the ideal optimization path of "strictly increasing the probability of y+ and strictly decreasing the probability of y-," severely undermining the effectiveness of the debiasing system. Summary of the Invention

[0006] The purpose of the present invention is to solve the two problems of "unquantifiable bias strength" and "abnormally high probability of biased answer output" in the debiasing training of the direct preference optimization (DPO) method in the prior art, and to propose a fine-grained debiasing method and system for large language models based on bias strength.

[0007] The specific technical solutions adopted in the present invention are as follows:

[0008] In a first aspect, the present invention provides a large language model fine-grained debiasing method based on bias strength, comprising:

[0009] S1. Obtain a training dataset consisting of debiased samples, where the debiased samples are triplets consisting of an input question, a neutral answer, and a biased answer. Input each debiased sample into a bias assessment model group consisting of multiple large language models. Perform multi-model voting on the more neutral of the two answers in the triplet through group decision making, and calculate the bias strength of the debiased sample based on the multi-model voting results.

[0010] S2. Iteratively fine-tune the large language model in batches using the training dataset to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model outputs debiased neutral answers to user questions; in each round of fine-tuning, the original large language model without fine-tuning is used as the reference model , the large language model obtained in the previous round of fine-tuning is used as the model to be optimized , by the reference model and the model to be optimized Based on the input questions in the debiased samples, the probabilities of neutral answers and biased answers in the debiased samples are obtained respectively. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss and the model to be optimized is reversely optimized. .

[0011] As a preferred embodiment of the first aspect, the number of large language models included in the bias assessment model group is an even number not less than 2, and each of the included large language models has been pre-safely aligned.

[0012] As a preferred embodiment of the first aspect, when multi-model voting is performed through group decision-making, the triple corresponding to each debiased sample is input into each large language model in the bias evaluation model group, and the large language model is driven by the prompt word to judge which of the neutral answer and the biased answer contained in the input triple is more neutral and vote, and the first vote that all large language models in the bias evaluation model group believe that the neutral answer is more neutral is counted. and the second vote for those who think the biased answer is more neutral , the first vote and second votes The ratio of the difference to the sum is taken as the bias strength of the debiased sample.

[0013] As a preferred embodiment of the first aspect above, in each round of fine-tuning, each debiased sample in the current training batch needs to input the input problem contained in the reference model and the model to be optimized In the answer text, the probability distribution of the word output by the model for each position in the word list is obtained, so as to calculate the first probability of the neutral answer output by the optimized model based on the model output and the actual neutral answer and bias answer in the debiased sample. , the second probability of the model to be optimized outputting biased answers , the third probability of the reference model outputting a neutral answer , the fourth probability of the reference model outputting biased answers .

[0014] As a preferred embodiment of the first aspect, the fine-grained debiasing loss term is a negative log-likelihood loss scaled by the absolute value of the bias strength, wherein the likelihood part is the activation output after multiplying the weight factor, the sign function of the bias strength, and the probability difference, and the probability difference is the first probability. With the third probability The logarithm of the ratio minus the second probability With the fourth probability The logarithm of the ratio is calculated.

[0015] As a preferred embodiment of the first aspect, the probability decoupling regularization loss term is a negative log-likelihood loss, wherein the likelihood part is the activation output after the negative of the weight factor is multiplied by the regularization term, and the regularization term is the weighted sum of two logarithmic terms, and the first weighted term is the first probability With the third probability The logarithm of the ratio, weighted by the second number of votes First place in votes and second votes The second weighted term is the second probability With the fourth probability The logarithm of the ratio, weighted by the first vote First place in votes and second votes The ratio of the sum.

[0016] In a second aspect, the present invention provides a large language model fine-grained debiasing system based on bias strength, comprising:

[0017] A group decision module is configured to obtain a training dataset consisting of debiased samples, where the debiased sample is a triple consisting of an input question, a neutral answer, and a biased answer. Each debiased sample is input into a bias assessment model group consisting of multiple large language models. A multi-model vote is performed on the more neutral of the two answers in the triple through group decision, and the bias strength of the debiased sample is calculated based on the multi-model vote result.

[0018] The fine-tuning training module uses the training data set to iteratively fine-tune the large language model in batches to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model can output debiased neutral answers to user questions; in each round of fine-tuning, the original large language model without fine-tuning is used as the reference model , the large language model obtained in the previous round of fine-tuning is used as the model to be optimized , by the reference model and the model to be optimized Based on the input questions in the debiased samples, the probabilities of neutral answers and biased answers in the debiased samples are obtained respectively. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss and the model to be optimized is reversely optimized. .

[0019] In a third aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the large language model fine-grained debiasing method based on bias strength as described in any of the solutions in the first aspect above.

[0020] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the large language model fine-grained debiasing method based on bias strength as described in any one of the solutions in the first aspect above can be implemented.

[0021] In a fifth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0022] The memory is used to store computer programs;

[0023] The processor is configured to implement the large language model fine-grained debiasing method based on bias strength as described in any one of the solutions of the first aspect above when executing the computer program.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This paper adopts a multi-model collaborative fine-grained data annotation method, quantifying the bias strength of samples through a voting mechanism of multiple mainstream large language models, solving the problem of unmeasurable bias strength in traditional methods;

[0026] (2) The present invention designs an improved DPO (Direct Preference Optimization) loss function, which distinguishes the optimization weights of different biases by weighting the bias strength, and introduces a probability decoupling regularization term to block the same-direction growth of the generation probabilities of neutral answers and biased answers, thereby preventing the abnormal increase of bias probability during the debiasing process.

[0027] (3) This invention achieves fine-grained and differentiated debiasing optimization, which can effectively remove bias in large language models and effectively improve the stability of training. It is suitable for scenarios such as dialogue systems and text generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the steps of the fine-grained debiasing method for large language models based on bias strength.

[0029] Figure 2 This is the overall data flow diagram of the present invention.

[0030] Figure 3 Schematic diagram of the module composition of the fine-grained debiasing system for large language models based on bias strength.

[0031] Figure 4 It is a structural diagram of computer electronic equipment. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0033] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0034] This paper provides a large language model fine-grained debiasing method based on bias strength. It includes two core modules that operate in tandem: (1) a fine-grained data annotation method based on multi-model voting, which calculates bias strength through group decision-making of multiple mainstream large models, breaking through the limitations of binary biased data; and (2) an improved loss function that integrates bias strength with probabilistic decoupling regularization. The former achieves differentiated optimization for different bias strengths, while the latter prevents the probabilities of neutral and biased answers from increasing in the same direction, thereby improving the stability of debiasing training. The specific implementation of the above large language model fine-grained debiasing method is described in detail below.

[0035] In a preferred embodiment of the present invention, the above-mentioned large language model fine-grained debiasing method based on bias strength includes steps S1 and S2, and the overall data flow is as follows: Figure 2 As shown, the specific implementation methods of steps S1 and S2 are described below.

[0036] S1. Obtain a training dataset consisting of debiased samples, where the debiased sample is a triple consisting of an input question, a neutral answer, and a biased answer. Input each debiased sample into a bias assessment model group consisting of multiple large language models. Through group decision-making, multi-model voting is performed on the relatively more neutral answer of the two answers in the triple. The bias strength of the debiased sample is calculated based on the multi-model voting results.

[0037] It should be noted that the training dataset in this invention consists of a series of debiased examples. This training dataset can use existing datasets, such as the biasDPO, hh-rlhf, and PKU-SafeRLHF-QA datasets, or it can be constructed independently. The debiased examples are input questions, neutral answers, and biased answers, all of which can be in text form.

[0038] It should also be noted that the above-mentioned bias assessment model group needs to include multiple existing large language models, and the number of large language models included must be an even number not less than 2. Each large language model included must have undergone safe alignment in advance to ensure that it can accurately identify the neutrality and bias of the answers.

[0039] The role of the bias assessment model group is to further vote on the relative bias of the answers in the debiased samples in the training data set. In an embodiment of the present invention, when multi-model voting is performed through group decision-making, the triple corresponding to each debiased sample is input into each large language model in the bias assessment model group, and the large language model is driven by the prompt word (Prompt) to determine which of the neutral answer and the biased answer contained in the input triple is more neutral and vote, and the first vote that all large language models in the bias assessment model group believe is more neutral is counted. and the second vote for those who think the biased answer is more neutral , the first vote and second votes The difference and sum The ratio between them is taken as the bias strength p of the debiased sample.

[0040] Continue to see Figure 2 As shown, in an embodiment of the present invention, data flows to The specific implementation process of fine-grained data annotation based on multi-model voting in step S1 above is as follows:

[0041] 1. Construction of the Bias Evaluation Model Group: K mainstream large language models (such as GPT4, Claude 3, and DeepSeek) are selected to form the Bias Evaluation Committee (BEC), with k ≥ 2 and an even number. The larger the k value, the finer the granularity of the bias evaluation model group's assessment of bias intensity, and the more effective it is in distinguishing biases of varying severity. The reasons for selecting mainstream large language models are: (1) These models undergo rigorous security alignment training, resulting in highly secure output content and a more reliable alternative to human neutrality judgment; (2) The multi-model voting mechanism can offset the potential bias of individual models and improve the robustness of group decision-making.

[0042] 2. Voting decision: Traditional DPO training data includes input question x, neutral answer y+, and biased answer y-. The triple (x, y+, y-) is given to each BEC member (i.e., the large language model in BCE) and asked to determine which answer is more neutral / unbiased and vote for the corresponding answer. The number of first-place votes is counted. (number of votes that think y+ is more neutral) and second votes (Votes that consider y- to be more neutral).

[0043] 3. Bias strength calculation: by formula: Calculate the bias strength p, which ranges from -1 to 1. When p approaches 1, BCE members unanimously agree that y+ is a neutral response and y- is strongly biased. When p approaches -1, the opposite is true. The absolute value of p reflects the bias strength. A larger absolute value indicates a more significant bias in the sample, and will be assigned a higher weight in subsequent optimizations.

[0044] Therefore, based on the above step S1, each debiased sample in the training data set can get the first vote , second number of votes And the bias strength p, each debiased sample used for subsequent training steps can be expressed as .

[0045] S2. Iteratively fine-tune the large language model in batches using the training dataset to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model outputs debiased neutral answers to user questions; in each round of fine-tuning, the original large language model without fine-tuning is used as the reference model , the large language model obtained in the previous round of fine-tuning is used as the model to be optimized , by the reference model and the model to be optimized Based on the input questions in the debiased samples, the probabilities of neutral answers and biased answers in the debiased samples are obtained respectively. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss and the model to be optimized is reversely optimized. .

[0046] It should be noted that in each round of fine-tuning, a batch needs to be sampled from all training datasets for training. For the sake of convenience, the batch sampled in each round of fine-tuning is regarded as the current batch of this round of fine-tuning. Each debiased sample in the current training batch needs to input the input problem contained in it into the reference model. and the model to be optimized In the answer text, the probability distribution of the word output by the model for each position in the word list is obtained, so as to calculate the first probability of the neutral answer in the debiased sample output by the optimized model based on the model output and the actual neutral answer and bias answer in the debiased sample. , the second probability of biased answers in the debiased samples output by the optimized model , the third probability of the neutral answer in the reference model output debiased sample , the fourth probability of biased answers in the reference model output debiased samples .

[0047] The first probability , the second probability , the third probability , the fourth probability It can be used together with the multi-model voting results and bias strength p to calculate the fine-grained debiasing loss term and the probabilistic decoupling regularization loss term.

[0048] In an embodiment of the present invention, the above-mentioned fine-grained debiasing loss term is a negative log-likelihood loss scaled by the absolute value of the bias strength, wherein the likelihood part is a weight factor, a sign function of the bias strength, a probability difference The activation output after multiplication, the probability difference By the first probability With the third probability The logarithm of the ratio minus the second probability With the fourth probability The logarithm of the ratio is calculated.

[0049] In an embodiment of the present invention, the probability decoupling regularization loss term is a negative log-likelihood loss, where the likelihood part is the activation output after the negative of the weight factor is multiplied by the regularization term, and the regularization term is the weighted sum of two logarithmic terms, the first weighted term is the first probability With the third probability The logarithm of the ratio, weighted by the second number of votes First place in votes and second votes The second weighted term is the second probability With the fourth probability The logarithm of the ratio of First place in votes and second votes The ratio of the sum.

[0050] It should be noted that the aforementioned fine-grained debiasing loss and probabilistic decoupling regularization loss describe the loss of a single sample. Since this invention employs batch training, their actual calculations require the expected value (i.e., the average value) of all samples in the batch. This is a conventional technique and will not be further elaborated.

[0051] Continue to see Figure 2 As shown, in an embodiment of the present invention, data flows to arrive This corresponds to the loss function calculation and model parameter fine-tuning update process in step S2 above.

[0052] In the above data flow The four output probabilities can be calculated as follows: , the second probability , the third probability , the fourth probability To maintain generality, the following describes the model using input x and answer y (y+ or y-). ( or ) The process of calculating the probability of output y is as follows:

[0053] Represent the answer text y output by the model as a token sequence , among which Indicates that y is modeled The ith token after the token segmentation of the token segmenter. Then we have:

[0054]

[0055] in: is the first t-1 generated tokens, T is the length of the output sequence, is the tth token of the generated sequence, Indicates that the tth token is generated when the first t-1 tokens have been generated. The probability of , which is obtained by softmax processing of the output generated by the model at step t:

[0056]

[0057] Among them It's a model The vocabulary size (all possible tokens), Is the model for the current token The unnormalized score of .

[0058] Therefore, for the input x, the model You can output the token in the vocabulary for each position in the answer text y Therefore, for any answer that needs to calculate the probability (Right now or ), you can first determine the answer The actual word at each position, and then determine the probability value of this word in the probability distribution, the product of the probability values ​​of all positions can be regarded as the model Output answer It should be noted that in actual calculations, multiplication operations are relatively complex, so the present invention can convert the multiplication of all position probability values ​​into addition through logarithmic transformation. That is to say, (x, y+), (x, y-) are respectively sent to the model to be optimized and reference models , the original first probability , the second probability , the third probability , the fourth probability The logarithmic transformation can be further performed to obtain the log values ​​of the following four output probabilities: 、 、 、 , representing the probability of the optimized model outputting a neutral answer and a biased answer, and the probability of the reference model outputting a neutral answer and a biased answer, respectively, which are used for the subsequent probability decoupling regularization term R and probability difference .

[0059] Data Flow Represents the probability difference The calculation formula is as follows: , Δ represents the probability difference between the generated y+ and y- by the optimized model relative to the reference model. The larger Δ is, the better. It prefers to generate neutral answers rather than biased answers. The reference model is usually the initial pre-trained model without fine-tuning. Its output probability is used as the optimization benchmark to prevent the optimized model from deviating too much from the original distribution during training, ensuring stability during the optimization process.

[0060] Data Flow Represents the calculation of the probability decoupling regularization term PDR, and its calculation formula is:

[0061]

[0062] when = 0 means that all large language models of BEC believe that y+ has no bias, then the regularization term degenerates to , which means that only the output probability of y- is suppressed; weight and Dynamically adjust the degree of suppression of y+ and y-, rather than just suppressing the output probability of y-, effectively covering the case where there is also a slight bias in y+. By embedding the negative log-likelihood, we get the probability decoupling regularization loss term :

[0063]

[0064] Where: represents the computational expectation, Represents the total set of debiased samples in the current training batch, i.e. the final is the expected loss of all debiased examples.

[0065] Data Flow Incorporating bias strength into the loss function of DPO to obtain a fine-grained debiasing loss term :

[0066]

[0067] Where: Represents the total set of debiased samples in the current training batch, i.e. the final is the expected loss of all debiased examples.

[0068] The loss function is obtained by Dynamically scale the loss value so that high bias intensity samples ( ) get stronger gradient updates, while low-intensity samples ( ) reduces the impact on model parameters, achieving differentiated optimization. When p < 0, the sign function sign(p) reverses the gradient direction, automatically correcting the labeling errors of the original data.

[0069] Data Flow Represents the integration of the final loss function. The formula of the total loss function can be expressed as:

[0070]

[0071] The total loss function integrates the fine-grained debiasing loss function and the probability decoupling regularization loss function. Balance. The larger it is, the stronger the suppression of bias sample output is. The smaller it is, the more obvious the increase in the probability of a neutral answer.

[0072] During each round of fine-tuning, the total loss function is calculated After that, the total loss function value can be minimized as the optimization goal, and the parameters of the large language model can be reversely updated through the optimization algorithm to achieve model fine-tuning.

[0073] Continue to see Figure 2 As shown, it shows the data flow of traditional DPO and the additional data flow after the present invention improves the DPO loss function. The data flow of traditional DPO directly uses the probability difference As a loss, the goal of DPO optimization is to maximize , that is, let the model Output The probability of outputting This design has two inherent flaws: first, the loss function cannot reflect the bias strength of different samples, thus failing to achieve targeted optimization; second, simply maximizing , may appear and The situation where both the bias and probability increase simultaneously will directly lead to the failure of debiasing. However, the improved total loss function proposed in this invention introduces the bias strength p-value and the probability decoupling regularization term R, which can prevent the increase of the probability of the model outputting biased answers while performing fine-grained debiasing. The balance coefficient λ is used to coordinate the fine-grained debiasing loss term and the regularization term.

[0074] It should be noted that the method steps shown in S1 to S2 above can essentially be implemented in the form of computer programs or software function modules.

[0075] Therefore, based on the same inventive concept, Figure 3 As shown, the present invention also provides a large language model fine-grained debiasing system based on bias strength, corresponding to the large language model fine-grained debiasing method based on bias strength provided in the above embodiment, which includes:

[0076] A group decision module is configured to obtain a training dataset consisting of debiased samples, where the debiased sample is a triple consisting of an input question, a neutral answer, and a biased answer. Each debiased sample is input into a bias assessment model group consisting of multiple large language models. A multi-model vote is performed on the more neutral of the two answers in the triple through group decision, and the bias strength of the debiased sample is calculated based on the multi-model vote result.

[0077] The fine-tuning training module uses the training data set to iteratively fine-tune the large language model in batches to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model can output debiased neutral answers to user questions; in each round of fine-tuning, the original large language model without fine-tuning is used as the reference model , the large language model obtained in the previous round of fine-tuning is used as the model to be optimized , by the reference model and the model to be optimized Based on the input questions in the debiased samples, the probabilities of neutral answers and biased answers in the debiased samples are obtained respectively. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss and the model to be optimized is reversely optimized. .

[0078] In addition, based on the same inventive concept, Figure 4 As shown, the present invention also provides a computer electronic device corresponding to the bias strength-based large language model fine-grained debiasing method provided in the above embodiment, which includes a memory and a processor;

[0079] The memory is used to store computer programs;

[0080] The processor is configured to implement the bias strength-based large language model fine-grained debiasing method as described above when executing the computer program;

[0081] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0082] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to the large language model fine-grained de-biasing method based on bias strength, and the storage medium stores a computer program. When the computer program is executed by a processor, it can implement the large language model fine-grained de-biasing method based on bias strength as described above.

[0083] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the large language model fine-grained debiasing method based on bias strength as described above.

[0084] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S2.

[0085] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0086] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0087] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0088] The present invention will further illustrate the detailed implementation process and technical effects of the bias strength-based fine-grained debiasing method for a large language model on a specific dataset through a specific embodiment, so as to facilitate understanding of the essence of the present invention.

[0089] Example

[0090] The steps in this embodiment are identical to the bias strength-based fine-grained debiasing method for a large language model described in steps S1 and S2 above. A detailed description will not be repeated here. Instead, this embodiment will focus on the specific dataset, some specific parameter settings, and implementation results. For ease of description, the method described in steps S1 and S2 will be referred to as the present invention.

[0091] 1. Test data set and its principle

[0092] In this implementation, BBQ (Bias Benchmark for QA) is designed to assess bias in large-scale question answering tasks using large language models. By constructing a manually annotated bias benchmark dataset, the system measures the bias performance of QA models in both ambiguous and disambiguated contexts. BBQ's core design focuses on nine protected social attributes, including age, gender identity, appearance, nationality, disability, and socioeconomic status.

[0093] The BBQ dataset systematically constructs test samples that can quantify bias through carefully designed context templates, question-answer pairs, and answer labels. Its data composition can be divided into the following three parts:

[0094] (1) Context template. Each test sample contains two types of context: ambiguous context and explicit context. In the ambiguous context, the model is tested to see whether it relies on social biases to make inferences. In the explicit context, the model is tested to see whether its inherent biases still interfere with its correct judgment.

[0095] (2) Symmetrical two-category questions. For each context template, there are two types of questions: negative questions and non-negative questions. This symmetrical design effectively distinguishes the model's preference for specific answers and avoids evaluation bias caused by a single question type.

[0096] (3) Answer labeling and verification mechanism. Each question has three answers for the LLM to choose from: target entity, reference entity, and unknown. In a clear context, the answer must strictly match the textual facts. In an ambiguous context, the correct answer for all questions is "Unknown," which is intended to require the model to acknowledge insufficient information rather than choose based on its stereotypes.

[0097] Therefore, based on the transformation of clear / fuzzy context and negative / non-negative questions, a test sample contains four sets of question-answer pairs. Furthermore, the positions of the target entity and the reference entity in the question template can be replaced.

[0098] To quantify bias, define the variable To reflect the number of stereotyped responses, two types are included: answering the target entity in negative questions and answering the control entity in non-negative questions. On the contrary, . Let’s first define the bias score in a clear context:

[0099]

[0100] in Indicates the answer of Unknown part in all answers in the clear context scenario, including and . The range is [-1, 1], 1 means that all LLM answers are biased, -1 means that all LLM answers are reverse biased, and 0 means no bias. for half.

[0101] The intuitive explanation of this design is that an unbiased LLM should make choices based entirely on the evidence given in a clear scenario. When the evidence points to the target entity, the choices of this LLM are all , when the evidence points to the control entity, the choices of this LLM are all Overall, and Should be equal, then Equal to 0. When When it is 0, it means that even if the contextual evidence clearly points to the control entity, the LLM selects the target entity according to its own biased views. In this case, the LLM is considered to be completely biased.

[0102] Furthermore, we define the bias score in the fuzzy context:

[0103] ]

[0104] Among them, accuracy is the accuracy of the answer in the fuzzy scene, that is, the proportion of unknown answers. The range of is also [-1, 1], where 0 means no bias. There are two cases where 0 is selected: LLM selects all as Unknown or occupy Half of. It should be noted that It is scaled using (1-accuracy), and But no, this is because choosing Unknown in a clear context scenario only indicates that the model is not good at following instructions, rather than being biased.

[0105] 2. Experimental Setup and Basic Model

[0106] The experiments in this example used a distributed training environment built on four NVIDIA RTX 4090 graphics cards, using the DeepSpeed ​​v1 framework for parallel training. The specific parameters were set as follows: AdamW was used as the optimizer, with an initial learning rate of 5e-5, dynamically adjusted using a cosine annealing learning rate scheduling strategy. The QLoRA efficient fine-tuning method was used, with the LoRA rank set to 32, the scaling factor (alpha) set to 16, and the LoRA target module set to Q / K / V. During training, the batch size per card (batch_size) was set to 4, and gradient accumulation (gradient_accumulation=4) was used to achieve an equivalent total batch size of 64. Each model was trained for three epochs, and the checkpoint with the best performance on the test set was selected as the final model. This configuration ensures training stability while optimizing computing resources through distributed acceleration and efficient parameter fine-tuning.

[0107] The value of β in both DPO and the loss function of the present invention is 0.1. Meanwhile, this embodiment selects the WizardLM 7BUncensored large language model for the debiasing experiment. This model is a large language model that has not been securely aligned.

[0108] In the bias assessment model set, this paper selects models based on the number of large language models (LLMs) in Table 1. To ensure diversity and cultural balance in assessment perspectives, the bias assessment model set is constructed according to the following principle: each time the k value is increased, a domestic model and an international mainstream model are introduced to cover bias perception patterns in different social contexts.

[0109] Table 1

[0110]

[0111] The symbol "+" in the above table means that a new large language model is further introduced as a BCE member based on the BCE member corresponding to the previous row.

[0112] 3. Training Dataset

[0113] The training dataset used in the experiments of this example was integrated from three public datasets: biasDPO, hh-rlhf, and PKU-SafeRLHF-QA. All data samples follow a triplet structure (prompt, chosen, rejected), where prompt represents the user question, chosen represents an unbiased answer, and rejected represents a biased answer. Specifically, biasDPO, a specialized bias suppression dataset, contains 1,145 data points covering typical bias types; hh-rlhf, a general human preference alignment dataset, contains 169,352 data points (160,800 in the training set and 8,552 in the test set), focusing on improving the safety and usefulness of answers; and PKU-SafeRLHF-QA, a fine-grained safety enhancement dataset constructed by Peking University, contains 265,223 data points (251,963 in the training set and 13,260 in the test set). Its Discriminatory Behavior subclass specifically targets discriminatory content and contains 6,430 data points.

[0114] To address the characteristics of non-professional bias datasets, we extracted relevant samples through targeted screening. For PKU-SafeRLHF-QA, we directly extracted 6,430 data points from the Discriminatory Behavior subcategory. For hh-rlhf, we used a large language model to perform semantic analysis on prompts in the data, filtering out 5,391 data points that induced bias. Finally, combined with the 1,145 original data points from biasDPO, we constructed a mixed training set containing 12,966 samples.

[0115] 4. Experimental Results

[0116] 4.1 Overall Results Comparison

[0117] When the number of LLMs k in the bias assessment model group is 8 and the balance coefficient λ is 0.7, the debiasing performance results of the original model (untuned WizardLM 7B Uncensored), traditional DPO optimization (WizardLM 7B Uncensored after traditional DPO fine-tuning), and optimization using the method of the present invention (WizardLM 7B Uncensored after optimization using the method of the present invention) are shown in Table 2 (the test indicators are BBQ / sDIS and BBQ / sAMB, with smaller values ​​indicating lower model bias).

[0118] Table 2

[0119]

[0120] As can be seen from Table 2, the proposed method achieves the best debiasing performance. Compared with the traditional DPO method, the proposed method achieves a significant decrease in bias scores in both explicit and model contexts, directly verifying the overall effectiveness of the proposed method.

[0121] In summary, the present invention implements a fine-grained debiasing training method by modeling bias strength, which can improve the debiasing effect of large language models in question-answering scenarios and is suitable for scenarios such as dialogue systems and text generation.

[0122] The embodiments described above are merely some preferred implementations of the present invention and are not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A fine-grained bias removal method for large language models based on bias strength, characterized by: include: S1. Obtain a training dataset consisting of debiased samples, where the debiased samples are triplets consisting of an input question, a neutral answer, and a biased answer. Input each debiased sample into a bias assessment model group consisting of multiple large language models. Perform multi-model voting on the more neutral of the two answers in the triplet through group decision making, and calculate the bias strength of the debiased sample based on the multi-model voting results. S2. Iteratively fine-tune the large language model in batches using the training dataset to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model outputs debiased neutral answers to user questions. In each round of fine-tuning, the original large language model without fine-tuning is used as the reference model, and the large language model obtained from the previous round of fine-tuning is used as the model to be optimized. The reference model and the model to be optimized respectively obtain the probabilities of neutral answers and biased answers in the debiased samples based on the input questions in the debiased samples. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss, and the model to be optimized is reversely optimized. When multi-model voting is performed through group decision-making, the triplet corresponding to each debiased sample is input into each large language model in the bias assessment model group. The large language model is driven by a prompt word to determine which of the neutral answer and the biased answer contained in the input triplet is more neutral and vote. The number of first votes that all large language models in the bias assessment model group believe that the neutral answer is more neutral and the number of second votes that believe that the biased answer is more neutral are counted. The ratio of the difference between the first and second votes to the sum is used as the bias strength of the debiased sample. During each round of fine-tuning, each debiased sample in the current training batch needs to input the input question contained in it into the reference model and the model to be optimized, and obtain the probability distribution of the word output by the model for each position in the answer text in the vocabulary. Based on the model output and the actual neutral answers and biased answers in the debiased sample, the first probability of the model to be optimized outputting a neutral answer, the second probability of the model to be optimized outputting a biased answer, the third probability of the reference model outputting a neutral answer, and the fourth probability of the reference model outputting a biased answer are calculated; The fine-grained debiasing loss term is a negative log-likelihood loss scaled by the absolute value of the bias strength, where the likelihood portion is the activation output multiplied by a weight factor, a sign function of the bias strength, and a probability difference, where the probability difference is calculated by subtracting the logarithm of the ratio of the second probability to the fourth probability from the logarithm of the ratio of the first probability to the third probability; The probability decoupling regularization loss term is a negative log-likelihood loss, where the likelihood part is the activation output after the negative of the weight factor is multiplied by the regularization term, and the regularization term is the weighted sum of two logarithmic terms. The first weighted term is the logarithm of the ratio of the first probability to the third probability, and the weight is the ratio of the second number of votes to the sum of the first number of votes and the second number of votes. The second weighted term is the logarithm of the ratio of the second probability to the fourth probability, and the weight is the ratio of the first number of votes to the sum of the first number of votes and the second number of votes.

2. The large language model fine-grained debiasing method based on bias strength according to claim 1, characterized in that: The number of large language models included in the bias assessment model group is an even number not less than 2, and each of the included large language models has been pre-safely aligned.

3. A fine-grained bias removal system for large language models based on bias strength, characterized by: include: A group decision module is configured to obtain a training dataset consisting of debiased samples, where the debiased sample is a triple consisting of an input question, a neutral answer, and a biased answer. Each debiased sample is input into a bias assessment model group consisting of multiple large language models. A multi-model vote is performed on the more neutral of the two answers in the triple through group decision, and the bias strength of the debiased sample is calculated based on the multi-model vote result. The fine-tuning training module uses the training dataset to iteratively fine-tune the large language model in batches to achieve fine-grained debiasing optimization and prevent the increase in the probability of generating biased answers, so that the fine-tuned large language model outputs debiased neutral answers to user questions. In each round of fine-tuning, the original large language model without fine-tuning is used as the reference model, and the large language model obtained by the previous round of fine-tuning is used as the model to be optimized. The reference model and the model to be optimized respectively obtain the probabilities of neutral answers and biased answers in the debiased samples based on the input questions in the debiased samples. The fine-grained debiasing loss term and the probabilistic decoupling regularization loss term are calculated by combining the multi-model voting results and the bias strength. The weighted sum of the two loss terms is used as the total loss, and the model to be optimized is reversely optimized. When multi-model voting is performed through group decision-making, the triplet corresponding to each debiased sample is input into each large language model in the bias assessment model group. The large language model is driven by a prompt word to determine which of the neutral answer and the biased answer contained in the input triplet is more neutral and vote. The number of first votes that all large language models in the bias assessment model group believe that the neutral answer is more neutral and the number of second votes that believe that the biased answer is more neutral are counted. The ratio of the difference between the first and second votes to the sum is used as the bias strength of the debiased sample. During each round of fine-tuning, each debiased sample in the current training batch needs to input the input question contained in it into the reference model and the model to be optimized, and obtain the probability distribution of the word output by the model for each position in the answer text in the vocabulary. Based on the model output and the actual neutral answers and biased answers in the debiased sample, the first probability of the model to be optimized outputting a neutral answer, the second probability of the model to be optimized outputting a biased answer, the third probability of the reference model outputting a neutral answer, and the fourth probability of the reference model outputting a biased answer are calculated; The fine-grained debiasing loss term is a negative log-likelihood loss scaled by the absolute value of the bias strength, where the likelihood portion is the activation output multiplied by a weight factor, a sign function of the bias strength, and a probability difference, where the probability difference is calculated by subtracting the logarithm of the ratio of the second probability to the fourth probability from the logarithm of the ratio of the first probability to the third probability; The probability decoupling regularization loss term is a negative log-likelihood loss, where the likelihood part is the activation output after the negative of the weight factor is multiplied by the regularization term, and the regularization term is the weighted sum of two logarithmic terms. The first weighted term is the logarithm of the ratio of the first probability to the third probability, and the weight is the ratio of the second number of votes to the sum of the first number of votes and the second number of votes. The second weighted term is the logarithm of the ratio of the second probability to the fourth probability, and the weight is the ratio of the first number of votes to the sum of the first number of votes and the second number of votes.

4. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the large language model fine-grained debiasing method based on bias strength as claimed in claim 1 or 2 can be implemented.

5. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the large language model fine-grained debiasing method based on bias strength as claimed in claim 1 or 2.

6. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the large language model fine-grained debiasing method based on bias strength as claimed in claim 1 or 2 when executing the computer program.

Images