Neuron role contribution-based model quantification fairness improvement method

Abstract

The invention discloses a model quantification fairness improvement method based on neuron role contribution. The method comprises the following steps: step 1, obtaining a model to be quantized and constructing a sample pair; 2, performing prejudice analysis on each layer of neurons in the model obtained in the step 1, and performing neuron role identification; step 3, optimizing the model based on a differential quantization strategy; and 4, constructing a fairness-improved lightweight model, and constructing the lightweight model by using the multi-channel optimization vector representation generated in the step 3. According to the model quantification fairness improvement method based on the neuron role contribution, the biased neuron and the normal neuron are processed respectively, the quantification strategy is optimized, and balance between fairness and performance is achieved.

Description

Technical Field

[0001] The present invention relates to the field of model compression security, and more specifically to a method for improving model quantization fairness based on neuron role contribution. Background Art

[0002] With the widespread application of deep neural networks (DNN) in image recognition, natural language processing, recommendation systems and other fields, the complexity of models has increased dramatically, posing a huge challenge to the storage and computing capabilities of edge devices. To address this problem, model compression technologies such as model pruning, knowledge distillation, tensor decomposition, and model quantization have been widely studied and applied, among which model quantization is particularly prominent in the industry for its high efficiency and practicality.

[0003] Since most existing compression works pursue model compression and acceleration, the robustness vulnerability of DNN introduced by the compression model is an often overlooked problem. For example, the inherent unfairness problem of a biased deep neural network (DNN) model may be further amplified after compression. Existing studies have shown that machine learning models, especially DNNs, may make unfair predictions for vulnerable groups during training and use due to data bias or improper algorithm design. For example, when lending institutions such as banks and trust companies strive to improve the efficiency of their loan systems, they often tend to obtain pre-trained models directly from public model libraries such as HuggingFace. This convenient deployment method is usually accompanied by risks, that is, due to the lack of in-depth professional analysis and comprehensive security assessment, the downloaded model may have inherent subtle biases.

[0004] In addition, in order to further improve the system's response speed and adaptive processing capabilities, lending institutions will also use model compression technologies such as quantization to customize the downloaded model for secondary development and optimization, and then deploy it into the loan approval system. Unfortunately, during the supply chain development process, if the bias in the pre-trained model cannot be effectively identified and eliminated, it will easily be retained and embedded in the downstream system.

[0005] When such a system evaluates loan applications based on biased models, it may lead to biased approval results. Specifically, some applicants who should have stable repayment ability and meet loan conditions, such as applicants who are temporarily unhoused, may be mistakenly classified as high-risk borrowers due to the inherent bias of the system, and face the dilemma of having their loan applications rejected or having to bear higher interest rates. This harms the interests of applicants and seriously violates the principles of fairness and justice that loan approval should follow. Biased models will have problems when applied to real-world scenarios, and those unbiased models may also inadvertently introduce bias because they do not consider implicit variables. Therefore, the fairness guarantees of traditional large models and lightweight models deployed at the edge should be widely studied and explored in practical applications.

[0006] There has been a lot of research on DNN debiasing methods, such as reweighting, adversarial training, etc. These methods can reduce bias, but their debiasing effect has not been proven for the special requirements of model quantization, that is, these debiasing methods are not specifically designed to ensure model accuracy while debiasing. Intuitively speaking, debiasing and quantizing models can be achieved in three ways, namely, debiasing the model before quantization: ensuring the prediction or classification accuracy of the quantized model, but new bias problems may be introduced in the process of fine-tuning the quantized model, and it is difficult to ensure that the quantized model is still unbiased; debiasing the quantized model: ensuring the fairness of the compressed model, but the effort to repair the accuracy loss caused by debiasing is usually costly; debiasing during quantization: ensuring that the quantized model is effectively debiased while ensuring model accuracy is a promising solution, but there is currently no relevant research. Summary of the invention

[0007] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for improving the fairness of model quantization based on neuron role contribution. The method debiases the model while quantizing, and reduces the model size and bias impact while ensuring the model accuracy. By analyzing the model activation value, it is found that abnormally high neurons will cause output deviation, and the optimal threshold is selected by Bayesian optimization to balance the main task performance and debiasing effect, without introducing new data, reducing complexity.

[0008] To achieve the above object, the present invention provides the following technical solution: a model quantification fairness improvement method based on neuron role contribution, comprising the following steps: Step 1: Obtain the model to be quantified and construct sample pairs; Step 2: Perform bias analysis on each layer of neurons in the model obtained in step 1 to identify the neuron roles; Step 3: Optimize the model based on differentiated quantitative strategies; Step 4: Build a lightweight model with improved fairness. Use the multi-channel optimized vector representation generated in step 3 to build a lightweight model. Then, by evaluating the binary quantization effect of biased neurons and the flexible quantization effect of normal neurons, comprehensively consider the fairness and performance of the quantization model, and finally complete the deployment of the quantization model on the loan approval system.

[0009] As a further improvement of the present invention, the specific steps of obtaining the model to be quantified and constructing the sample pair in step 1 are as follows: Step 1, obtaining the loan approval system model to be quantized and the required quantization bit width; Step 1 and 2: Obtain relevant data samples of the applicant, including but not limited to: the applicant's income level, occupation category, credit score, financial history, and sensitive attributes of the applicant, and then record the data samples as , , where the total number of sample pairs is , No. The normal attribute data of an applicant is recorded as , will The data of applicants changing their housing status attributes is recorded as .

[0010] As a further improvement of the present invention, the specific steps of performing neuron role identification in step 2 are as follows: Step 21: Input the sample pairs obtained in step 12 into the model obtained in step 11, calculate the activation value differences generated by these sample pairs on each layer of neurons, and use the Tanh function to normalize the activation value differences to obtain the activation difference , calculate the activation value difference; Step 22: Introduce the bias neuron rate as the ordinate, draw the BNI curve, and use the Bayesian optimization method of Gaussian process to evaluate the impact of different thresholds on classification accuracy and bias mitigation effect, and iteratively search for the optimal threshold. ,Finally we distinguish biased neurons from normal neurons, ,complete the division of neuron roles, and complete the identification of biased ,neurons.

[0011] As a further improvement of the present invention, the specific steps of calculating the activation value difference in step 21 are as follows: Step 2: Input the sample pairs with different sensitive attribute states into the model; Step 2: Calculate the difference in activation values ​​generated by these sample pairs on neurons in each layer. The specific calculation formula is as follows: in, It is Tier The activation difference of neurons , is the total number of sample pairs, is the target model; Step 2-3: Use the Tanh function to normalize the activation value difference to get the activation difference .

[0012] As a further improvement of the present invention, the specific steps of bias neuron identification in step 22 are as follows: Step 221: Activation difference after processing with Tanh function The BNI curve was plotted as the abscissa and the bias neuron rate as the ordinate; Step 222: Through the BNI curve and The intersection initialization threshold ; Step 223: Use the Bayesian optimization algorithm to optimize the initial threshold determined by the BNI curve, with the classification accuracy as the target, and iterate the optimization. ; Step 224: Define the objective function as the loss function for model training, as shown below: in, is the sample size, is the neuron discrimination threshold, is the true label of the sample, For the Input features of samples, including sensitive attributes , For the model The predicted probability of samples; Step 225: Add neuron judgment to the model layer, that is ,but , simulate the removal of biased neurons in the Bayesian optimization process, and optimize the threshold ; Step 226: In Bayesian optimization, use the threshold obtained from the BNI curve As the initial threshold, the optimization threshold range is limited to ,in, is a hyperparameter; Step 227: Alternative Models in Bayesian Optimization Based on Gaussian Processes The fitting objective function is expressed as: in, is the objective function, i.e., the classification accuracy of the optimization model. is the mean function, is the covariance function; Step 228: Use expected improvements As the acquisition function used, it is defined as follows: in, is the known optimal point, and the objective function is at point The distribution at is Gaussian , denoted as , and is a posteriori The mean and standard deviation of in, A standardized value indicating the degree of improvement, Mean value point The uncertainty at the threshold is large, and the larger uncertainty encourages the algorithm to explore more thresholds. The possibility of and are the standard normal distribution cumulative distribution function and the standard normal distribution probability density function, Used to measure the probability of improvement, the scale used to measure improvement; Step 229: Determine the optimal threshold based on Bayesian optimization To distinguish normal neurons from biased neurons; among them, normal neurons are defined as neurons that are strongly correlated with the performance of the model's main task, and biased neurons are defined as neurons that may be overly sensitive to certain sensitive features.

[0013] As a further improvement of the present invention, the specific steps of optimizing the model based on the differentiated quantization strategy in step 3 are as follows: Step 31, binary quantization processing is performed on the bias neurons; Step 32: For normal neurons that are not identified as biased neurons, a conventional quantization strategy is used, using flexible multi-bit quantization. The quantization formula is as follows: in, is the weight of the full-precision model, is the offset, is the quantitative scaling factor of the model, calculated as: in, is the number of quantization bits, is the set of weights to be quantized, ; Step 33: According to the quantization strategy, adjust the weight of each layer of the model to generate a multi-channel optimized vector representation.

[0014] The beneficial effects of the present invention are as follows: 1) by analyzing the role contribution of neurons, biased neurons and normal neurons are distinguished, and mixed precision quantization is used to effectively reduce model bias, while maximally maintaining the main task performance of the model; 2) the present invention does not require the introduction of additional training data or labels for debiasing operations; 3) the present invention has strong adaptability in both structured data models and unstructured data scenarios, and provides flexible support for a variety of quantization bit widths, thereby helping downstream systems to achieve efficient and fair model deployment and performance optimization in different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is the overall flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the BNI curve of the method of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0017] Reference Figure 1 to Figure 2 As shown, a model quantization fairness improvement method based on neuron role contribution in this embodiment includes the following steps: 1) Obtain the model that needs to be quantified and construct sample pairs 1.1) Obtain the model. Obtain the loan approval system model to be quantized and the required quantization bit width. The present invention takes 8-bit quantization bit width as an example; 1.2) Construct data sample pairs. The present invention uses housing status as a sensitive attribute to introduce the subsequent methods (housing status can be divided into owning a property or renting a house. Some applicants rent a house but their income is enough to repay the loan). Obtain relevant data samples of the applicant, including but not limited to: the applicant's income level, occupation category, credit score, financial history and other characteristic data, as well as the applicant's housing status attribute (sensitive attribute). The data sample pair is denoted as , , where the total number of sample pairs is , No. The normal attribute data of an applicant is recorded as , will The data of applicants changing their housing status attributes is recorded as The constructed sample pairs are used to analyze the response behavior of model neurons to different sensitive attribute states, providing a basis for subsequent neuron role identification.

[0018] 2) Neuron role identification In the present invention, biased neurons and normal neurons are defined according to the different neuron activation differences of sample pairs. In the neuron role identification step, bias analysis is first performed on each layer of neurons in the quantization model to calculate the activation difference. Secondly, the threshold for dividing biased neurons and normal neurons is generated based on the drawn bias neuron index (Bias Neuron Identification, BNI). Finally, the specific steps are as follows: 2.1) Calculation of activation value difference. The sample pairs obtained in step 1.2) (samples with different sensitive attribute states) are input into the model obtained in step 1.1), and the activation value differences generated by these sample pairs on each layer of neurons are calculated. The activation value differences are normalized using the Tanh function to obtain the activation difference. ; 2.1.1) Input sample pairs with different sensitive attribute states into the model.

[0019] 2.1.2) Calculate the difference in activation values ​​generated by these sample pairs on neurons in each layer. The specific calculation formula is as follows: (1) in, It is Tier The activation difference of neurons , is the total number of sample pairs, is the target model.

[0020] 2.1.3) Use the Tanh function to normalize the activation value difference and get the activation difference .

[0021] 2.2) Biased neuron identification. The biased neuron rate is introduced as the ordinate, and the BNI curve is drawn. The Bayesian optimization method of Gaussian process is used to evaluate the impact of different thresholds on classification accuracy and bias mitigation effect, and the optimal threshold is iteratively searched. , and finally distinguish biased neurons from normal neurons, completing the division of neuron roles.

[0022] 2.2.1) Activation difference after processing with Tanh function The BNI curve is plotted with the bias neuron rate as the horizontal axis and the bias neuron rate as the vertical axis.

[0023] 2.2.2) Through the BNI curve and The intersection initialization threshold ,like Figure 2 The BNI curve is shown in the schematic diagram.

[0024] 2.2.3) Use the Bayesian optimization algorithm to optimize the initial threshold determined by the BNI curve, with the classification accuracy as the target, and iterate the optimization .

[0025] 2.2.4) Define the objective function as the loss function of model training, as follows: (2) in is the sample size, is the neuron discrimination threshold, is the true label of the sample, For the Input features of samples, including sensitive attributes , For the model The predicted probability of a sample.

[0026] 2.2.5) Add neuron judgment to the model layer, that is ,but , simulate the removal of biased neurons in the Bayesian optimization process, and optimize the threshold .

[0027] 2.2.6) In Bayesian optimization, the threshold obtained from the BNI curve is used As the initial threshold, the optimization threshold range is limited to ,in is a hyperparameter.

[0028] 2.2.7) Alternative models in Bayesian optimization based on Gaussian processes The fitting objective function is expressed as: (3) in is the objective function, i.e., the classification accuracy of the optimization model. is the mean function, is the covariance function.

[0029] 2.2.8) Use expected improvements As the acquisition function used, it is defined as follows: (4) in is the known optimal point, and the objective function is at point The distribution at is Gaussian , denoted as , and is a posteriori The mean and standard deviation of .

[0030] (5) in, A standardized value indicating the degree of improvement, Mean value point The uncertainty at the threshold is large, and the larger uncertainty encourages the algorithm to explore more thresholds. The possibility of and They are the standard normal distribution cumulative distribution function (CDF) and the standard normal distribution probability density function (PDF), Used to measure the probability of improvement, Used to measure the scale of improvement.

[0031] 2.2.9) Determine the optimal threshold based on Bayesian optimization To distinguish between normal neurons and biased neurons. Normal neurons are defined as neurons that are strongly related to the performance of the model's main task, and if they are mishandled, they will reduce the performance of the model. Biased neurons are defined as neurons that may be overly sensitive to certain sensitive features. These features are not directly related to the model's prediction task, but may be inadvertently learned by the model and affect the decision-making process.

[0032] 3) Optimization model based on differentiated quantitative strategy Based on the normal neurons and bias neurons determined in step 2), different quantification strategies are adopted for neurons with different roles.

[0033] 3.1) Binary quantization of bias neurons The biased neurons identified in step 2) are quantized to greatly reduce their biased impact on model decisions and improve the resource utilization efficiency of the model. Using binary quantization technology, the weights or activation values ​​of biased neurons are limited to {−1,1}. Binarization greatly simplifies the output representation of neurons and reduces the bit requirements of the model in storage and calculation. This strategy significantly reduces the impact of biased neurons in the decision-making process, thereby reducing the impact of the bias they may introduce on the overall judgment of the model and improving the fairness of the model.

[0034] 3.2) Use conventional quantification strategies for normal neurons A conventional quantization strategy is used for normal neurons that are not identified as bias neurons to maximize model performance while optimizing resource usage. Normal neurons use flexible multi-bit quantization (such as 4 bits, 6 bits, or 8 bits) according to quantization requirements. The quantization formula is as follows: (6) in is the weight of the full-precision model, is the offset, is the quantitative scaling factor of the model, calculated as: (7) in, is the number of quantization bits (such as 4 bits, 6 bits, 8 bits), is the weight set to be quantized. Symmetric quantization is used in the present invention. .

[0035] Differentiated quantization can minimize the accuracy loss caused by quantization while retaining the expressive power of neurons, ensuring that the model still has high performance after quantization.

[0036] 3.3) Adjust the quantized weights of each layer of neurons According to the quantization strategy, the weight of each layer of the model is adjusted to generate a multi-channel optimized vector representation. Differentiated quantization strategies are applied to neurons in each layer, and biased neurons and normal neurons use differentiated quantization precision respectively. Multi-channel representation ensures that the quantization results of neurons in each layer can meet the dual requirements of fairness and performance. The generated multi-channel optimized vector provides efficient weight distribution support for the subsequent lightweight model construction.

[0037] 4) Lightweight model construction to improve fairness 4.1) Using the multi-channel optimized vector representation generated in step 3), build a lightweight model; 4.2) By evaluating the binary quantization effect of biased neurons and the flexible quantization effect of normal neurons, the fairness and performance of the quantization model are comprehensively considered, and finally the deployment of the quantization model on the loan approval system is completed.

[0038] In summary, this embodiment aims at the existing model quantization method that fails to effectively solve the model bias problem. The traditional quantization strategy uniformly processes all neurons, which is easy to amplify the bias and affect the fairness and performance of the model. A model quantization fairness improvement method based on the contribution of neuron roles is proposed. By separately processing biased neurons and normal neurons, the quantization strategy is optimized to achieve a balance between fairness and performance. Specifically, the neuron role is first identified, and the sample pairs with different sensitive attribute states are input into the model to calculate the difference in neuron activation values, and the difference is normalized to the biased neuron index (Bias Neuron Identification, BNI) through the Tanh function. The BNI curve is drawn in combination with the biased neuron rate to obtain the initial threshold, and then the Bayesian optimization algorithm is used to predict and evaluate the effects of different thresholds, and the optimal threshold is iteratively found to distinguish biased neurons from normal neurons. Subsequently, differentiated quantization is performed according to the neuron role, and binary quantization is used for biased neurons to reduce the impact of bias; a flexible quantization strategy is used for normal neurons to balance performance and resource consumption. Finally, a fair quantization model is generated, which effectively alleviates the model bias problem caused by biased neurons while compressing the model scale, and ensures the fairness and performance of the model.

[0039] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A model quantification fairness improvement method based on neuron role contribution, characterized by: The steps include: Step 1: Obtain the model to be quantified and construct sample pairs; Step 2: Perform bias analysis on each layer of neurons in the model obtained in step 1 to identify the neuron roles; Step 3: Optimize the model based on differentiated quantitative strategies; Step 4: Build a lightweight model with improved fairness. Use the multi-channel optimized vector representation generated in step 3 to build a lightweight model. Then, by evaluating the binary quantization effect of biased neurons and the flexible quantization effect of normal neurons, comprehensively consider the fairness and performance of the quantization model, and finally complete the deployment of the quantization model on the loan approval system.

2. The method for improving fairness of model quantification based on neuron role contribution according to claim 1, characterized in that: The specific steps of obtaining the model to be quantified and constructing sample pairs in step 1 are as follows: Step 1, obtaining the loan approval system model to be quantized and the required quantization bit width; Step 1 and 2: Obtain relevant data samples of the applicant, including but not limited to: the applicant's income level, occupation category, credit score, financial history, and sensitive attributes of the applicant, and then record the data samples as , , where the total number of sample pairs is , No. The normal attribute data of an applicant is recorded as , will The data of applicants changing their housing status attributes is recorded as .

3. The method for improving fairness of model quantification based on neuron role contribution according to claim 2, characterized in that: The specific steps of performing neuron role identification in step 2 are as follows: Step 21: Input the sample pairs obtained in step 12 into the model obtained in step 11, calculate the activation value differences generated by these sample pairs on each layer of neurons, and use the Tanh function to normalize the activation value differences to obtain the activation difference , calculate the activation value difference; Step 22: Introduce the bias neuron rate as the ordinate, draw the BNI curve, and use the Bayesian optimization method of Gaussian process to evaluate the impact of different thresholds on classification accuracy and bias mitigation effect, and iteratively search for the optimal threshold. ,Finally we distinguish biased neurons from normal neurons, ,complete the division of neuron roles, and complete the identification of biased ,neurons.

4. The method for improving fairness of model quantification based on neuron role contribution according to claim 3, characterized in that: The specific steps of calculating the activation value difference in step 21 are as follows: Step 2: Input the sample pairs with different sensitive attribute states into the model; Step 2.12: Calculate the difference in activation values ​​generated by these sample pairs on neurons in each layer. The specific calculation formula is as follows: ； in, It is Tier The activation difference of neurons , is the total number of sample pairs, is the target model; Step 2-3: Use the Tanh function to normalize the activation value difference to get the activation difference .

5. The method for improving fairness of model quantification based on neuron role contribution according to claim 3 or 4, characterized in that: The specific steps of bias neuron identification in step 22 are as follows: Step 221: Activation difference after processing with Tanh function The BNI curve was plotted as the abscissa and the bias neuron rate as the ordinate; Step 222: Through the BNI curve and The intersection initialization threshold ; Step 223: Use the Bayesian optimization algorithm to optimize the initial threshold determined by the BNI curve, with the classification accuracy as the target, and iterate the optimization. ; Step 224: Define the objective function as the loss function of model training, as shown below: ； in, is the sample size, is the neuron discrimination threshold, is the true label of the sample, For the Input features of samples, including sensitive attributes , For the model The predicted probability of samples; Step 225: Add neuron judgment to the model layer, that is ,but , simulate the removal of biased neurons in the Bayesian optimization process, and optimize the threshold ; Step 226: In Bayesian optimization, use the threshold obtained from the BNI curve As the initial threshold, the optimization threshold range is limited to ,in, is a hyperparameter; Step 227: Alternative Models in Bayesian Optimization Based on Gaussian Processes The fitting objective function is expressed as: ； in, is the objective function, i.e., the classification accuracy of the optimization model. is the mean function, is the covariance function; Step 228: Use expected improvements As the acquisition function used, it is defined as follows: ； in, is the known optimal point, and the objective function is at point The distribution at is Gaussian , denoted as , and is a posteriori The mean and standard deviation of ； in, A standardized value indicating the degree of improvement, Mean value point The uncertainty at the threshold is large, and the larger uncertainty encourages the algorithm to explore more thresholds. The possibility of and are the standard normal distribution cumulative distribution function and the standard normal distribution probability density function, Used to measure the probability of improvement, the scale used to measure improvement; Step 229: Determine the optimal threshold based on Bayesian optimization To distinguish normal neurons from biased neurons; among them, normal neurons are defined as neurons that are strongly correlated with the performance of the model's main task, and biased neurons are defined as neurons that may be overly sensitive to certain sensitive features.

6. The method for improving fairness of model quantification based on neuron role contribution according to claim 3 or 4, characterized in that: The specific steps for optimizing the model based on the differentiated quantitative strategy in step 3 are as follows: Step 31, binary quantization processing is performed on the bias neurons; Step 32: For normal neurons that are not identified as biased neurons, a conventional quantization strategy is used, using flexible multi-bit quantization. The quantization formula is as follows: ； in, is the weight of the full-precision model, is the offset, is the quantitative scaling factor of the model, calculated as: ； in, is the number of quantization bits, is the set of weights to be quantized, ; Step 33: According to the quantization strategy, adjust the weight of each layer of the model to generate a multi-channel optimized vector representation.

Images