Deep neural network pruning technology-oriented fairness optimization and image classification method

Abstract

The invention discloses a deep neural network pruning technology-oriented fairness optimization and image classification method, and relates to the technical field of artificial intelligence security, and the fairness optimization method comprises the steps of obtaining a training data set; training the initial neural network model through the training data set to obtain a neural network model; pruning the neural network model; obtaining geometric metrics of different sensitive attribute subgroup samples through the training data set and the pruned neural network model, carrying out average processing on the geometric metrics of the different sensitive attribute subgroup samples, calculating a range fairness constraint term, and calculating total loss by using the range fairness constraint term and a classification loss term. And performing fine tuning training on the pruned neural network model through total loss to obtain a target pruning model. According to the method, the inter-group performance can be balanced on the feature level, so that the defect that the existing research cannot be directly deployed in terminal equipment with limited resources and can not be quickly integrated with the existing pruning technology is overcome.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security technology, and in particular to fairness optimization and image classification methods for deep neural network pruning technology. Background Art

[0002] Deep neural networks have demonstrated excellent performance in a variety of deep learning tasks, including image classification, face recognition, object detection, and semantic segmentation. However, as the complexity of the tasks increases, the model architecture gradually becomes larger, which poses a significant challenge to model deployment on terminal devices. To address this problem, model compression technology is widely used to reduce the number of model parameters and computational costs to maintain model efficiency while ensuring prediction accuracy. As one of the main means of model compression, pruning technology aims to optimize the model structure, reduce resource consumption, and minimize the impact on model performance by identifying and removing "redundant" or "unnecessary" weights in the model.

[0003] However, the application of pruning technology may not only introduce new biases, but also aggravate existing biases, which is manifested in the widening performance differences between different subgroups of the model after pruning. For example, in the face classification task, the pruned model may enhance the recognition accuracy of a specific ethnic group (such as people with white skin), while reducing the recognition accuracy of another ethnic group (such as people with darker skin). This phenomenon becomes more obvious as the degree of pruning increases. Similarly, unfair artificial intelligence systems may also lead to serious consequences in practical applications. For example, unfairness in the classification of chest X-ray images may lead to medical diagnosis errors; and unfairness in face recognition may intensify discrimination against specific social groups. If the original model already has bias, the pruning process may further amplify these unfair phenomena.

[0004] Although there is evidence that the pruning process may amplify model bias, there is still limited understanding of how pruning specifically affects model fairness. Existing studies have attempted to explain this phenomenon by analyzing the differences in gradient norms and Hessian matrices between different groups. However, such factors are computationally expensive and difficult to operate when faced with models with a large number of parameters, and fail to provide practical guidance for alleviating bias caused by pruning. Furthermore, existing fair pruning schemes often require complex calculations and sorting of all parameters, or the introduction of additional network modules, which not only increases computational complexity and storage requirements, but also runs counter to the goal of pruning to simplify the model. At the same time, existing research mainly designs specific fair pruning frameworks, which cannot be flexibly integrated into different pruning techniques.

[0005] Therefore, it is particularly important to explore a fairness optimization and image classification method for deep neural network pruning technology that can be directly deployed in resource-limited terminal devices and quickly integrated with existing pruning technologies. Summary of the invention

[0006] In order to solve the technical problems existing in the above-mentioned prior art, the present invention proposes a fairness optimization and image classification method for deep neural network pruning technology, aiming to solve the shortcomings of existing research that it cannot be directly deployed in terminal devices with limited resources and cannot be quickly integrated with existing pruning technology.

[0007] On the one hand, to achieve the above-mentioned purpose, the present invention provides a fairness optimization method for deep neural network pruning technology, comprising:

[0008] Obtaining a training data set, wherein the training data set is image data including target labels and sensitive attributes;

[0009] Train the initial neural network model using the training data set to obtain a neural network model;

[0010] Pruning the neural network model;

[0011] The geometrical metrics of samples of different sensitive attribute subgroups are obtained through the training data set and the pruned neural network model, and the geometrical metrics of samples of different sensitive attribute subgroups are averaged to calculate the extreme fairness constraint item, and the overall loss function is calculated using the extreme fairness constraint item and the classification loss item. The pruned neural network model is fine-tuned through the overall loss function to obtain the target neural network pruning model.

[0012] Preferably, the neural network model is used to extract sample feature vectors of the training data set, and the neural network model includes a feature extractor and a classifier.

[0013] Preferably, pruning the neural network model includes:

[0014] Set the pruning rate;

[0015] Calculate the L1 norm of all parameters of the neural network model and sort them;

[0016] The sorted parameters are proportionally set to zero according to the pruning rate to prune the neural network model.

[0017] Preferably, during the process of pruning the neural network model, the last classification layer in the neural network model is not pruned.

[0018] Preferably, calculating the extreme fairness constraint item includes:

[0019] The training image data set is divided into several batches. In the training of each batch, a feature vector is obtained after each sample is input into a feature extractor, and a geometric metric corresponding to each sample is calculated;

[0020] The average length and average angle of K sensitive attribute subgroup samples in the current batch are respectively counted, and the extreme fairness constraint item is calculated according to the average length and average angle.

[0021] Preferably, the geometric measurement corresponding to the sample includes the length of the sample and the angle of the sample, the length of the sample is calculated using the L1 norm of the feature vector, and the angle of the sample is calculated using the classification weight corresponding to the feature vector and the target label of the feature vector.

[0022] Preferably, calculating the extreme fairness constraint item through the average length and the average angle includes:

[0023] The difference between the maximum geometric metric and the minimum geometric metric among the K sensitive attribute subgroups is measured, and the extreme fairness constraint item is calculated, specifically:

[0024]

[0025]

[0026] L fair =λ length L length +λ angle L angle ;

[0027] Where, L length is the range length constraint, is the maximum average length of the K sensitive attribute subgroups, is the minimum average length of the K sensitive attribute subgroups, L angle is the extreme angle constraint term, is the maximum average angle among the K sensitive attribute subgroups, is the minimum average angle among the K sensitive attribute subgroups, L fair is the extreme fairness constraint, λ length is the weight coefficient of the extreme length constraint, λ angle is the weight coefficient of the extreme angle constraint.

[0028] Preferably, the total loss function of the target neural network pruning model includes a classification loss term and a fairness loss term, wherein the classification loss term uses a cross entropy loss function term to calculate the classification loss of the sample, and the fairness loss term uses a geometric metric range that constrains different sensitive subgroups to calculate the fairness loss of the sample;

[0029] Wherein, the total loss function is:

[0030] L=L ce +Lfair ;

[0031] In the formula, L is the total loss function, L ce is the classification loss term, L fair It is a fair loss item.

[0032] Preferably, in the stage of fine-tuning training, an improved total loss function is used for training, and the remaining parameters of the model are gradient updated by the Adam algorithm.

[0033] On the other hand, to achieve the above object, the present invention also provides an image classification method, comprising:

[0034] Get the image to be classified;

[0035] The image to be classified is input into the neural network pruning model constructed by the method to obtain a classification result.

[0036] Compared with the prior art, the present invention has the following advantages and technical effects:

[0037] (1) The present invention can be directly deployed on the model of the terminal device to improve fairness. Since the geometric metric used in the present invention is easy to calculate, it only needs the intermediate variables of conventional training to calculate it, without additional calculation, and does not require the use of additional network modules. This is more suitable for the actual deployment scenario of deep neural networks. The method of the present invention takes into account the relationship between the effect of pruning and practical application, and has high operability and practicality.

[0038] (2) The present invention can be quickly integrated into the training phase of an existing pruning algorithm, thereby directly improving the fairness effect of the pruning algorithm that only considers computational efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A flowchart of a fairness optimization method for deep neural network pruning technology according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram for visualizing the practical significance of geometric measurements in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] This embodiment discloses a fairness optimization method for deep neural network pruning technology, which aims to solve the potential bias problem introduced by existing pruning technology in applications and the shortcomings of existing research that cannot be directly deployed in terminal devices with limited resources and quickly integrated with existing pruning technology, so as to achieve the purpose of optimizing the fairness of the pruning model.

[0045] Fairness optimization methods for deep neural network pruning techniques, such as Figure 1 ,include:

[0046] Obtain a training data set, where the training data set is image data containing target labels and sensitive attributes;

[0047] Train the initial neural network model through the training data set to obtain the neural network model;

[0048] Prune the neural network model;

[0049] Through the training data set and the pruned neural network model, the geometric measurements of samples of different sensitive attribute subgroups are obtained, and the geometric measurements of samples of different sensitive attribute subgroups are averaged, and the extreme fairness constraint item is calculated. The overall loss function is calculated using the extreme fairness constraint item and the classification loss item. The pruned neural network model is fine-tuned through the overall loss function to obtain the target neural network pruning model.

[0050] Specifically, first, a training image dataset including target labels and sensitive attributes is obtained; secondly, an initial neural network model is trained to extract sample feature vectors; then, the trained neural network model is pruned according to the set pruning rate; the average geometric metric of each sensitive attribute subgroup of the pruned model in each batch training is calculated, and an extreme fairness constraint term is designed; finally, the model is iteratively trained using the classification loss term and the fairness loss term, and the model gradient is updated to finally obtain a pruned model that is both fair and accurate.

[0051] Furthermore, a training image data set is obtained. In this embodiment, the data set includes multiple samples, each of which includes training image data, a corresponding target label, and sensitive attributes (such as gender, race, etc.).

[0052] Specifically, a high-accuracy neural network model is trained on the acquired data set. The neural network model includes a feature extractor and a classifier, which can effectively extract the sample feature vector X and achieve accurate classification, thereby more accurately calculating the geometric measurement of the sample.

[0053] Furthermore, pruning the neural network model includes:

[0054] Set the pruning rate p%. The larger the p%, the more weights are set to zero or directly removed, and the smaller the capacity of the neural network and the more efficient it is.

[0055] Calculate the L1 norm of all parameters of the neural network model and sort them;

[0056] According to the set pruning rate, the sorted parameters are proportionally set to zero to prune the neural network model.

[0057] Specifically, the trained neural network model is pruned according to the set pruning rate. The pruning method used in this embodiment can be replaced with different importance calculation methods, such as gradient importance, activation value importance, and can be replaced with different pruning granularities, such as filter pruning and channel pruning. This embodiment is applied in the training process included in its pruning process to alleviate the unfairness caused by pruning. Since the classification weight W needs to be used for calculation later, the last classification layer is not pruned by default.

[0058] Furthermore, the calculation of the extreme fairness constraint item includes:

[0059] The training image dataset (i.e., training set) is divided into several batches. In the training of each batch, the feature vector obtained after each sample is input into the feature extractor is obtained, and the geometric metric corresponding to each sample is calculated;

[0060] The average length and average angle of K sensitive attribute subgroup samples in the current batch are respectively counted, and the extreme fairness constraint item is calculated according to the average length and average angle.

[0061] Specifically, the geometric measurements corresponding to the sample include the length of the sample and the angle of the sample:

[0062] Sample length: Use the L1 norm of the feature vector X to calculate the length ‖X‖;

[0063] The angle of the sample: The angle α is calculated by the classification weight W corresponding to the feature vector X and its target label:

[0064] Where W T is the transpose of the classification weight W, and ||W|| is the L1 norm of the classification weight W.

[0065] In this embodiment, the angle is measured by the cosine similarity between the feature vector X and the classification weight W.

[0066] For samples of K sensitive attribute subgroups (such as different races, different genders) in the current batch, the average length and average angle of the sensitive subgroups are counted. For example, if there are n samples in the current batch, k The average length of the kth subgroup of samples is: The average angle is: In the formula, is the length of the i-th sample in the k-th subgroup, is the angle of the i-th sample in the k-th subgroup.

[0067] like Figure 2 , average length Represents the classification confidence of samples in the sensitive attribute subgroup. When the length is larger, the sample is farther away from the classification center, and thus it is easier to be correctly classified. The average angle Represents the intra-class compactness of samples in sensitive subgroup k. The smaller the value, the closer the sample feature is to the correct category weight.

[0068] Furthermore, since the length ||X|| and angle α corresponding to the sample are related to the loss calculated for each sample:

[0069]

[0070] In the formula, n is the total number of samples, |Y| is the total number of categories, is the jth incorrect category y of the kth sample jk Corresponding classification weight W yjk The L1 norm of is the kth sample feature vector X k and the jth incorrect category y jk The corresponding classification weight W yjk The angle between is the kth sample feature vector X k With the correct category y k The corresponding classification weight W k The angle between is the correct category y of the kth sample k Corresponding classification weight W k The L1 norm of .

[0071] The size of the loss is related to the accuracy of sample classification, so it can be used as a variable to control the differences between different sensitive subgroups to design fairness constraints;

[0072] The difference between the maximum and minimum geometric metrics among K sensitive attribute subgroups is measured, and the extreme fairness constraint is calculated to achieve the fairness of the model. The formula is as follows:

[0073]

[0074]

[0075] L fair =λ length L length +λ angle L angle ;

[0076] Where, L length is the range length constraint, is the maximum average length of the K sensitive attribute subgroups, is the minimum average length of the K sensitive attribute subgroups, L angle is the extreme angle constraint term, is the maximum average angle among the K sensitive attribute subgroups, is the minimum average angle among the K sensitive attribute subgroups, L fair is the extreme fairness constraint, λ length is the weight coefficient of the extreme length constraint, λ angle is the weight coefficient of the extreme angle constraint.

[0077] In this embodiment, λ length =0.01,λ angle =0.01.

[0078] Furthermore, the total loss function of the target neural network pruning model includes a classification loss term and a fairness loss term, wherein the classification loss term uses a cross entropy loss function term to calculate the classification loss of the sample, and the fairness loss term uses a geometric metric range that constrains different sensitive subgroups to calculate the fairness loss of the sample;

[0079] Wherein, the total loss function is:

[0080] L=L ce +L fair ;

[0081] In the formula, L is the total loss function, L ce is the classification loss term, L fair is the fair loss item;

[0082]

[0083] In the formula, f θ (x k ) is the predicted output corresponding to the k-th sample input of the neural network f with model parameters θ, y k is the correct category corresponding to the kth sample, x kis the kth sample, P(f θ (x k )=y k |x k ) is the probability that the kth input sample is predicted as the correct category by the neural network f with model parameters θ.

[0084] Furthermore, in the fine-tuning training phase after pruning, an improved total loss function is used for training, and the Adam algorithm is used to perform gradient updates on the remaining parameters of the model, so that the model can constrain the geometric metrics between different sensitive groups, thereby constraining the performance differences of each subgroup, and ultimately improving the fairness of the pruned model while ensuring accuracy.

[0085] This embodiment can be quickly integrated into the training phase of the existing pruning algorithm, thereby directly improving the fairness effect of the pruning algorithm that only considers computational efficiency and accuracy. As shown in Table 1, by applying it in the training phase of different pruning algorithms, the effect of ensuring fairness while improving accuracy is achieved.

[0086] This embodiment can achieve good results on different tasks. As shown in Table 2, it can maintain classification accuracy and improve fairness at a high pruning rate in different models and data sets.

[0087] The fairness metric in Tables 1 and 2 uses standard deviation, that is, the standard deviation of the accuracy of each sensitive attribute subgroup as a measure of the difference between statistical subgroups. The smaller the value, the more fair it is.

[0088] Table 1

[0089] Pruning method Classification accuracy Standard Deviation SNIP 79.881 15.893 SNIP+ours 78.110 15.045 Lottery 77.565 14.056 Lottery+ours 77.429 13.404 Grasp 77.020 15.484 Grasp+ours 77.202 13.056 FPGM (pruning 70%) 87.420 8.083 FPGM (pruning 70%) + ours 88.101 7.962 FPGM (pruning 90%) 82.732 9.672 FPGM (pruning 90%) + ours 81.880 7.881

[0090] Table 2

[0091]

[0092]

[0093] The optimization method of this embodiment is suitable for actual deployment on terminal devices, and can also be quickly integrated into existing pruning technologies to effectively address fairness risks that may arise when applying pruning technologies. Through this method, while ensuring the efficiency and lightweight of the model, the potential bias introduced by the pruning process can be reduced or eliminated, thereby improving the overall fairness and reliability of the model.

[0094] This embodiment also provides an image classification method, including:

[0095] Get the image to be classified;

[0096] The image to be classified is input into the target neural network pruning model constructed by the above method to obtain the classification result.

[0097] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A fairness optimization method for deep neural network pruning technology, characterized in that: include: Obtaining a training data set, wherein the training data set is image data including target labels and sensitive attributes; Train the initial neural network model using the training data set to obtain a neural network model; Pruning the neural network model; The geometrical metrics of samples of different sensitive attribute subgroups are obtained through the training data set and the pruned neural network model, and the geometrical metrics of samples of different sensitive attribute subgroups are averaged to calculate the extreme fairness constraint item, and the overall loss function is calculated using the extreme fairness constraint item and the classification loss item. The pruned neural network model is fine-tuned through the overall loss function to obtain the target neural network pruning model.

2. The fairness optimization method for deep neural network pruning technology according to claim 1 is characterized in that: The neural network model is used to extract sample feature vectors of the training data set, and the neural network model includes a feature extractor and a classifier.

3. The fairness optimization method for deep neural network pruning technology according to claim 1 or 2, characterized in that: Pruning the neural network model includes: Set the pruning rate; Calculate the L1 norm of all parameters of the neural network model and sort them; The sorted parameters are proportionally set to zero according to the pruning rate to prune the neural network model.

4. The fairness optimization method for deep neural network pruning technology according to claim 3 is characterized in that: During the process of pruning the neural network model, the last classification layer in the neural network model is not pruned.

5. The fairness optimization method for deep neural network pruning technology according to claim 1, characterized in that: Calculating the extreme fairness constraint item includes: The training image data set is divided into several batches. In the training of each batch, a feature vector is obtained after each sample is input into a feature extractor, and a geometric metric corresponding to each sample is calculated; The average length and average angle of K sensitive attribute subgroup samples in the current batch are respectively counted, and the extreme fairness constraint item is calculated according to the average length and average angle.

6. The fairness optimization method for deep neural network pruning technology according to claim 5, characterized in that: The geometrical metric corresponding to the sample includes the length of the sample and the angle of the sample. The length of the sample is calculated using the L1 norm of the feature vector, and the angle of the sample is calculated using the classification weight corresponding to the feature vector and the target label of the feature vector.

7. The fairness optimization method for deep neural network pruning technology according to claim 5, characterized in that: The extreme fairness constraint item is calculated by the average length and the average angle, including: The difference between the maximum geometric metric and the minimum geometric metric among the K sensitive attribute subgroups is measured, and the extreme fairness constraint item is calculated, specifically: L fair =λ length L length +λ angle L angle ; Where, L length is the range length constraint, is the maximum average length of the K sensitive attribute subgroups, is the minimum average length of the K sensitive attribute subgroups, L angle is the extreme angle constraint term, is the maximum average angle among the K sensitive attribute subgroups, is the minimum average angle among the K sensitive attribute subgroups, L fair is the extreme fairness constraint, λ length is the weight coefficient of the extreme length constraint, λ angle is the weight coefficient of the extreme angle constraint.

8. The fairness optimization method for deep neural network pruning technology according to claim 1, characterized in that: The total loss function of the target neural network pruning model includes a classification loss term and a fairness loss term, wherein the classification loss term uses a cross entropy loss function term to calculate the classification loss of the sample, and the fairness loss term uses a geometric metric range that constrains different sensitive subgroups to calculate the fairness loss of the sample; Wherein, the total loss function is: L=L ce +L fair ; In the formula, L is the total loss function, L ce is the classification loss term, L fair It is a fair loss item.

9. The fairness optimization method for deep neural network pruning technology according to claim 1, characterized in that: In the fine-tuning training stage, the improved total loss function is used for training, and the Adam algorithm is used to perform gradient updates on the remaining parameters of the model.

10. An image classification method, characterized in that: include: Get the image to be classified; The image to be classified is input into the target neural network pruning model constructed by the method according to any one of claims 1 to 9 to obtain a classification result.

Images