Method and device for enhancing robust adversarial fairness based on cross weight

Abstract

The invention discloses a cross weight-based robust adversarial fairness enhancement method and device, and relates to the technical field of computer vision. The method comprises the following steps: constructing a robust adversarial fairness enhancement architecture based on cross weight; inputting the image data set of the road scene as a clean sample into an adversarial sample generation module to generate an adversarial sample; inputting the clean samples and the adversarial samples into a classification model respectively to output corresponding category probabilities; respectively inputting the category probability of the clean sample and the category probability of the adversarial sample into an evaluation variable module to obtain an evaluation variable; calculating a category weight and a sample adaptive weight through a cross weight calculation module to obtain a cross weight; and inputting the category probability of the clean sample and the category probability of the adversarial sample into a dynamic label updating module to update an attack label and a training label, calculating a loss function based on a cross weight, and training the classification model to obtain a trained classification model. According to the invention, the accuracy of classification model identification can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an adversarial robust fairness enhancement method and apparatus based on cross-weights. Background Technology

[0002] In modern society, deep neural networks have been widely applied in numerous fields. In computer vision, current research indicates that visual deep neural networks are vulnerable to adversarial attacks, and various defensive measures have been proposed to enhance the adversarial robustness of the models. Among these, adversarial training is considered one of the most widely used and effective defense strategies. However, while current adversarial training methods can enhance the adversarial nature of the models, they also exacerbate the performance unfairness between classes, leading to significant performance gaps between different classes. The emergence of various current adversarial robustness fairness methods has mitigated this phenomenon and improved the overall performance of the system.

[0003] Current adversarial robust fairness methods exhibit limitations in optimizing adversarial robust fairness. First, they do not modify the adversarial example generation process. Current fairness methods employ adversarial attacks, perturbing samples during iterations to gravitate towards a specific class. This perturbation causes the feature spatial locations of perturbed samples to cluster near locations of more easily attacked classes. The resulting complex spatial distribution introduces additional challenges to the training process, leading to training difficulties for certain classes. Furthermore, current reweighting methods use class-level weights to balance sample features. This results in hard and easy samples within the same class receiving the same weights, ignoring the differences between individual samples. Therefore, the potential to further improve adversarial fairness through weighting is limited.

[0004] In deep learning-based visual intelligent recognition systems, the adversarial robustness of the model is directly related to the reliability and security of the system in real-world environments. However, existing adversarial training methods have the following shortcomings, which seriously affect the prediction accuracy and fairness of the model in practical applications; some categories are difficult to learn effectively in adversarial training, resulting in a significant decrease in the recognition accuracy of certain key categories during actual deployment. The existing technologies have the following technical problems: (1) The differences in class weights in existing weighted adversarial training are too large, causing the model to over-focus on easily classified samples and ignore difficult examples, thereby reducing the overall generalization performance of the system. In practical applications, this imbalance can lead to misjudgments by the model in key scenarios such as high-risk driving decisions. (2) Existing methods rely only on class-level weights and cannot distinguish the difficulty of different samples within the same category, resulting in insufficient robustness to high-difficulty samples; this may lead to serious consequences in real-world security monitoring and remote diagnosis scenarios.

[0005] In summary, existing adversarial training methods suffer from practical application problems in intelligent recognition systems, such as insufficient model robustness, low accuracy in key category recognition, and poor system reliability due to imbalanced class learning, unreasonable weight allocation, and coarse sample granularity. This provides a more reliable, fair, and robust model training solution for AI applications with high security requirements, such as autonomous driving, smart healthcare, and financial identity authentication. Summary of the Invention

[0006] To address the technical problems of low accuracy in key category identification caused by imbalanced class learning, unreasonable weight allocation, and coarse sample granularity in existing classification models, this invention provides an adversarial robust fairness enhancement method and apparatus based on cross-weights. The technical solution is as follows:

[0007] On the one hand, a cross-weight-based adversarial robust fairness enhancement method is provided, which is implemented by a cross-weight-based adversarial robust fairness enhancement device, and the method includes:

[0008] S1. Obtain an image dataset of a highway scene; wherein the image dataset consists of images of individual objects of different categories; wherein, a category label is assigned to each category of images to construct a category dataset; construct an adversarial robustness and fairness enhancement architecture based on cross-weights; the architecture includes: an adversarial example generation module, a ResNet classification model, an evaluation variable module, a cross-weight calculation module, and a dynamic label update module;

[0009] S2. Input the category dataset as clean samples into the adversarial sample generation module, and generate adversarial samples by randomly adding noise to the clean samples; input the clean samples into the ResNet classification model, and output the class probability of the clean samples; input the adversarial samples into the ResNet classification model, and output the class probability of the adversarial samples.

[0010] S3. Input the class probability of clean samples into the evaluation variable module for calculation to obtain the class evaluation variable; input the class probability of adversarial samples into the evaluation variable module for calculation to obtain the sample adaptive evaluation variable; calculate the class evaluation variable and the sample adaptive evaluation variable under adversarial sample conditions and non-adversarial sample conditions respectively to obtain the evaluation variable under adversarial conditions and the evaluation variable under non-adversarial sample conditions.

[0011] S4. The evaluation variables under adversarial conditions and non-adversarial sample conditions are mixed and divided into two groups of evaluation variables through normalization. The two groups of evaluation variables are then input into the cross-weight calculation module and processed through the constructed aggregation function to obtain the class weights and sample adaptive weights. Based on the class weights and sample adaptive weights, the cross weights are obtained.

[0012] S5. Input the class probabilities of clean samples and adversarial samples into the dynamic label update module to update the predefined attack labels and training labels. Based on the cross weights, update and train the ResNet classification model by calculating the loss function to obtain the trained ResNet classification model.

[0013] S6. Acquire the front view image collected during autonomous driving; input the front view image into the trained ResNet classification model to identify the object category and object location in the image.

[0014] On the other hand, a cross-weight-based adversarial robust fairness enhancement device is provided, which is applied to a cross-weight-based adversarial robust fairness enhancement method. The device includes:

[0015] A construction unit is used to acquire an image dataset of a highway scene; wherein the image dataset consists of images of individual objects of different categories; wherein, a category label is labeled for each category of images to construct a category dataset; and an adversarial robustness and fairness enhancement architecture based on cross weights is constructed; the architecture includes: an adversarial example generation module, a ResNet classification model, an evaluation variable module, a cross weight calculation module, and a dynamic label update module;

[0016] The generation unit is used to take the categorical dataset as clean samples, randomly add adversarial noise to the clean samples to generate adversarial samples; input the clean samples into the ResNet classification model and output the class probability of the clean samples; input the adversarial samples into the ResNet classification model and output the class probability of the adversarial samples.

[0017] The first acquisition unit is used to input the class probability of clean samples into the evaluation variable module for calculation to obtain the class evaluation variable; input the class probability of adversarial samples into the evaluation variable module for calculation to obtain the sample adaptive evaluation variable; and perform calculations on the class evaluation variable and the sample adaptive evaluation variable under adversarial sample conditions and non-adversarial sample conditions respectively to obtain the evaluation variable under adversarial conditions and the evaluation variable under non-adversarial sample conditions.

[0018] The second acquisition unit is used to mix the evaluation variables under adversarial conditions and the evaluation variables under non-adversarial sample conditions and divide them into two groups of evaluation variables through normalization; input the two groups of evaluation variables into the cross-weight calculation module, process them through the constructed aggregation function, and obtain the class weight and sample adaptive weight; and obtain the cross weight based on the class weight and sample adaptive weight.

[0019] The training unit is used to input the class probabilities of clean samples and adversarial samples into the dynamic label update module to update the predefined attack labels and training labels. Based on the cross weights, the ResNet classification model is updated and trained by calculating the loss function to obtain the trained ResNet classification model.

[0020] The recognition unit is used to acquire a front-view image collected during autonomous driving; and input the front-view image into a trained ResNet classification model to identify the object category and object location in the image.

[0021] On the other hand, a cross-weight-based adversarial robust fairness enhancement device is provided, the cross-weight-based adversarial robust fairness enhancement device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the methods described above for cross-weight-based adversarial robust fairness enhancement methods is implemented.

[0022] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described cross-weight-based adversarial robust fairness enhancement methods.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0024] This invention generates adaptive labels for attack and training by starting from the current self-guided label and adding iteration and source labels, supporting cross-weighted and class confusion attacks.

[0025] This invention studies adversarial fairness from the perspective of class-specific directionality in adversarial attacks, and proposes a class confusion attack to alleviate the complexity of model feature space distribution caused by adversarial attacks.

[0026] This invention proposes a weighted algorithm from the perspectives of samples and categories, establishing cross-weighted adversarial training to enhance category balance during training and improve the accuracy of the classification model. This invention addresses the technical problem of low accuracy in key category recognition in autonomous driving intelligent recognition systems caused by imbalanced category learning, unreasonable weight allocation, and coarse sample granularity. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of an adversarial robustness fairness enhancement method based on cross-weights provided in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of an overall frame structure provided in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of the calculation process for an evaluation variable update module provided in an embodiment of the present invention;

[0031] Figure 4 This is a flowchart of a cross-weight calculation module provided in an embodiment of the present invention;

[0032] Figure 5 This is a flowchart of a dynamic tag update module calculation provided in an embodiment of the present invention;

[0033] Figure 6 This is a block diagram of an adversarial robust fairness enhancement device based on cross-weights provided in an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the structure of an adversarial robust fairness enhancement device based on cross-weights provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0036] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0037] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0038] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0039] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0040] This invention provides a method for enhancing adversarial robustness fairness based on cross-weights. This method can be implemented by a device for enhancing adversarial robustness fairness based on cross-weights, which can be a terminal or a server. Figure 1 The flowchart shown is for an adversarial robust fairness enhancement method based on cross-weights. The processing flow of this method may include the following steps:

[0041] S1. Obtain an image dataset of a highway scene; the image dataset consists of images of individual objects of different categories; each category of image is labeled with a category label to construct a category dataset; construct an adversarial robustness and fairness enhancement architecture based on cross-weights; the architecture includes: an adversarial example generation module, a ResNet classification model, an evaluation variable module, a cross-weight calculation module, and a dynamic label update module.

[0042] Among them, such as Figure 2 The diagram shown is a schematic diagram of an overall frame structure provided by an embodiment of the present invention.

[0043] Optionally, the evaluation variable module is used to evaluate the class performance and sample performance of the number of classes in the dataset and the number of samples in the training batch, respectively.

[0044] Among them, the adversarial example generation module is used to generate adversarial examples;

[0045] The cross-weight calculation module is used to calculate the class weights and sample adaptive weights, and to calculate the cross weights based on the class weights and sample adaptive weights.

[0046] The dynamic label update module is used to update predefined attack labels and training labels.

[0047] In one feasible implementation, the adversarial robust fairness enhancement architecture based on cross-weights constructed in this embodiment of the invention consists of two parts: class confusion attack and cross-weight adversarial training. By combining the two parts, adversarial robust fairness training is performed, thereby improving the adversarial robust fairness of the classification model. The overall architecture operates through continuous adversarial attacks and adversarial training, ultimately obtaining a robust classification model with adversarial robust fairness.

[0048] S2. Take the category dataset as clean samples, randomly add adversarial noise to the clean samples to generate adversarial samples; input the clean samples into the ResNet classification model and output the class probability of the clean samples; input the adversarial samples into the ResNet classification model and output the class probability of the adversarial samples.

[0049] In this dataset, each image corresponds to a clean sample's class probability, an adversarial sample's class probability, and a true label.

[0050] In one feasible implementation, the category dataset is used as clean samples and input into the adversarial sample generation module. Noise is randomly added to the clean samples and input into the ResNet classification model. The loss function value is obtained by calculating the cross-entropy loss function. Adversarial noise is calculated based on the loss function value and obtained by backpropagation of gradients in the ResNet classification model. The clean samples are added with adversarial noise and continuously updated with adversarial noise to obtain the final adversarial noise. Based on the final adversarial noise, it is superimposed on the original image according to an adaptive perturbation interval to obtain the adversarial sample.

[0051] The process of updating against noise is represented by the following formula (1):

[0052] (1)

[0053] in, Indicates resistance to noise; This represents the output of the ResNet classification model for the class probability model of adversarial examples; Indicates an attack tag; Represents the cross-loss entropy function; This indicates the adaptive perturbation interval.

[0054] The process of generating adversarial examples is represented by the following formula (2):

[0055] (2)

[0056] in, Indicates adversarial examples; Indicates resistance to noise; This indicates a clean sample.

[0057] In one feasible implementation, during the adversarial training phase, this embodiment of the invention employs cross-weighted adversarial training to optimize the training of adversarial examples, thereby promoting more balanced learning across classes.

[0058] Among them, such as Figure 3 The diagram shown is a flowchart of the calculation process for an evaluation variable update module provided in an embodiment of the present invention.

[0059] S3. Input the class probability of clean samples into the evaluation variable module for calculation to obtain the class evaluation variable; input the class probability of adversarial samples into the evaluation variable module for calculation to obtain the sample adaptive evaluation variable; calculate the class evaluation variable and the sample adaptive evaluation variable under adversarial sample conditions and non-adversarial sample conditions respectively to obtain the evaluation variable under adversarial conditions and the evaluation variable under non-adversarial sample conditions.

[0060] Optionally, S3 inputs the class probability of clean samples into the evaluation variable module for calculation to obtain class evaluation variables; it also inputs the class probability of adversarial examples into the evaluation variable module for calculation to obtain sample adaptive evaluation variables, including:

[0061] S31. Input the class probabilities of clean samples into the evaluation variable module, calculate the cross-entropy loss using the cross-entropy loss function, and calculate the class margin loss using the class margin loss function.

[0062] S32. Sum the cross-entropy loss and the class margin loss, and update them using the exponential moving average method to obtain the update amount of the class evaluation variable; wherein, the calculation formula for the update amount of the class evaluation variable is expressed by the following formula (3):

[0063] (3)

[0064] in, This represents the output of the ResNet classification model for the class probability of x. This represents the category evaluation calculation function; This represents the actual label of x; Represents the cross-entropy function; This represents the category margin loss function;

[0065] S33. Based on the category update variable, obtain the category evaluation variable; use the category interval loss function to calculate the category interval loss and directly obtain the sample adaptive evaluation variable.

[0066] Optionally, the process of calculating the category evaluation variable and the sample adaptive evaluation variable under adversarial and non-adversarial conditions respectively for S3, and obtaining the evaluation variables under adversarial and non-adversarial conditions, is represented by the following formulas (4)-(5):

[0067] (4)

[0068] (5)

[0069] in, This represents the updated category evaluation variable; This indicates that the parameter is being updated; the default value is 0.9. This represents the category evaluation variable before the update; This indicates an operation to calculate the average value based on the category; This represents the category evaluation calculation function; Indicates the sample adaptive evaluation variable; This represents the category margin loss function; This represents the output of the ResNet classification model for the class probability of x. This indicates the actual label.

[0070] Among them, such as Figure 4 The diagram shown is a flowchart of a cross-weight calculation module provided in an embodiment of the present invention.

[0071] S4. Divide the evaluation variables under adversarial conditions and non-adversarial sample conditions into two groups of evaluation variables; input the two groups of evaluation variables into the cross-weight calculation module, process them through the constructed aggregation function, and obtain the class weights and sample adaptive weights; obtain the cross weights based on the class weights and sample adaptive weights.

[0072] The calculation process of the cross weight is expressed by the following formula (6):

[0073] (6)

[0074] in, Indicates cross weights; Indicates the sample adaptive weights; Indicates category weight; This represents the matrix transpose symbol.

[0075] In one feasible implementation, samples of different categories tend to have different perturbation intervals during adversarial training. To achieve this interval requirement, this embodiment of the invention designs an adaptive perturbation interval construction strategy. By evaluating the adversarial performance provided for each category through category weights, a category-specific interval is derived, which is an adaptive perturbation interval that better meets the requirements of the current training batch samples. The calculation process of the adaptive perturbation interval is expressed by the following formula (7):

[0076] (7)

[0077] in, Indicates the adaptive perturbation interval; This indicates the original perturbation interval, which defaults to 8 / 255. Indicates category weight; This represents the logarithmic value calculated based on the sample size, with a base of 10.

[0078] Optionally, S4 mixes the evaluation variables under adversarial conditions and the evaluation variables under non-adversarial sample conditions, and divides them into two groups of evaluation variables through normalization. The two groups of evaluation variables are then input into the cross-weight calculation module and processed using a constructed aggregation function, including:

[0079] In order to convert the evaluation variables into weight factors compatible with the loss function, they are normalized to a mean of 1. Therefore, the embodiments of the present invention use the following formula (8) to perform the normalization operation, and the transformation result is represented by ~.

[0080] (8)

[0081] in, C represents the normalized evaluation variable; C represents the category. This indicates that the evaluation variables will undergo a softmax operation; This represents the evaluation variables, including categorical variables and sample adaptive variables.

[0082] The evaluation variables under adversarial conditions and under non-adversarial sample conditions are mixed and divided into two groups of evaluation variables through normalization. The first group of evaluation variables is processed by the constructed aggregation function to obtain the class weights. The second group of evaluation variables is processed by the constructed aggregation function to obtain the sample adaptive weights. The process of the aggregation function is represented by the following formula (9):

[0083] (9)

[0084] in, This represents an aggregate function that combines the opposing and clean parts of a variable; Represents normalized adversarial example evaluation variables; Represents normalized clean sample evaluation variables; This indicates element-wise multiplication.

[0085] Among them, such as Figure 5 The diagram shown is a flowchart of a dynamic tag update module calculation provided in an embodiment of the present invention.

[0086] S5. Input the class probabilities of clean samples and adversarial samples into the dynamic label update module to update the predefined attack labels and training labels. Based on the cross weights, update and train the ResNet classification model by calculating the loss function to obtain the trained ResNet classification model.

[0087] In current fairness methods, the adversarial example generation process relies on labels and a cross-entropy loss function. However, labels only provide attack directions for hard-labeled categories, failing to provide directions for other categories. This leads to attacks biased towards easier categories, resulting in unfairness in adversarial attacks. Therefore, the embodiments of this invention...

[0088] In this embodiment of the invention, a category confusion attack is designed to convert the one-hot encoded source label. An iterative process is introduced, and attack iteration soft tags are defined during the adversarial attack process. To make the attack more equitable.

[0089] Optionally, S5 inputs the class probabilities of clean samples and adversarial samples into the dynamic label update module to update predefined attack labels and training labels, including:

[0090] S51. Based on the class probabilities of clean samples and adversarial samples, the predefined attack labels are updated using one-hot encoded source labels and defined attack evaluation parameters to obtain the updated attack labels; where the one-hot encoded source labels are the labels corresponding to the clean samples.

[0091] The process of updating the predefined attack labels is represented by the following formulas (10)-(11):

[0092] (10)

[0093] (11)

[0094] in, This indicates an attack that iterates over soft tags; This represents the label smoothing parameter, with a default value of 0.5. Indicates the true label; This represents the model class probability output for adversarial examples. Indicates adversarial examples; This indicates the updated attack tag; This indicates that the parameter is being updated; the default value is 0.9. This indicates the attack tag before the update; Indicates the number of training rounds.

[0095] Among these methods, refining the original self-guided labels can effectively improve the overall adversarial robustness of the model. However, by iteratively updating the self-guided labels, further updates become ineffective after a certain number of iterations, leading to model training stagnation or even degradation.

[0096] In one feasible implementation, the one-hot encoded source label y is introduced into the iterative process to replace the model probability output of clean samples. This prevents self-guided labels from completely overlapping with the model output and makes them closer to the true labels. Therefore, self-guided labels can consistently and effectively guide the training of the ResNet classification model. Definition For training labels, from Begin by defining training iteration attack soft labels during the adversarial attack process. attack soft tags based on training iterations The training labels are updated using an exponential moving average process. .

[0097] S52. Based on the attack iteration soft labels, update the predefined training labels through an exponential moving average process and the set update parameters to obtain the updated training labels.

[0098] In one feasible implementation, the process of updating the predefined training labels is represented by the following formulas (12)-(13):

[0099] (12)

[0100] (13)

[0101] in, This indicates an attack on soft labels during training iterations; This represents the label smoothing parameter, which defaults to 0.2. y represents the model class probability output of the adversarial example; y represents the true label. This indicates the updated training labels; This represents the training labels before the update; r represents the update parameter, with a default value of 0.9.

[0102] Alternatively, the loss function of S5 is expressed by the following formula (14):

[0103] (14)

[0104] in, The loss function is represented by N; the number of samples is represented by C; and the number of classes is represented by C. This represents the probability of an adversarial example being input into the ResNet classification model's output. Indicates adversarial examples; Indicates training labels; This indicates the cross weight.

[0105] S6. Acquire the front view image collected during autonomous driving; input the front view image into the trained ResNet classification model to identify the object category and object location in the image.

[0106] Among them, non-adversarial robust target classification models, such as VIT or ResNet, can directly replace the original model position after the above training process and directly participate in the original function of autonomous driving system or image classification and recognition device without modifying the upstream and downstream input and output structure. They can also obtain resistance to adversarial attacks and class fairness under adversarial attack conditions, thus improving overall robustness.

[0107] This invention generates adaptive labels for attack and training by starting from the current self-guided label and adding iteration and source labels, supporting cross-weighted and class confusion attacks.

[0108] This invention studies adversarial fairness from the perspective of class-specific directionality in adversarial attacks, and proposes a class confusion attack to alleviate the complexity of model feature space distribution caused by adversarial attacks.

[0109] This invention proposes a weighted algorithm from the perspectives of samples and categories, establishing cross-weighted adversarial training to enhance category balance during training and improve the accuracy of the classification model. This invention addresses the technical problem of low accuracy in key category recognition in autonomous driving intelligent recognition systems caused by imbalanced category learning, unreasonable weight allocation, and coarse sample granularity.

[0110] Figure 6 This is a block diagram of an adversarial robust fairness enhancement device based on cross-weights, provided in an embodiment of the present invention. This device is used in an adversarial robust fairness enhancement method based on cross-weights. (Refer to...) Figure 6 The device includes a construction unit 610, a generation unit 620, a first acquisition unit 630, a second acquisition unit 640, a training unit 650, and a recognition unit 660. Wherein:

[0111] Construction unit 610 is used to acquire an image dataset of a highway scene; wherein the image dataset consists of images of individual objects of different categories; wherein, a category label is labeled for each category of images to construct a category dataset; an adversarial robustness and fairness enhancement architecture based on cross weights is constructed; the architecture includes: an adversarial example generation module, a ResNet classification model, an evaluation variable module, a cross weight calculation module, and a dynamic label update module;

[0112] The generation unit 620 is used to input the category dataset as clean samples into the adversarial sample generation module, and generate adversarial samples by randomly adding noise to the clean samples; input the clean samples into the ResNet classification model and output the class probability of the clean samples; input the adversarial samples into the ResNet classification model and output the class probability of the adversarial samples.

[0113] The first acquisition unit 630 is used to input the class probability of clean samples into the evaluation variable module for calculation to obtain the class evaluation variable; input the class probability of adversarial samples into the evaluation variable module for calculation to obtain the sample adaptive evaluation variable; and perform calculations on the class evaluation variable and the sample adaptive evaluation variable under adversarial sample conditions and non-adversarial sample conditions respectively to obtain the evaluation variable under adversarial conditions and the evaluation variable under non-adversarial sample conditions.

[0114] The second acquisition unit 640 is used to mix and divide the evaluation variables under adversarial conditions and the evaluation variables under non-adversarial sample conditions into two groups of evaluation variables; input the two groups of evaluation variables into the cross-weight calculation module respectively, process them through the constructed aggregation function, and obtain the class weight and sample adaptive weight; and obtain the cross weight based on the class weight and sample adaptive weight.

[0115] Training unit 650 is used to input the class probabilities of clean samples and adversarial samples into the dynamic label update module to update the attack label and training label. Based on the cross weight, the ResNet classification model is updated and trained by calculating the loss function to obtain the trained ResNet classification model.

[0116] The recognition unit 660 is used to acquire a front-view image collected during autonomous driving; and input the front-view image into a trained ResNet classification model to identify the object category and object location in the image.

[0117] Optionally, the evaluation variable module is used to perform category performance evaluation and sample performance evaluation on the number of categories in the dataset and the number of samples in the training batch, respectively.

[0118] The adversarial sample generation module is used to generate adversarial samples.

[0119] The cross-weight calculation module is used to calculate the class weight and the sample adaptive weight, and to calculate the cross weight based on the class weight and the sample adaptive weight.

[0120] The dynamic label update module is used to update predefined attack labels and training labels.

[0121] Optionally, the step of inputting the class probability of clean samples into the evaluation variable module for calculation to obtain class evaluation variables; and inputting the class probability of adversarial examples into the evaluation variable module for calculation to obtain sample adaptive evaluation variables, includes:

[0122] Input the class probabilities of clean samples into the evaluation variable module, calculate the cross-entropy loss using the cross-entropy loss function, and calculate the class margin loss using the class margin loss function.

[0123] The cross-entropy loss and class margin loss are summed and updated using the exponential moving average method to obtain the update amount of the class evaluation variable; the formula for calculating the update amount of the class evaluation variable is expressed by the following formula (1):

[0124] (1)

[0125] in, This represents the ResNet classification output with respect to the class probability of input x; This represents the category evaluation calculation function; This represents the actual label of x; Represents the cross-entropy function; This represents the category margin loss function;

[0126] Based on the category update variable, the category evaluation variable is obtained; the category interval loss function is used to calculate the category interval loss, and the sample adaptive evaluation variable is directly obtained.

[0127] Optionally, the process of calculating the category evaluation variable and the sample adaptive evaluation variable under adversarial sample conditions and non-adversarial sample conditions respectively, and obtaining the evaluation variables under adversarial and non-adversarial sample conditions, is represented by the following formulas (2)-(3):

[0128] (2)

[0129] (3)

[0130] in, This represents the updated category evaluation variable; This indicates that the parameter is being updated; the default value is 0.9. This represents the category evaluation variable before the update; This indicates an operation to calculate the average value based on the category; This represents the category evaluation calculation function; y represents the true label of x; Indicates the sample adaptive evaluation variable; This represents the category margin loss function; This represents the output of the ResNet classification model for the class probability of x. This indicates the actual label.

[0131] Optionally, the evaluation variables under adversarial conditions and non-adversarial sample conditions are mixed and normalized into two groups of evaluation variables; the two groups of evaluation variables are then input into the cross-weight calculation module and processed using a constructed aggregation function, including:

[0132] The evaluation variables under adversarial conditions and under non-adversarial sample conditions are mixed and divided into two groups of evaluation variables through normalization. The first group of evaluation variables is processed by the constructed aggregation function to obtain the class weights. The second group of evaluation variables is processed by the constructed aggregation function to obtain the sample adaptive weights. The process of the aggregation function is represented by the following formula (4):

[0133] (4)

[0134] in, Represents aggregate functions; This represents the normalized adversarial example evaluation variable; Represents the evaluation variable for the clean sample after normalization; This indicates element-wise multiplication.

[0135] Optionally, the step of inputting the class probabilities of clean samples and adversarial samples into the dynamic label update module to update the predefined attack labels and training labels includes:

[0136] Based on the class probabilities of adversarial examples, predefined attack labels are updated using one-hot encoded source labels and defined attack evaluation parameters to obtain updated attack iteration soft labels; where the one-hot encoded source labels are the labels corresponding to clean samples.

[0137] Based on the updated attack labels, the predefined training labels are updated using the exponential moving average method and the set update parameters to obtain the updated training labels.

[0138] Optionally, the loss function is expressed by the following formula (5):

[0139] (5)

[0140] in, The loss function is represented by N; the number of samples updated in one training iteration is represented by C; and the number of classes in the dataset is represented by C. This represents the probability of an adversarial example being input into the ResNet classification model's output. Indicates adversarial examples; Indicates training labels; This indicates the cross weight.

[0141] This invention generates adaptive labels for attack and training by starting from the current self-guided label and adding iteration and source labels, supporting cross-weighted and class confusion attacks.

[0142] This invention studies adversarial fairness from the perspective of class-specific directionality in adversarial attacks, and proposes a class confusion attack to alleviate the complexity of model feature space distribution caused by adversarial attacks.

[0143] This invention proposes a weighted algorithm from the perspectives of samples and categories, establishing cross-weighted adversarial training to enhance category balance during training and improve the accuracy of the classification model. This invention addresses the technical problem of low accuracy in key category recognition in autonomous driving intelligent recognition systems caused by imbalanced category learning, unreasonable weight allocation, and coarse sample granularity.

[0144] Figure 7 This is a schematic diagram of the structure of an adversarial robust fairness enhancement device based on cross-weights provided in an embodiment of the present invention, as shown below. Figure 7 As shown, the adversarial robustness fairness enhancement device based on cross-weights can include the above-mentioned Figure 6 The illustrated cross-weight-based adversarial robust fairness enhancement device 710 may optionally include a first processor 2001.

[0145] Optionally, the adversarial robust fairness enhancement device 710 based on cross-weights may also include a memory 2002 and a transceiver 2003.

[0146] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0147] The following is combined Figure 7 The components of the cross-weight-based adversarial robust fairness enhancement device 710 are described in detail below:

[0148] The first processor 2001 is the control center of the cross-weighted adversarial robust fairness enhancement device 710. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0149] Optionally, the first processor 2001 can perform various functions of the cross-weighted adversarial robust fairness enhancement device 710 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0150] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 7 CPU0 and CPU1 are shown in the diagram.

[0151] In a specific implementation, as one example, the adversarial robust fairness enhancement device 710 based on cross-weights may also include multiple processors, for example... Figure 7 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0152] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0153] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the cross-weighted adversarial robust fairness enhancement device 710. Figure 7 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0154] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0155] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 7 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0156] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be interfaced through the cross-weight-based adversarial robust fairness enhancement device 710. Figure 7 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0157] It should be noted that, Figure 7 The structure of the cross-weighted adversarial robust fairness enhancement device 710 shown in the figure does not constitute a limitation on the router. Actual cross-weighted adversarial robust fairness enhancement devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0158] Furthermore, the technical effect of the adversarial robust fairness enhancement device 710 based on cross-weights can be referred to the technical effect of the adversarial robust fairness enhancement method based on cross-weights described in the above method embodiments, and will not be repeated here.

[0159] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0160] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0161] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0162] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0163] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0164] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0165] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0167] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0170] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A cross-weight based adversarial robust fairness enhancement method, characterized in that, The method comprises: S1, acquiring an image data set of a highway scene; wherein the image data set is composed of single object images of different categories; wherein a category label is marked for each category of image, a category data set is constructed, and a cross-weight-based adversarial robust fairness enhancement architecture is constructed; the architecture comprises: an adversarial sample generation module, a ResNet classification model, an evaluation variable module, a cross-weight calculation module, and a dynamic label updating module; S2, inputting the category data set as a clean sample into the adversarial sample generation module, generating an adversarial sample by randomly adding noise to the clean sample; inputting the clean sample into the ResNet classification model to output the category probability of the clean sample; inputting the adversarial sample into the ResNet classification model to output the category probability of the adversarial sample; S3, inputting the category probability of the clean sample into the evaluation variable module for calculation to obtain a category evaluation variable; inputting the category probability of the adversarial sample into the evaluation variable module for calculation to obtain a sample adaptive evaluation variable; calculating the category evaluation variable and the sample adaptive evaluation variable under the conditions of having and not having the adversarial sample respectively to obtain the evaluation variable under the adversarial condition and the evaluation variable under the condition of not having the adversarial sample; S4, mixing the evaluation variable under the adversarial condition and the evaluation variable under the condition of not having the adversarial sample through normalization processing to divide them into two groups of evaluation variables; inputting the two groups of evaluation variables into the cross-weight calculation module respectively, processing them through the constructed aggregation function to obtain a category weight and a sample adaptive weight; and obtaining a cross-weight based on the category weight and the sample adaptive weight; S5, inputting the category probability of the clean sample and the category probability of the adversarial sample into the dynamic label updating module to update a predefined attack label and a training label, updating and training the ResNet classification model through calculation of a loss function based on the cross-weight to obtain a trained ResNet classification model; S6, acquiring a front perspective image collected in autonomous driving; inputting the front perspective image into the trained ResNet classification model to identify the object category and object position in the image.

2. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, The evaluation variable module is used for performing category performance evaluation and sample performance evaluation on the number of categories of the data set and the number of samples of the training batch respectively; The adversarial sample generation module is used for generating an adversarial sample; The cross-weight calculation module is used for calculating a category weight and a sample adaptive weight, and calculating a cross-weight based on the category weight and the sample adaptive weight; The dynamic label updating module is used for updating a predefined attack label and a training label.

3. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, In S3, the category probability of the clean sample is inputted into the evaluation variable module for calculation to obtain a category evaluation variable; In S3, the category probability of the adversarial sample is inputted into the evaluation variable module for calculation to obtain a sample adaptive evaluation variable, which comprises: S31, inputting the category probability of the clean sample into the evaluation variable module, calculating a cross-entropy loss through a cross-entropy loss function, and calculating a category interval loss through a category interval loss function; S32, sum the cross-entropy loss and the class gap loss, update by the exponential moving average method to obtain the update amount of the class evaluation variable; wherein the calculation formula of the update amount of the class evaluation variable is represented by the following formula (1): （1） wherein, represents a class probability output of the ResNet classification model for x; represents a class evaluation computation function; represents a true label of x; represents a cross-entropy function; represents an inter-class margin loss function; S33, based on the class update variable, obtain the class evaluation variable; use the class gap loss function to calculate the class gap loss to directly obtain the sample adaptive evaluation variable.

4. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, The process of S3 for calculating the evaluation variable under the condition of the adversarial sample and the condition of no adversarial sample respectively is represented by the following formulas (2)-(3): （2） （3） wherein, denotes the updated class evaluation variable; denotes the update parameter, default is 0.9; denotes the pre-update class evaluation variable; denotes the mean operation according to the class; denotes the class evaluation calculation function; y denotes the true label of x; denotes the sample adaptive evaluation variable; denotes the class interval loss function; denotes the class probability model output of the ResNet classification model for x; denotes the true label.

5. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, The process of S4 for mixing the evaluation variable under the condition of the adversarial sample and the evaluation variable under the condition of no adversarial sample through normalization processing to divide them into two groups of evaluation variables; input the two groups of evaluation variables into the cross weight calculation module for processing by the constructed aggregation function, including: Mixing the evaluation variable under the condition of the adversarial sample and the evaluation variable under the condition of no adversarial sample through normalization processing to divide them into two groups of evaluation variables; input the first group of evaluation variables into the constructed aggregation function for processing to obtain the class weight; input the second group of evaluation variables into the constructed aggregation function for processing to obtain the sample adaptive weight; wherein the process of the aggregation function processing is represented by the following formula (4): （4） wherein, denotes a pooling function; denotes a normalized adversarial sample evaluation variable; denotes a normalized clean sample evaluation variable; denotes an element-wise multiplication.

6. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, The S5 for inputting the class probability of the clean sample and the class probability of the adversarial sample into the dynamic label update module to update the predefined attack label and the training label, including: S51, based on the class probability of the clean sample and the class probability of the adversarial sample, update the predefined attack label by the one-hot encoded original label and the defined attack evaluation parameter to obtain the updated attack label; wherein the one-hot encoded original label is the label corresponding to the clean sample; S52, according to the updated attack label, update the predefined training label by the exponential moving average method and the set update parameter to obtain the updated training label.

7. The cross-weight based adversarial robust fairness enhancement method according to claim 1, wherein, The loss function of S5 is represented by the following formula (5): （5） wherein, represents a loss function; N represents the number of samples of one update of training; C represents the category of the data set; represents the probability of the output of the ResNet classification model input by the adversarial sample; represents the adversarial sample; represents the training label; represents the cross weight.

8. A cross-weight-based adversarial robust fairness enhancement device for implementing the cross-weight-based adversarial robust fairness enhancement method according to any one of claims 1-7, characterized in that, The device comprises: The construction unit is configured to obtain an image dataset of a highway scene; wherein the image dataset is composed of single object images of different categories; wherein a category label is marked for each category of image to construct a category dataset; and an adversarial robust fairness enhancement architecture based on cross weight is constructed; the architecture comprises: an adversarial sample generation module, a ResNet classification model, an evaluation variable module, a cross weight calculation module, and a dynamic label update module; The generation unit is configured to input the category dataset as a clean sample into the adversarial sample generation module to generate an adversarial sample by randomly adding noise to the clean sample; input the clean sample into the ResNet classification model to output the class probability of the clean sample; and input the adversarial sample into the ResNet classification model to output the class probability of the adversarial sample; The first obtaining unit is configured to input the class probability of the clean sample into an evaluation variable module to obtain a class evaluation variable, and input the class probability of the adversarial sample into the evaluation variable module to obtain a sample adaptive evaluation variable; and the class evaluation variable and the sample adaptive evaluation variable are respectively calculated under an adversarial sample condition and a non-adversarial sample condition to obtain an evaluation variable under the adversarial condition and an evaluation variable under the non-adversarial sample condition; The second obtaining unit is configured to mix the evaluation variable under the adversarial condition and the evaluation variable under the non-adversarial sample condition through normalization processing to divide them into two groups of evaluation variables; input the two groups of evaluation variables into a cross weight calculation module to obtain a class weight and a sample adaptive weight by processing through an aggregation function constructed in the cross weight calculation module; and obtain a cross weight based on the class weight and the sample adaptive weight. The training unit is configured to input the class probability of the clean sample and the class probability of the adversarial sample into a dynamic label updating module to update a predefined attack label and a training label, update and train the ResNet classification model by calculating a loss function based on the cross weight, and obtain a trained ResNet classification model. The identification unit is configured to obtain a front perspective image collected in autonomous driving, and input the front perspective image into the trained ResNet classification model to identify an object class and an object position in the image.

9. A cross-weight based adversarial robust fairness enhancement device, characterized in that, The adversarial robust fairness enhancement device based on the cross weight comprises: a processor; a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program code, which can be called and executed by the processor to implement the method of any one of claims 1 to 7.

Images