Suspicious model backdoor category location method based on model assimilation

Abstract

The present invention relates to the field of machine learning, and specifically to a method for locating suspicious model backdoor categories based on model assimilation. The method uses an image dataset to calculate the assimilation degree of the model to determine whether the model has been attacked by a backdoor. When the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category, thereby locating the suspicious category. Data is cleaned based on the suspicious category to obtain a clean dataset. A balanced dataset is generated based on the clean dataset. The model is retrained based on the balanced dataset. The present invention can locate suspicious poisoning categories in a dataset by measuring the differences in attention patterns of different categories in the dataset. This method can effectively detect backdoor models and poisoning types in a variety of backdoor attack scenarios without relying on known backdoor information, thereby improving the accuracy and robustness of backdoor attack detection.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to a method for locating suspicious model backdoor categories based on model assimilation. Background Art

[0002] With the widespread application of deep neural networks (DNNs) in computer vision, model security issues are gaining increasing attention. Backdoor attacks have become a major threat to models. Backdoor attacks use specific triggers to cause the model to misclassify when the trigger occurs during the prediction phase. Due to the diversity of backdoor triggers, model owners lack access to relevant information about backdoor attacks, making them highly concealed.

[0003] Currently, backdoor attack defense primarily focuses on methods such as model purification and model detection. However, these methods suffer from high computational costs, strong dependence on attack methods, and high false positive rates. Furthermore, while other backdoor defense methods such as model fine-tuning and feature separation improve model robustness to a certain extent, their high computational complexity leads to long training times and also require a certain number of clean and poisoned samples for experimental use, which also presents certain limitations. Therefore, effectively detecting backdoor models and poisoning types in various backdoor attack scenarios without relying on known backdoor information is crucial for improving the accuracy and robustness of backdoor attack detection and reducing its computational complexity and training time. Summary of the Invention

[0004] To address the problems of high computational cost, strong dependence on attack methods, and high false detection rate in existing backdoor attack detection methods, this paper aims to utilize the unique performance of assimilation in backdoor models and locate suspicious poisoning categories by measuring the differences in attention patterns of different categories in the dataset. The present invention provides a method for locating suspicious model backdoor categories based on model assimilation, which includes the following steps:

[0005] Step S1: Input an image dataset containing several categories, use the image dataset to calculate the assimilation degree of the model, and determine whether the model is attacked by a backdoor;

[0006] Step S2: When the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category, thereby locating the suspicious category;

[0007] Step S3: Perform data cleaning based on the suspicious categories to obtain a clean data set;

[0008] Step S4: Generate a balanced dataset based on the clean dataset; and retrain the model based on the balanced dataset.

[0009] Preferably, in step S1, the assimilation degree of the model is calculated using the image dataset to determine whether the model is subject to a backdoor attack, specifically:

[0010] Obtain several intermediate layers from the model to form a set of feature extraction layers;

[0011] For each category of samples in the image dataset, generating a feature map corresponding to the sample according to each intermediate layer in the feature extraction layer set, and calculating an attention map corresponding to each feature map;

[0012] performing a cross-category attention map comparison on the image dataset according to the attention map;

[0013] According to the result of the cross-category attention map comparison, the assimilation degree is calculated to obtain the assimilation score of each category in the image dataset, thereby determining whether the model is subject to a backdoor attack.

[0014] Preferably, in step S1, a cross-category attention map comparison of the image dataset is performed based on the attention map, specifically:

[0015] For any two categories in the image dataset, the Frobenius norm difference between the attention maps of each intermediate layer corresponding to the two categories is calculated.

[0016] Preferably, in step S1, the assimilation degree is calculated based on the result of the cross-category attention map comparison to obtain the assimilation score of each category in the image dataset, thereby determining whether the model is subject to a backdoor attack, specifically:

[0017] Calculating the standard deviation of the Frobenius norm difference between the attention maps of all intermediate layers corresponding to the two categories to obtain the assimilation score of each category in the image dataset;

[0018] According to the assimilation score of each category, it is determined whether the model is attacked by a backdoor.

[0019] Preferably, in step S2, when the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category, thereby locating the suspicious category, specifically:

[0020] When the model is attacked by a backdoor, the covariance matrix of the attention map of each sample in the category is calculated;

[0021] Calculate the average covariance matrix of the category based on the covariance matrix of the attention maps of all samples in the category;

[0022] For each sample in the category, the covariance discriminant score of the sample is calculated according to the covariance matrix of the sample and the average covariance matrix of the category; and the top ten highest covariance discriminant scores are taken for analysis to locate the suspicious category.

[0023] Preferably, in step S2, the covariance matrix of the attention map of each sample in the category is calculated, specifically:

[0024] Extract the attention maps of all intermediate layers corresponding to each sample in the category and flatten the attention maps into vectors;

[0025] Based on the vector, the covariance matrix of the attention map of each sample in the category is calculated.

[0026] Preferably, in step S2, for each sample in the category, the covariance discriminant score of the sample is calculated based on the covariance matrix of the sample and the average covariance matrix of the category; and the top ten highest covariance discriminant scores are taken for analysis to locate the suspicious category, specifically:

[0027] For each sample in the category, calculating the Frobenius norm difference between the covariance matrix of the sample and the average covariance matrix of the category to obtain a covariance discriminant score of the sample;

[0028] The top ten highest covariance discriminant scores are taken. If the samples corresponding to the top ten highest covariance discriminant scores are concentrated in a certain category, the category is determined to be a suspicious category.

[0029] Preferably, in step S3, data cleaning is performed according to the suspicious categories to obtain a clean data set, specifically:

[0030] Clean the samples of the suspicious categories to obtain a clean data set;

[0031] Alternatively, after isolating the suspicious category from the image dataset, anomaly detection is performed on samples in the suspicious category to determine whether the sample is embedded with a backdoor trigger; if so, the sample is removed from the image dataset; if not, the separate isolation of the sample is released to obtain a clean dataset.

[0032] Preferably, in step S4, a balanced data set is generated based on the clean data set, specifically:

[0033] Based on the samples removed from the clean dataset, non-poisoned samples that match the characteristics of the removed samples are generated through data augmentation and generative adversarial networks;

[0034] The non-poisoned samples are added to the clean dataset to generate a balanced dataset.

[0035] Preferably, in step S4, the model is retrained based on the balanced data set, specifically:

[0036] The model is retrained using the balanced dataset and regularization-based robustness constraints are introduced during training.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] First, the present invention uses multi-layer feature map extraction and attention map generation, and calculates the assimilation degree of each category using Frobenius norm difference, which can accurately capture the attention pattern assimilation phenomenon caused by backdoor attacks in the model. By analyzing the assimilation scores of different categories in the model, it can effectively identify whether the model is subject to backdoor attacks and locate potential poisoning types.

[0039] Second, this paper uses covariance discriminant analysis (CDA) to calculate the covariance matrix of each sample and compare it with the average covariance matrix of its category, which can capture subtle structural differences between samples. By using covariance discriminant analysis to sort samples, the paper can accurately locate poisoned samples and significantly improve the accuracy of backdoor attack detection.

[0040] Third, the present invention combines sample elimination and isolation mechanisms to ensure the purity of the model training dataset by screening and eliminating samples with high covariance discriminant analysis scores. Furthermore, through data augmentation and generative adversarial networks (GANs), high-quality samples similar to real data samples are generated to balance the distribution of data across different categories and prevent overfitting of the model due to data imbalance.

[0041] Fourth, the present invention further enhances the model's defense capabilities by retraining the model and introducing robustness constraints, ensuring that the model no longer relies on backdoor trigger features in actual application scenarios; combined with real-time backdoor detection and dynamic update mechanisms, it ensures the model's continued security and robustness during operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0043] Figure 1 This is a flow chart of the suspicious model backdoor category positioning method based on model assimilation provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It will be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] The terms "comprise," "comprising," and "having," and any variations thereof, as used herein, are intended to cover non-exclusive inclusions. For example, a process, method, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0046] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0047] See also Figure 1 As shown, the present invention provides a suspicious model backdoor category positioning method based on model assimilation, which includes the following steps:

[0048] Step S1: Input an image dataset containing several categories, use the image dataset to calculate the assimilation degree of the model, and determine whether the model is attacked by a backdoor.

[0049] Furthermore, in step S1, the assimilation degree of the model is calculated using the image dataset to determine whether the model is subject to a backdoor attack, specifically:

[0050] Obtain several intermediate layers from the model to form a set of feature extraction layers;

[0051] For each category of samples in the image dataset, generate the feature map corresponding to the sample according to each intermediate layer in the feature extraction layer set, and calculate the attention map corresponding to each feature map;

[0052] Based on the attention maps, we compare the attention maps across categories on the image dataset.

[0053] Based on the results of the cross-category attention map comparison, the assimilation degree is calculated to obtain the assimilation score of each category in the image dataset, thereby determining whether the model is subject to backdoor attacks.

[0054] Furthermore, in step S1, a cross-category attention map comparison of the image dataset is performed based on the attention map, specifically:

[0055] For any two categories in the image dataset, the Frobenius norm difference between the attention maps of each intermediate layer corresponding to the two categories is calculated.

[0056] Furthermore, in step S1, the assimilation degree is calculated based on the results of the cross-category attention map comparison to obtain the assimilation score of each category in the image dataset, thereby determining whether the model is subject to a backdoor attack. Specifically,

[0057] The standard deviation of the Frobenius norm difference between the attention maps of all intermediate layers corresponding to the two categories is calculated to obtain the assimilation score of each category in the image dataset;

[0058] Based on the assimilation score of each category, it is determined whether the model is attacked by a backdoor.

[0059] In practical applications, the model is first input with an image dataset containing several categories, which contains data that may contain backdoor samples.

[0060] For a data sample X in the image dataset, it is first forward propagated through the convolutional layer of the deep neural network in the model to generate the feature maps of each intermediate layer of the model. Let l1, l2, ..., l i are several intermediate layers obtained from the image classification model, then let is the selected feature extraction layer set. For each intermediate layer l i , the intermediate layer l can be obtained i Feature Map ,in Represents the middle layer l i The feature map has a size of C×H×W, where C represents the number of channels, and H and W represent the height and width of the feature map.

[0061] Preferably, feature maps of each data sample can be extracted from multiple intermediate layers of the model (such as the 5th, 15th, 30th and 40th layers) , where the selection of the above intermediate layers is based on their ability to better represent the hierarchical information of the attention pattern.

[0062] For each feature map , use the following formula to calculate the corresponding attention map ,

[0063]

[0064] In the above formula, Representation feature map The power of the element, C represents the number of channels, ‖‖ F represents the Frobenius norm, Represents a constant to prevent division by zero; the above process ensures that attention maps related to samples are generated on multiple intermediate layers.

[0065] For each category i and category j of the image dataset, calculate the difference in Frobenius norm between the attention maps of each intermediate layer l , the formula is as follows:

[0066]

[0067] In the above formula, and Represents the feature map of category i and category j, ‖‖ F Denotes the Frobenius norm. By calculating the difference in Frobenius norms between attention maps of multiple intermediate layers, the degree of assimilation of attention maps between each category i and category j can be obtained.

[0068] Based on the difference of Frobenius norm across categories (i.e., two different categories in the image dataset) , use the following formula to calculate the assimilation score of category i ,

[0069]

[0070] In the above formula, represents the standard deviation across categories, and C represents the number of channels. The larger it is, the smaller the difference between the attention maps of category i and other categories in the image dataset is, that is, the more likely category i is to be affected by the backdoor attack.

[0071] By analyzing the assimilation score of each category in the image dataset, we assess whether the samples in each category exhibit overall assimilation. For example, we can compare the assimilation score of each category with a preset assimilation score threshold. If the assimilation score is greater than the preset assimilation score threshold, we determine that the model has been attacked by a backdoor related to that category; otherwise, we determine that the model has not been attacked by a backdoor.

[0072] In step S2, when the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category, thereby locating the suspicious category.

[0073] Furthermore, in step S2, when the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category, thereby locating the suspicious category, specifically:

[0074] When the model is attacked by a backdoor, the covariance matrix of the attention map of each sample in the category is calculated;

[0075] Calculate the average covariance matrix of the category based on the covariance matrix of the attention maps of all samples in the category;

[0076] For each sample in the category, the covariance discriminant score of the sample is calculated based on the covariance matrix of the sample and the average covariance matrix of the category; and the top ten highest covariance discriminant scores are taken for analysis to locate the suspicious category.

[0077] Furthermore, in step S2, the covariance matrix of the attention map of each sample in the category is calculated, specifically:

[0078] Extract the attention map of all intermediate layers corresponding to each sample in the category and flatten the attention map into a vector;

[0079] Based on the vector, calculate the covariance matrix of the attention map of each sample in the category.

[0080] Furthermore, in step S2, for each sample in the category, the covariance discriminant score of the sample is calculated based on the covariance matrix of the sample and the average covariance matrix of the category; and the top ten highest covariance discriminant scores are analyzed to locate the suspicious category, specifically:

[0081] For each sample in the category, calculate the Frobenius norm difference between the sample's covariance matrix and the category's average covariance matrix to obtain the sample's covariance discriminant score;

[0082] Take the top ten highest covariance discriminant scores. If the samples corresponding to the top ten highest covariance discriminant scores are concentrated in a certain category, the category is determined to be a suspicious category.

[0083] In practical applications, for each sample, extract its attention map in each intermediate layer , and flatten it into a vector , used for further covariance analysis.

[0084] Specifically, the covariance matrix of the attention map of each sample is calculated using the following formula: ,

[0085]

[0086] In the above formula, Represents the covariance matrix of the vectorized attention map, which describes the structural relationship between different regions of the attention map.

[0087] In calculating the mean covariance matrix for each category , the mean covariance matrix It is obtained by averaging the covariance matrices of all samples in each category.

[0088] For sample X, through its covariance matrix The average covariance matrix with its category The Frobenius norm difference between the two is used to calculate the covariance discriminant score of sample X. , and its calculation formula is as follows,

[0089]

[0090] In the above formula, ‖‖ F represents the Frobenius norm.

[0091] When the covariance discriminant score of sample X The larger the value, the greater the deviation of sample C from the normal structural pattern of its category, and the more likely it is to be vulnerable to a backdoor attack. For example, the covariance discriminant score of each sample X can be compared with a preset discriminant score threshold. If the covariance discriminant score exceeds the preset discriminant score threshold, sample X is considered a suspicious sample; otherwise, sample X is considered not a suspicious sample. For another example, for all samples within the same category, all samples are ranked from high to low according to their respective covariance discriminant scores. In this ranking, samples with higher covariance discriminant scores (i.e., samples positioned closer to the top of the ranking) are more likely to be vulnerable to a backdoor attack. Alternatively, by analyzing samples with the highest covariance discriminant scores, the category most likely to be vulnerable to a backdoor attack can be identified. Generally, samples with high covariance discriminant scores are concentrated in a specific category, indicating that category may contain a backdoor. Preferably, the top ten highest covariance discriminant scores are taken. If the samples corresponding to the top ten highest covariance discriminant scores are concentrated in a particular category, the category is determined to be a suspicious category.

[0092] In step S3, data cleaning is performed based on the suspicious categories to obtain a clean data set.

[0093] Furthermore, in step S3, data cleaning is performed based on the suspicious categories to obtain a clean data set, specifically:

[0094] Clean samples of suspicious categories to obtain a clean data set;

[0095] Alternatively, after isolating the suspicious category from the image dataset, anomaly detection is performed on the samples in the suspicious category to determine whether the sample is embedded with a backdoor trigger; if so, the sample is removed from the image dataset; if not, the separate isolation of the sample is lifted to obtain a clean dataset.

[0096] In practical applications, samples with a covariance discriminant score greater than a preset threshold or with the highest covariance discriminant score can be extracted. These samples are considered most likely to be affected by a backdoor attack and are therefore marked as suspicious. If a large number of suspicious samples exist in the image dataset (i.e., the training dataset), these can be directly removed from the dataset to obtain a clean dataset. Alternatively, these suspicious samples can be isolated and then retested. This involves isolating the suspicious samples from the dataset and then performing anomaly detection (e.g., using feature-based adversarial detection) to determine whether the suspicious samples contain embedded backdoor triggers. If so, the suspicious samples are removed from the dataset; if not, the isolation of the suspicious samples is lifted, resulting in a clean dataset.

[0097] Step S4: Generate a balanced dataset based on the clean dataset; and retrain the model based on the balanced dataset.

[0098] Furthermore, in step S4, a balanced data set is generated based on the clean data set, specifically:

[0099] Based on the samples removed from the clean dataset, non-poisoned samples that match the characteristics of the removed samples are generated through data augmentation and generative adversarial networks;

[0100] The non-poisoned samples were added to the clean dataset to generate a balanced dataset.

[0101] Furthermore, in step S4, the model is retrained based on the balanced data set, specifically:

[0102] Retrain the model using a balanced dataset and introduce regularization-based robustness constraints during training.

[0103] In practical applications, for the clean dataset obtained above, new samples can be generated through data augmentation and generative adversarial networks (GANs) to supplement the removed samples. Specifically, GANs are used to generate high-quality images similar to the original removed samples. The GANs generator and discriminator undergo adversarial training to generate a large number of non-poisoned samples (i.e., non-poisoned images) that match the characteristics of the removed samples, ensuring that the distribution of the dataset is not imbalanced due to sample removal. For categories with high covariance discriminant scores (i.e., backdoor poisoning categories), new non-poisoned samples are generated to balance the number of samples across categories, ensuring that the model does not overfit certain categories due to data imbalance.

[0104] Retraining the model using samples from a balanced dataset ensures that the model no longer relies on backdoor trigger features, reducing the likelihood of further attacks. Furthermore, during model retraining, robustness constraints based on regularization are introduced to improve the model's resilience to anomalous data. This regularization can employ L2 regularization or other adversarial regularization methods. Continuously monitor the model's performance on the balanced dataset during retraining to ensure robust defenses in real-world scenarios.

[0105] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using a general-purpose hardware platform, or alternatively, through a combination of hardware and software. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. The suspicious model backdoor category location method based on model assimilation is characterized by: The method comprises the following steps: Step S1: Input an image dataset containing several categories, use the image dataset to calculate the assimilation degree of the model, and determine whether the model is attacked by a backdoor, which includes: Obtain several intermediate layers from the model to form a set of feature extraction layers; For each category of samples in the image dataset, generating a feature map corresponding to the sample according to each intermediate layer in the feature extraction layer set, and calculating an attention map corresponding to each feature map; According to the attention map, performing a cross-category attention map comparison on the image dataset, comprising: For any two categories in the image dataset, calculate the Frobenius norm difference between the attention maps of each intermediate layer corresponding to the two categories; Based on the result of the cross-category attention map comparison, an assimilation degree calculation is performed to obtain the assimilation score of each category in the image dataset, thereby determining whether the model is subject to a backdoor attack, which includes: Calculating the standard deviation of the Frobenius norm difference between the attention maps of all intermediate layers corresponding to the two categories to obtain the assimilation score of each category in the image dataset; Based on the assimilation score of each category, determine whether the model is attacked by a backdoor; Step S2: When the model is attacked by a backdoor, covariance discriminant analysis is used to calculate the covariance index of each category to locate the suspicious category, which includes: When the model is attacked by a backdoor, the covariance matrix of the attention map of each sample in the category is calculated, which includes: Extract the attention maps of all intermediate layers corresponding to each sample in the category and flatten the attention maps into vectors; Calculate the covariance matrix of the attention map of each sample in the category according to the vector; Calculate the average covariance matrix of the category based on the covariance matrix of the attention maps of all samples in the category; For each sample in the category, the covariance discriminant score of the sample is calculated based on the covariance matrix of the sample and the average covariance matrix of the category; and the top ten highest covariance discriminant scores are analyzed to locate the suspicious category, which includes: For each sample in the category, calculating the Frobenius norm difference between the covariance matrix of the sample and the average covariance matrix of the category to obtain a covariance discriminant score of the sample; Take the top ten highest covariance discriminant scores. If the samples corresponding to the top ten highest covariance discriminant scores are concentrated in a certain category, then determine that category as a suspicious category; Step S3: Perform data cleaning based on the suspicious categories to obtain a clean data set; Step S4: Generate a balanced dataset based on the clean dataset; and retrain the model based on the balanced dataset.

2. The method according to claim 1, characterized in that In step S4, a balanced data set is generated based on the clean data set, specifically: Based on the samples removed from the clean dataset, non-poisoned samples that match the characteristics of the removed samples are generated through data augmentation and generative adversarial networks; The non-poisoned samples are added to the clean dataset to generate a balanced dataset.

3. The method according to claim 2, characterized in that In step S4, the model is retrained based on the balanced data set, specifically: The model is retrained using the balanced dataset and regularization-based robustness constraints are introduced during training.

Images