Backdoor sample purification method based on multi-view difference guided reconstruction

Abstract

The application provides a backdoor sample purification method based on multi-view difference guided reconstruction, belongs to the technical field of information security, applies multiple random masks to an input image, reconstructs local areas blocked by the masks by using a pre-trained MAE, retains original pixels of unblocked areas, and generates a group of reconstructed images corresponding to different masks; quantifies structural differences, prediction differences and semantic differences between the input image and each reconstructed image, applies a suspicious contribution amplification mechanism with direction perception, enhances suspicious area response, and fuses the three types of differences to obtain a trigger heat map; and identifies potential trigger related areas based on the trigger heat map, and performs reconstruction processing on the areas to obtain purified images. The application avoids the distribution deviation and false deletion risk caused by the filtering method, concentrates the purification operation on the high suspicious local area, and balances the backdoor removal effect and the clean sample classification performance.

Description

Technical Field

[0001] This invention belongs to the field of information security technology, specifically, it relates to a backdoor sample purification method based on multi-view difference-guided reconstruction. Background Technology

[0002] Deep neural networks (DNNs) have been widely applied in scenarios such as facial recognition, autonomous driving, and medical image diagnosis. However, DNNs are vulnerable to backdoor attacks, which seriously affect the reliability of deployed models. In a typical backdoor attack, an attacker injects a small number of poisoned samples with trigger patterns into the training set, causing the model to associate the trigger with a target label specified by the attacker. During the inference phase, the backdoor model performs normally on clean samples, but will produce incorrect predictions for samples with triggers. Therefore, backdoor defense against poisoned data has become an important research direction for improving the security of DNN training.

[0003] Existing backdoor attacks can be mainly divided into two categories based on whether the labels of the poisoned samples are modified: label poisoning attacks and clean label attacks. Specifically, label poisoning attacks inject a trigger designed by the attacker into some training samples and modify its label to the target category.

[0004] While various label poisoning attacks have been proposed, label modification can introduce semantic inconsistencies between the image and its label, making poisoned samples easier to detect with existing defenses and even manual inspection. To further enhance the stealth of attacks, clean label attacks have been proposed. Unlike label poisoning attacks, clean label attacks preserve the original label of the poisoned sample, injecting triggers only into the sample. Furthermore, recent attack methods have further utilized imperceptible, dynamic, or sample-specific triggers to enhance stealth, and the continuous improvement in the stealth of backdoor attacks has significantly increased the difficulty of backdoor defense.

[0005] Existing backdoor defense methods for poisoning data can be mainly divided into two categories: filtering-based defense methods and purification-based defense methods; (1) Filter-based defense methods mitigate backdoor threats by detecting and removing suspected poisoned samples. These defenses typically use differences in response between clean and poisoned samples, such as changes in loss, deviations from the reference data distribution, and other anomalous responses, to determine whether a sample is poisoned. Samples identified as poisoned are then removed to prevent them from affecting model training or inference.

[0006] (2) Decontamination-based defense methods aim to restore poisoned samples to clean samples. This type of defense removes triggers while preserving the original semantic content of the sample. The decontamination process breaks the association between the triggers and the target label, so that the decontamination sample no longer activates the backdoor and restores normal classification behavior.

[0007] However, both of these methods have their own limitations: (1) Filtering-based defense methods rely heavily on the accurate identification of poisoned samples. However, directly removing samples identified as poisoned will change the original data distribution of the dataset and reduce data diversity, thereby further weakening the generalization ability of the model. In addition, some clean samples may be similar to poisoned samples in terms of model response and feature distribution. This may lead to the wrong deletion of clean samples during the filtering process, further affecting data quality. (2) Cleaning-based defense methods also have limitations. Localization-based cleaning defense methods first identify the trigger-related regions and then perform local cleaning. However, when the trigger is small, hidden, or has input adaptability, it is still difficult to accurately locate the trigger. Global transformation-based cleaning defense methods process the entire sample through noise injection, image transformation, or reconstruction. Such global operations may destroy benign semantic features and reduce classification accuracy. For cleaning-based defense methods, the difficulty lies in accurately locating and cleaning the trigger-related regions in the absence of prior knowledge of trigger patterns, while preserving the semantic features and classification accuracy of the samples. Summary of the Invention

[0008] Deep neural networks (DNNs) are vulnerable to backdoor attacks, where attackers contaminate training data to implant hidden backdoors into the victim model. However, in the absence of prior knowledge about triggers, existing data sanitization defense methods often struggle to effectively remove malicious trigger features while preserving good semantic features. To address this issue, this invention proposes a data sanitization framework based on mask autoencoder (SPeSMAE). SPeSMAE captures the structural, predictive, and semantic differences caused by reconstruction, locates trigger regions, and removes malicious trigger features from the located suspicious regions while retaining good semantic features.

[0009] This invention is achieved through the following technical solution: a backdoor sample purification method based on multi-view difference-guided reconstruction, the method specifically including the following steps: Step 1, Multi-mask reconstruction: Apply multiple random masks to the input image, use pre-trained MAE to reconstruct the local areas occluded by the masks, retain the original pixels of the unoccluded areas, and generate a set of reconstructed images corresponding to different masks. Step 2, Multi-view difference fusion and heatmap construction: Quantify the structural differences, prediction differences and semantic differences between the input image and each reconstructed image, apply the direction-aware suspicious contribution amplification mechanism to enhance the response of suspicious regions, and use a weighted summation method to fuse the three types of differences to construct a trigger heatmap; Step 3, Trigger Region Reconstruction and Cleaning: Based on the trigger heatmap, identify potential trigger-related regions, reconstruct these regions, and obtain the cleaned image.

[0010] Furthermore, in step 1, the input image is adjusted to the input resolution of the pre-trained MAE and then divided into several non-overlapping patch tokens; Perform multiple rounds of reconstruction operations, sampling multiple random masks in each round, and marking the occlusion status of each patch through the masks; Using a pre-trained MAE, the content of the occluded region is inferred based on the visible patch, generating a complete reconstructed image and adjusting it back to the original input size; Only the pixels in the occluded areas are replaced with the reconstructed content, while the original pixels in the unoccluded areas are retained, to obtain the reconstructed image with the corresponding mask; at the same time, the total number of occlusions for each patch is counted.

[0011] Furthermore, in step 2, the process of quantifying semantic differences is as follows: CLIP is used as a zero-shot classifier, and the image-label semantic entropy score is calculated based on the semantic compatibility of the image and the label. By comparing the changes in semantic entropy scores before and after mask reconstruction, the differences are accumulated onto the corresponding occluded patches. The changes are then normalized by combining the number of occlusions for each patch to obtain the patch-level semantic difference score.

[0012] Furthermore, in step 2, the quantification process of the predicted difference is as follows: the prediction entropy score is calculated based on the prediction probability of the sample by the suspicious model; By comparing the changes in prediction entropy scores before and after mask reconstruction, the differences are accumulated onto the corresponding occluded patches. The differences are then normalized by combining the number of occlusions for each patch to obtain the patch-level prediction difference score.

[0013] Furthermore, in step 2, the quantification process of structural differences is as follows: the structural similarity index is used to measure the structural changes between the original image and the reconstructed image; The structural differences are accumulated to the corresponding occluded patches, and then normalized by combining the number of occlusions of each patch to obtain a patch-level structural difference score.

[0014] Furthermore, in step 2, the suspected contribution amplification mechanism of direction perception specifically refers to: For each difference perspective, the average difference corresponding to all reconstructed images of the same input image is calculated as the image-specific reconstruction benchmark under that perspective; the original difference generated by a single reconstruction is converted into a suspicious contribution, amplifying the abnormal response that deviates from the reconstruction benchmark and highlighting the features of the potential trigger region.

[0015] Furthermore, in step 3, potential trigger-related regions are located based on the response intensity of the trigger heatmap, and content reconstruction is performed only in these regions to eliminate the influence of the triggers, while the remaining image regions retain their original pixels, ultimately generating a cleaned image.

[0016] A backdoor sample purification system based on multi-view difference-guided reconstruction; The system includes a multi-mask reconstruction module, a multi-view difference fusion and heatmap construction module, and a trigger region reconstruction and purification module; The multi-mask reconstruction module is used to apply multiple random masks to the input image, reconstruct the local area occluded by the mask using a pre-trained MAE, retain the original pixels of the unoccluded area, and generate a set of reconstructed images corresponding to different masks. The multi-view difference fusion and heatmap construction module is used to quantify the structural differences, prediction differences and semantic differences between the input image and each reconstructed image. It applies a direction-aware suspicious contribution amplification mechanism to enhance the response of suspicious regions and uses a weighted summation method to fuse the three types of differences to construct a trigger heatmap. The trigger region reconstruction and purification module identifies potential trigger-related regions based on the trigger heatmap, reconstructs these regions, and obtains the purified image.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: The SPeSMAE proposed in this invention does not rely on prior knowledge of the attack type, trigger pattern, or trigger location. Instead, it uses the anomalous differences induced by mask reconstruction to inversely characterize the importance of the trigger region. On the one hand, it avoids the distribution shift and false deletion risks associated with filtering methods. On the other hand, it differs from cleanup methods that perform coarse-grained global processing on the entire image, because this invention focuses the cleanup operation on highly suspicious local areas, thus having a better chance of balancing backdoor removal effectiveness and clean sample classification performance.

[0018] This invention starts with the anomalies induced by backdoor attacks, pointing out that there are exploitable differences between poisoned and clean samples at three levels: structure, prediction, and semantics. These differences are then used as a unified clue for region localization under conditions without prior triggers. Multi-round random mask reconstruction is used to characterize the contribution of local regions to the anomaly response of the backdoor.

[0019] This invention also introduces a direction-aware suspicious contribution enhancement mechanism on top of multi-view difference attribution, in order to strengthen fine-grained anomaly responses that may otherwise be submerged by random reconstruction noise, making potential triggering regions easier to identify in the final heatmap.

[0020] Experiments were conducted on multiple datasets, model architectures, and representative attack settings. The results show that the method can significantly reduce the attack success rate while maintaining the accuracy of clean samples, demonstrating good effectiveness and generalization potential. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the SPeSMAE model of the present invention; Figure 2 This is a schematic diagram of a backdoor defense method in the prior art. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Unless otherwise specified, the experimental methods used in the following examples are conventional methods. Unless otherwise specified, the materials, reagents, methods, and instruments used are all conventional materials, reagents, methods, and instruments in the art, and can be obtained commercially by those skilled in the art.

[0024] Example: This invention proposes a data sanitization framework SPeSMAE based on a masked autoencoder. The basic idea is that backdoor triggers introduce anomalous responses into the local structure of samples, model prediction behavior, and image-label semantic relationships. Therefore, if multiple random masks are applied to the input samples and reconstruction is performed using a pre-trained masked autoencoder, then when some masks cover the trigger-related regions, the reconstruction results will exhibit observable differences from the original image in terms of structure, prediction, and semantics.

[0025] Based on this observation, SPeSMAE first performs multiple rounds of random mask reconstruction on the input image, and then attributes the abnormal contribution of local patches from three dimensions: structural difference, prediction difference and semantic difference, and integrates the multi-view differences into a trigger heatmap. On this basis, the method further locates potential triggering regions and performs targeted purification and reconstruction only on suspicious regions, thereby suppressing malicious triggering features while preserving the benign semantic information of the samples as much as possible.

[0026] (1) Problem definition Problem Definition: The goal of the image classification task is to learn a model. Input image Mapping to tags , Representing image space, Represents the tag space. Indicates model parameters.

[0027] Consider an image classification task whose goal is to learn a classification model. The model will take the input image Mapping to tags .

[0028] here, Representing image space, This represents a label space containing C categories. This represents the model parameters. Given an input image x, the model's predicted label can be defined as:

[0029] in, Representation Model The probability assigned to class y given input x. (1.1) Backdoor Attack: A backdoor attack aims to establish a hidden trigger-target label association in the model through data poisoning. Specifically, the attacker first injects a function by applying a trigger. The attacker embeds a trigger designed by themselves into a portion of the training samples, thus constructing a poisoned training dataset. Subsequently, the model trained using this poisoned dataset is converted into a backdoor model, denoted as... During the reasoning stage, when the poisoned sample... When fed into a backdoor model, the model is induced to output the attacker-specified target label instead of the actual label. This process can be represented as:

[0030] Where y represents the true label of the original clean sample x.

[0031] (1.2) Purification Target: The target of the defense method of this invention is to construct a purification model during the reasoning phase. This is used to purify poisoned samples. Given a poisoned sample containing a trigger... Purification Model It will remove the embedded triggers and generate a cleaned image. This allows the cleaned image to recover the classification result of the original clean sample x. Formally, this objective can be expressed as:

[0032] (1.3) Threat Model: This invention adopts a poisoning-based threat model commonly used in previous backdoor defense research. Under this setting, attackers can inject pre-constructed samples with triggers into the training dataset during the training phase, forcing the model to learn the hidden association between the triggers and the attacker's specified target labels, thereby generating a backdoor model. Defenders have no prior knowledge about the attack type, trigger pattern, or trigger location, nor can they determine in advance whether an input sample is a poisoned sample.

[0033] Meanwhile, it is assumed that the defender has access to publicly available models and datasets. Under this threat model, the defender's goal is to sanitize poisoned samples during the inference phase to prevent them from activating the target labels specified by the attacker. Therefore, this defense aims to reduce the attack success rate (ASR) while maintaining the classification accuracy (ACC) on clean samples.

[0034] (2) Backdoor attack observation In backdoor attack research, poisoning-based attack paradigms can generally be divided into labeled poisoning attacks and clean-label attacks. Both attack paradigms generate backdoor models by injecting poisoned samples with embedded triggers into the training set, but they have different backdoor behaviors.

[0035] Tag poisoning attacks exploit the strong correlation between triggers and target tags, causing poisoned samples to be predicted as the target tag. In contrast, clean tag attacks also generate poisoned samples through trigger injection, but retain the original tag of the poisoned sample. Based on the different characteristics of these two types of backdoor attacks, poisoned samples and clean samples exhibit the following three types of differences: (2.1) Semantic differences: In label poisoning attacks, poisoned samples exhibit image-label mismatch, meaning the assigned labels deviate from the sample's true semantic content. In contrast, clean samples retain their original labels and maintain higher image-label semantic consistency. Based on this observation, this invention employs CLIP as a zero-shot classifier and uses image-label semantic entropy scores to characterize the semantic differences between poisoned and clean samples.

[0036] For each sample in the training set CLIP will be in the predefined tag set Generate a logits vector. This is used to measure the semantic compatibility between the image and each candidate label. Then, this invention predicts labels for the classification model based on CLIP. The corresponding probabilities are used to calculate the image-label semantic matching confidence score, which is then further converted into a semantic entropy score. :

[0037] in, CLIP indicates that the sample Assigned to prediction labels The probability, which measures the semantic similarity between image content and labels. A higher probability... This indicates a stronger inconsistency between the image semantics and the corresponding label, suggesting that the sample is more likely to be a poisoned sample.

[0038] On the contrary, normal This indicates a higher semantic consistency between the image and its corresponding label, suggesting that the sample is more likely to be a clean sample. Therefore, poisoned samples and clean samples exhibit semantic differences.

[0039] (2.2) Prediction discrepancies: In clean-label attacks, poisoned samples are constructed without altering their original labels. Simultaneously, the embedded triggers act as strong discriminative features and establish a strong correlation with the target label. Compared to clean samples, poisoned samples exhibit higher prediction confidence on suspicious models. Based on this observation, this invention employs the prediction entropy score of suspicious models. To characterize the predictive differences between poisoned and clean samples.

[0040]

[0041] in, This indicates that the suspicious model will sample Assigned to label The probability is used to measure the confidence level of the model's prediction for that sample.

[0042] lower This indicates that the model is effective with labels. A higher predictive confidence level indicates that the sample is more likely to be a poisoned sample. Conversely, a normal sample... This indicates that the sample is more likely to be clean. Therefore, the poisoned sample and the clean sample show a predictive difference.

[0043] (2.3) Structural differences: In backdoor attacks, triggers in poisoned samples are typically maliciously injected anomalous patterns that lack consistency with the normal features of the original image. In contrast, clean samples retain the inherent structural features of the original image. Based on this observation, this invention employs the structural similarity index SSIM to characterize the structural differences between poisoned and clean samples.

[0044]

[0045] in, Indicates the original clean sample Corresponding poisoning samples The structural similarity between them. High This indicates that the structural similarity between the two samples is low, suggesting that the structural differences caused by the trigger are more significant.

[0046] (3) Backdoor triggers are the main reason for the differences between poisoned and clean samples. Although different backdoor attacks use different trigger patterns, trigger injection usually changes the local structure of the sample, the predictive behavior of the model, or the image-label semantic relationship, thereby inducing abnormal responses at the structural, predictive and semantic levels.

[0047] Therefore, as Figure 1 As shown, SPeSMAE consists of three main steps. First, SPeSMAE applies multiple random masks to the input image and generates a set of reconstructed images using a pre-trained MAE. Second, it quantifies the structural, predictive, and semantic differences between the input image and its reconstructed counterpart, applies a direction-aware suspicious contribution amplification mechanism to enhance the response of suspicious regions, and fuses the three differences to construct a trigger heatmap. Finally, it identifies potential trigger regions based on the heatmap and reconstructs the corresponding regions to obtain the cleaned-up image.

[0048] (3.1) Multi-mask reconstruction: To evaluate the contribution of different local regions to backdoor-induced anomalies, SPeSMAE performs multi-mask reconstruction in patch-token space using a pre-trained MAE. Given an input image, SPeSMAE repeatedly samples multiple random patch masks. It then reconstructs the occluded regions while preserving the original content of the unoccluded regions. Thus, SPeSMAE obtains a set of reconstructed images in which different local regions are selectively modified, providing a basis for subsequent multi-view difference analysis.

[0049] In a formal way, make This represents the i-th input image. SPeSMAE first... Adjust to pre-trained MAE The required input resolution, and divide it into A number of non-overlapping patchtokens.

[0050] In an embodiment of the present invention, the input resolution of MAE is ,and Therefore, we get P = 196 patch tokens. SPeSMAE performs r rounds of reconstruction, sampling k random masks in each round. Let:

[0051] in This represents the k-th mask sampled in the r-th round of reconstruction. This represents the occlusion state of the t-th patch. Specifically, This indicates that the t-th patch is occluded, and This indicates that the patch remains visible.

[0052] For the k-th mask sampled in the r-th round, the pre-trained MAE uses the visible patch to infer the content of the occluded region. Then, it combines the recovered patches into a single image with a resolution of [resolution value missing]. The complete reconstructed image was then resized back to its original input size. .

[0053]

[0054] in, Indicates in the mask The reconstructed image is then generated and resized back to the original input size. Simultaneously, a patch-token mask is used. Reshaped into a The spatial grid was adjusted to the original input size. :

[0055] To avoid introducing additional errors in unoccluded areas during image scaling and MAE reconstruction, SPeSMAE only uses the MAE reconstructed content to replace occluded areas, while preserving the original pixels in the unoccluded areas. The final reconstructed image is defined as:

[0056] SPeSMAE performs R rounds of multimask reconstruction on each input image, sampling in each round. A random mask. After... After this operation, the reconstructed image set is as follows:

[0057] This reconstructed image set stores the reconstruction results of the same input image under different random masks, and provides a basis for subsequent comparison of the structural, predictive, and semantic differences between the original and reconstructed images. Since different patches may be occluded at different numbers during random sampling, this invention further records the occlusion count of the t-th patch:

[0058] Subsequently, It is used to normalize the cumulative difference contribution of each patch, thereby reducing the impact of inconsistent random occlusion frequencies on local contribution estimation.

[0059] (3.2) Multi-perspective differential attribution After obtaining the reconstructed image set, SPeSMAE attributes the differential changes induced by each reconstruction to the corresponding occluded patch. When a random mask covers the trigger-related region, MAE reconstructs these regions based on the natural image context. Therefore, the trigger effect is weakened, and the original and reconstructed images exhibit structural, predictive, and semantic differences. By accumulating differential values ​​across multiple random masks, SPeSMAE estimates the contribution of each patch to backdoor-induced anomalies from three complementary perspectives, thus providing a basis for subsequent trigger localization.

[0060] (3.2.1) Semantic Difference Score: From a semantic perspective, SPeSMAE measures the change in image-label semantic consistency before and after mask reconstruction. Let... Represents the original image The semantic difference score, let Represents the reconstructed image The semantic difference score. The semantic difference change caused by the k-th mask in the r-th round is defined as... .

[0061] Then, based on the occlusion records of multiple random masks, SPeSMAE attributes semantic difference changes to the corresponding occluded patches. Specifically, whenever a patch is occluded, these changes are accumulated for that patch and normalized using its occlusion count to obtain a patch-level semantic difference score.

[0062] higher The results indicate that rebuilding the t-th patch leads to a larger change in semantic response, suggesting that the patch may be related to semantic inconsistencies induced by the trigger.

[0063] (3.2.2) Prediction Difference Score: For the prediction perspective, SPeSMAE measures the change in the prediction response of the suspected model before and after mask reconstruction. Let... Represents the original image The predicted difference score, let Represents the reconstructed image The prediction difference score. The change in prediction difference caused by the k-th mask in the r-th round is defined as... Following the same attribution strategy described above, SPeSMAE aggregates predicted variance changes into patch-level predicted variance scores:

[0064] higher This indicates that rebuilding the t-th patch leads to a greater change in the predicted response of the suspicious model, suggesting that the patch has a stronger impact on the model's predictive behavior.

[0065] (3.2.3) Structural Difference Score: For a structural perspective, SPeSMAE measures the structural changes between the original and reconstructed images. For the reconstructed image generated by the k-th mask in the r-th round, the structural difference is defined as... Following the same attribution strategy described above, SPeSMAE aggregates structural variation changes into patch-level structural variation scores:

[0066] higher This indicates that rebuilding the t-th patch leads to more significant structural changes, suggesting that the patch is more likely to contain trigger-related exception patterns.

[0067] (3.2.4) Difference Heatmap Score: In real-world scenarios, it is impossible to know in advance whether an attack is a poisoned label attack or a clean label attack, nor is it possible to know the specific form of the trigger. Therefore, this invention adopts a three-view unified fusion strategy. It fuses the structural difference score through a weighted summation method. Predicted difference scores and semantic difference score This is used to construct the final patch-level difference heatmap. The formula is as follows:

[0068] in, and .

[0069] (3.3) Amplification of the suspected contribution of orientation perception Due to the complexity of image content and the randomness of multi-mask reconstruction, the difference response between the input image and its reconstructed counterpart is usually weak. To prevent the subtle responses generated by trigger-related regions from being diluted by normal reconstruction changes when the original differences are directly accumulated onto the occluded patch, this invention proposes a direction-aware suspicious contribution amplification method, namely Direction-aware Suspicious Contribution Amplification (DDCA). This method converts the original differences into suspicious contributions, thereby enhancing the discriminative anomalous response and making the potential trigger region more prominent in subsequent patch-level heatmaps.

[0070] For each perspective DDCA calculates the average difference across all reconstructed images of the same input image. This is used as the image-specific reconstruction baseline. Subsequently, the differences generated by the reconstruction corresponding to the k-th mask in the r-th round are converted into questionable contributions:

[0071] Here, the first factor represents the overall difference intensity of the current image at viewpoint v, while the second factor measures the relative deviation of the difference generated by the reconstruction corresponding to the k-th mask in the r-th round from a specific image reconstruction baseline. Parameters Control the amplification intensity, Control perspective level contribution, Used to ensure numerical stability.

[0072] SPeSMAE was evaluated on multiple datasets and different types of backdoor attacks. Experimental results show that SPeSMAE achieves a lower attack success rate while maintaining clean sample accuracy compared to state-of-the-art baseline methods. These results demonstrate the effectiveness of SPeSMAE in backdoor data cleansing.

[0073] A computer device system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method. A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0074] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0075] The memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0076] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means such as coaxial cable, optical fiber, digital subscriber line, DSL, or wireless means such as infrared, wireless, microwave, etc. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium such as a floppy disk, hard disk, magnetic tape; an optical medium such as a high-density digital video disc, DVD; or a semiconductor medium such as a solid-state disk, SSD, etc.

[0077] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0078] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0079] The foregoing has provided a detailed description of the backdoor sample purification method based on multi-view difference-guided reconstruction proposed in this invention, and has elucidated the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A backdoor sample cleanup method based on multi-view difference-guided reconstruction, characterized in that: The method specifically includes the following steps: Step 1, Multi-mask reconstruction: Apply multiple random masks to the input image, use pre-trained MAE to reconstruct the local areas occluded by the masks, retain the original pixels of the unoccluded areas, and generate a set of reconstructed images corresponding to different masks. Step 2, Multi-view difference fusion and heatmap construction: Quantify the structural differences, prediction differences and semantic differences between the input image and each reconstructed image, apply the direction-aware suspicious contribution amplification mechanism to enhance the response of suspicious regions, and use a weighted summation method to fuse the three types of differences to construct a trigger heatmap; Step 3, Trigger Region Reconstruction and Cleaning: Based on the trigger heatmap, identify potential trigger-related regions, reconstruct these regions, and obtain the cleaned image.

2. The method according to claim 1, characterized in that: In step 1, the input image is adjusted to the input resolution of the pre-trained MAE and then divided into several non-overlapping patchtokens; Perform multiple rounds of reconstruction operations, sampling multiple random masks in each round, and marking the occlusion status of each patch through the masks; Using a pre-trained MAE, the content of the occluded region is inferred based on the visible patch, generating a complete reconstructed image and adjusting it back to the original input size; Only the pixels in the occluded areas are replaced with the reconstructed content, while the original pixels in the unoccluded areas are retained, to obtain the reconstructed image with the corresponding mask; at the same time, the total number of occlusions for each patch is counted.

3. The method according to claim 2, characterized in that: In step 2, the process of quantifying semantic differences is as follows: CLIP is used as a zero-shot classifier, and the image-label semantic entropy score is calculated based on the semantic compatibility of the image and the label. By comparing the changes in semantic entropy scores before and after mask reconstruction, the differences are accumulated onto the corresponding occluded patches. The changes are then normalized by combining the number of occlusions for each patch to obtain the patch-level semantic difference score.

4. The method according to claim 3, characterized in that: In step 2, the quantification process of the predicted difference is as follows: the prediction entropy score is calculated based on the prediction probability of the sample by the suspicious model; By comparing the changes in prediction entropy scores before and after mask reconstruction, the differences are accumulated onto the corresponding occluded patches. The differences are then normalized by combining the number of occlusions for each patch to obtain the patch-level prediction difference score.

5. The method according to claim 4, characterized in that: In step 2, the quantification process of structural differences is as follows: the structural similarity index is used to measure the structural changes between the original image and the reconstructed image; The structural differences are accumulated to the corresponding occluded patches, and then normalized by combining the number of occlusions of each patch to obtain a patch-level structural difference score.

6. The method according to claim 5, characterized in that: In step 2, the suspected contribution amplification mechanism of direction perception specifically refers to: For each difference perspective, the average difference of all reconstructed images of the same input image is calculated as the image-specific reconstruction benchmark under that perspective; the original difference generated by a single reconstruction is converted into a suspicious contribution, amplifying the abnormal response that deviates from the reconstruction benchmark and highlighting the features of the potential trigger region.

7. The method according to claim 6, characterized in that: In step 3, potential trigger-related regions are located based on the response intensity of the trigger heatmap. Content reconstruction is performed only in these regions to eliminate the influence of the triggers, while the remaining image regions retain their original pixels, ultimately generating a cleaned image.

8. A backdoor sample purification system based on multi-view difference-guided reconstruction, characterized in that: The system is applied to the backdoor sample purification method based on multi-view difference-guided reconstruction as described in any one of claims 1 to 7; The system includes a multi-mask reconstruction module, a multi-view difference fusion and heatmap construction module, and a trigger region reconstruction and purification module; The multi-mask reconstruction module is used to apply multiple random masks to the input image, reconstruct the local area occluded by the mask using a pre-trained MAE, retain the original pixels of the unoccluded area, and generate a set of reconstructed images corresponding to different masks. The multi-view difference fusion and heatmap construction module is used to quantify the structural differences, prediction differences and semantic differences between the input image and each reconstructed image. It applies a direction-aware suspicious contribution amplification mechanism to enhance the response of suspicious regions and uses a weighted summation method to fuse the three types of differences to construct a trigger heatmap. The trigger region reconstruction and purification module identifies potential trigger-related regions based on the trigger heatmap, reconstructs these regions, and obtains the purified image.

9. A computer device system, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

Images