A Privacy-Preserving Federated Distillation and Backdoor Defense Method for Fine-Tuning Large Models

Abstract

This invention proposes a privacy-preserving federated distillation and backdoor defense method for large-scale model fine-tuning, belonging to the fields of artificial intelligence security and federated learning technology. The method includes: Step 1, dataset distillation and core representation extraction based on contribution-weighted federated pre-training and local neural feature function matching; Step 2, adaptive noise backdoor defense processing based on multi-feature fusion; and Step 3, verification of the large-scale model fine-tuning and backdoor defense effectiveness. This invention employs a dataset distillation mechanism based on neural feature function matching and an adaptive noise defense strategy, achieving an effective balance between data simplification, privacy protection, and backdoor defense robustness for large-scale models. This is of great significance for improving the security and reliability of artificial intelligence systems in distributed environments.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security and federated learning technology, specifically involving a privacy-preserving federated distillation and backdoor defense method for fine-tuning large models. Background Technology

[0002] In recent years, artificial intelligence technology based on large-scale pre-trained models has made groundbreaking progress. Its powerful capabilities in tasks such as natural language processing, computer vision, and multimodal understanding are driving the accelerated transformation of various industries towards intelligence. To enable these general-purpose models to adapt to the specialized needs of specific fields such as healthcare, finance, and autonomous driving, fine-tuning the models using high-quality domain data has become a crucial step. This domain data, often held by different companies, institutions, or individuals, contains rich and unique domain knowledge, which is irreplaceable in improving the accuracy, robustness, and reliability of models in specific downstream tasks.

[0003] However, in real-world applications, directly centralizing private data from various clients for model fine-tuning faces fundamental obstacles. Increasingly stringent data privacy and security regulations prevent raw data from being shared locally, creating the "data silo" problem in data fusion. Even under privacy compliance, the massive number of parameters in large models translates to enormous fine-tuning costs. Transmitting all massive amounts of distributed data to a central location for training would create unbearable network communication bandwidth pressure and latency.

[0004] To address these challenges, federated learning, as a distributed machine learning paradigm, has been widely studied. It allows clients to train models locally using private data, only uploading model updates to the server for aggregation. Theoretically, this can achieve collaborative model optimization without exposing the original data. However, directly applying traditional federated learning frameworks to large-scale model fine-tuning scenarios reveals a series of serious technical shortcomings. Large models often have billions or even hundreds of billions of parameters. Performing complete local training on resource-constrained edge clients and frequently uploading massive model updates incurs high computational and communication costs, making it difficult to meet the efficiency requirements of practical deployments. Furthermore, the data from each client typically originates from different environments and user groups, exhibiting significant non-independent and identically distributed characteristics. Traditional federated averaging and other aggregation algorithms are prone to model bias on non-independent and identically distributed data, leading to a significant decrease in the generalization performance of the aggregated global model on unseen new data or data from specific clients, failing to achieve the desired fine-tuning effect.

[0005] More seriously, the distributed and open nature of federated learning makes it highly vulnerable to backdoor attacks. Malicious clients can implant specific triggers, imperceptible to the human eye, into local data and upload corrupted model updates. This attack is highly stealthy; the aggregated global model will execute pre-defined malicious behavior when encountering input containing triggers, but behaves normally with normal inputs, severely jeopardizing the model's security and reliability. Existing technical solutions attempt to address some of these issues, but often compromise on other aspects, failing to achieve a comprehensive optimal solution. For example, to reduce communication overhead, some research employs knowledge distillation or dataset distillation, generating simplified representations of data on the client side before uploading. However, existing distillation methods struggle to effectively preserve the diversity and core features of global data in highly non-independent and identically distributed data distributions, resulting in distilled datasets that cannot support good generalization capabilities for the model. Regarding defense against backdoor attacks, existing solutions largely rely on anomaly detection of uploaded model updates on the server side or the addition of differential privacy noise. The former has limited ability to recognize complex, adaptive backdoor patterns for large models; the latter, when adding sufficient noise to ensure security, will inevitably impair the final accuracy and usability of the model, and cannot achieve a good balance between defense effectiveness and model performance.

[0006] In conclusion, given the current technological context, there is an urgent need for an innovative solution that can simultaneously meet the three core objectives of strict privacy protection, efficient collaborative learning, and strong security robustness. Summary of the Invention

[0007] This invention proposes a privacy-preserving federated distillation and backdoor defense method for large model fine-tuning, aiming to effectively solve the challenge of large model fine-tuning in scenarios where multiple clients cannot directly share private data. It overcomes the shortcomings of existing technologies, such as high communication overhead, insufficient model generalization ability, and difficulty in suppressing backdoor attack threats. This invention is applicable to large model training and fine-tuning scenarios involving multiple clients within a federated learning framework, supporting efficient fine-tuning and secure aggregation of image-text multimodal data under privacy-preserving conditions. In typical business scenarios such as cross-domain model pre-training, local data distillation and core feature extraction, and server-side secure fine-tuning, a dataset distillation mechanism based on neural feature function matching and an adaptive noise defense strategy are adopted. This achieves an effective balance between data simplification, privacy protection, and robustness against backdoor attacks, significantly improving the model's generalization performance and security reliability on specific downstream tasks.

[0008] This method mainly consists of an edge client, an edge server, and a cloud server. The edge client is responsible for local data distillation and core feature extraction; the edge server acts as a coordination and defense hub, responsible for aggregating distilled data and executing adaptive backdoor defense; and the cloud server is responsible for fine-tuning the large model and final verification.

[0009] Furthermore, the method comprises three steps: a dataset distillation and core representation extraction stage based on contribution-weighted federated pre-training and local neural feature function matching; an adaptive noise backdoor defense processing stage based on multi-feature fusion; and a large-scale model fine-tuning and backdoor defense performance verification stage. Specifically, it includes the following steps:

[0010] Step 1: In the dataset distillation and core representation extraction stage, which involves contribution-weighted federated pre-training and local neural feature function matching, the edge server and each client collaborate to build a globally pre-trained model. The model's feature diversity and generalization ability are enhanced by mixing model weights from different clients. Each client then uses the pre-trained model to map its local raw data to the latent space. A minimization optimization framework is constructed using neural feature function difference metrics, achieving data distribution compression from large-scale raw data to a high-fidelity small-scale distillation dataset. This stage significantly reduces subsequent communication overhead while ensuring the raw data remains locally, and provides the model with more representative core features.

[0011] Preferably, step one mainly includes two steps:

[0012] Step 1: The edge server and client train a globally pre-trained model through federated learning, and use a dynamic weighted aggregation strategy based on client update quality evaluation to enhance the model's generalization ability on non-independent and identically distributed data and suppress potential malicious updates. The specific process is as follows:

[0013] Edge server initializes global pre-trained model parameters and coordinate Each client participates in federated learning training.

[0014] In the During each round of federated learning, the edge server will display the current global model. Distribute to each client.

[0015] Each client Based on local multi-source heterogeneous data The received global model is trained locally, and the target is optimized. Get local model update .

[0016] The edge server calculates the robustness weights for updating the model on each client. The calculation method is a weighted combination of two evaluation results: updated similarity and loss change. The specific steps are as follows:

[0017] (1) Calculate the weights based on the updated similarity Computing Client Model update Update with all other client models The average cosine similarity, i.e. .

[0018] (2) Calculate the weights based on the change in loss. According to the client The weights are calculated based on the changes in the local training loss; the more significant the decrease in loss, the higher the weight. Specifically:

[0019] (1),

[0020] in, This is a hyperparameter greater than zero, used to adjust the sensitivity to changes in the magnitude of the loss. Indicates the first The client in the first The local loss value at the end of the local training round; Indicates the first The client in the first The local loss value at the end of the local training round.

[0021] (3) To and Normalization was performed separately to obtain and .

[0022] (4) Calculate the final robustness weights This is a linear combination of the two normalized weights mentioned above:

[0023] (2),

[0024] in, is a hyperparameter between 0 and 1, used to balance the importance of updating similarity and loss change in weight evaluation.

[0025] Edge servers based on the robustness weights Data volume of each client Weighted aggregation is performed to update the global pre-trained model:

[0026] (3),

[0027] Repeat the above process, when the federal training rounds Reach the preset maximum number of training rounds Training ends when the time is right, and the final global pre-trained model is obtained. . save All global pre-trained models obtained during round training .

[0028] Step 2: Each client uses a pre-trained model to map its local data to the latent space. Based on the NCFD (Neural Characteristic Function Discrepancy) metric, a minimization optimization framework is used to compress the data distribution. Simultaneously, output matching loss is introduced to enhance the semantic consistency and personalization adaptation of the distilled data. Distilled data is then generated and uploaded to the edge server. The specific process is as follows:

[0029] Each client builds a feature extractor based on a federated pre-trained model. Local raw multi-source heterogeneous data Feature representations are obtained by mapping to the latent space. ,in This represents the original data volume.

[0030] Client initializes small distillation dataset ,in This represents the amount of data after distillation. The initialization uses a noise initialization method: starting from a Gaussian distribution. Random sampling generates shapes as The initial synthetic data tensor, and constrain the pixel values ​​to Within the interval. Based on this, a distribution matching objective is constructed for the minimization optimization framework:

[0031] (4),

[0032] in, The neural feature function difference (NCFD) is defined as follows:

[0033] (5),

[0034] The output matching loss is defined as the KL divergence between the real data and the distilled data on the pre-trained model output distribution:

[0035] (6),

[0036] in, Indicates from the original dataset Real data samples were collected; Indicates the data from the distillation dataset Sampling of synthetic data samples; Sample network parameters for optimizing frequency parameters. The sampling distribution; It is a characteristic difference function; Frequency parameters The sampling distribution; The weight coefficients for outputting the matching loss have a range of values. .

[0037] The client uses an auxiliary sampling network. Dynamically optimize frequency parameters The sampling strategy aims to maximize the total loss function while optimizing the distillation dataset. To minimize the total loss function, an alternating optimization process is used to achieve the desired result from the original dataset. To distillation dataset The distributed compression optimization process includes:

[0038] (1) Fixed distillation dataset Optimize sampling network To maximize total loss:

[0039] (7),

[0040] (2) Fixed sampling network Optimize the distillation dataset To minimize the total loss:

[0041] (8),

[0042] Aligning the amplitude and phase information of neural features in the complex plane, the characteristic function is decomposed as follows:

[0043] (9),

[0044] By adjusting the weighting coefficients Balanced amplitude term Diversity of control and phase terms To ensure authenticity, the distilled data is combined with output matching loss to ensure that the semantic space of the distilled data remains consistent with that of the original data.

[0045] The optimized small distillation dataset Uploaded to an edge server, completing the compression process from raw large-scale data to high-fidelity small-scale distillation data.

[0046] Step Two: In the adaptive noise backdoor defense processing stage based on multi-feature fusion, the edge server performs comprehensive security detection and purification on the aggregated distillation dataset. By fusing five feature dimensions—edge, contrast, texture, anomaly, and gradient—a comprehensive suspicion score is calculated to accurately locate potential backdoor triggering areas. Based on this, an intelligent clustering algorithm is used to optimize interference locations, and adaptive multi-type noise combination interference is applied. This effectively disrupts backdoor triggering patterns while preserving the original image quality and useful information to the maximum extent. The secure distillation dataset constructed in this stage provides reliable data support for subsequent model fine-tuning.

[0047] Preferably, step two mainly includes two steps:

[0048] Step 1: After aggregating and distilling the data, the edge server calculates a comprehensive suspicion score by fusing five features: edge, contrast, texture, anomaly, and gradient. An adaptive threshold is then used to locate potential backdoor triggering areas. The specific process is as follows:

[0049] The edge server performs multi-dimensional feature extraction and fusion on the aggregated distillation data image to extract edge intensity features. Contrast characteristics Texture complexity features Outlier characteristics and gradient features After normalization, the results are linearly fused according to preset weights to generate a comprehensive suspicion score. :

[0050] (10)

[0051] The weighting coefficients are respectively , , , , , , , , , These are the normalized values ​​for the corresponding features.

[0052] Edge servers perform spatial clustering optimization on suspicious areas, using Euclidean distance as a basis. Clustering algorithms merge adjacent regions, where and For the coordinates of the region center, when Clustering and merging are performed at the pixel level, calculating the center coordinates, average suspicious score, and bounding box of each cluster, and then based on the maximum number of interference regions. and maximum repair area Filtering pixels, where settings and This is based on the fact that backdoor triggers are usually concealed and do not occupy too many pixels, while ensuring the protection of the original image quality.

[0053] Edge servers receive optimized information on suspicious areas. ,in The coordinates of the region center For the average suspicious score, This provides bounding box information, enabling precise localization for subsequent noise interference processing.

[0054] Step 2: The edge server performs clustering optimization on the identified suspicious regions and applies adaptive multi-type noise combination interference. While effectively disrupting the backdoor triggering mode, it preserves the original image quality to the maximum extent. Finally, a secure distillation dataset is constructed and uploaded. The specific process is as follows:

[0055] Edge servers based on suspicious area information Select an adaptive repair strategy; for each suspicious area... The server based on suspicious scores The dimensions of the repair area are calculated using the following formula:

[0056] (11),

[0057] The server selects the repair method based on the configuration parameters: when configured as local_mean, it uses the local mean filling method; otherwise, it uses the global mean filling method to ensure that suspicious areas receive appropriate repair processing.

[0058] The edge server applies local noise interference to the suspicious area, generating Gaussian noise. In the repaired area Internal injection, where noise intensity is based on suspicion score To perform adaptive adjustments, the following formula is used for calculation:

[0059] (12),

[0060] The baseline parameters set for the edge server are as follows: , .

[0061] The edge server applies global noise perturbation to the entire image, generating global Gaussian noise. It is applied to the entire image, where the global noise intensity is based on the overall suspiciousness score of the image. To perform adaptive adjustments, the following formula is used for calculation:

[0062] (13)

[0063] Through the coordinated interference of the aforementioned local and global noise, the server ensures that the backdoor triggering mode is effectively disrupted.

[0064] Edge servers perform image quality protection by limiting the processed pixel values ​​to a certain range. Within the specified range, constraints are applied using the following formula:

[0065] (14)

[0066] Through this constraint mechanism, the server ensures that the image quality is always maintained within an effective range.

[0067] Building a secure distillation dataset on edge servers ,in This represents an image that has undergone noise reduction processing. The original labels are displayed; the server performs integrity verification on the dataset to ensure that all images have undergone defense processing, and finally uploads the securely distilled dataset to the cloud server for subsequent federated learning tasks.

[0068] Among them, the noise parameters set by the edge server , Based on experimental analysis of the noise sensitivity of the backdoor trigger, the repair method configuration is set according to the actual application requirements, so as to maintain image quality to the maximum extent while ensuring effective defense.

[0069] Step 3: In the large-scale model fine-tuning and backdoor defense effectiveness verification phase, the cloud server uses the security distillation dataset that has undergone defense processing to fine-tune the large model. During the fine-tuning process, an efficient LoRA-based fine-tuning method is adopted while maintaining the integrity of the defense mechanism, achieving synergistic optimization of model performance and security. After fine-tuning, a multi-dimensional testing and evaluation framework is used to comprehensively verify the model's performance on the original task and its ability to defend against backdoor attacks, ultimately outputting a deployable model with high security and reliability.

[0070] Preferably, step three mainly includes two steps:

[0071] Step 1: The cloud server uses the defense dataset to fine-tune the large model, maintaining the integrity of the defense mechanism and achieving collaborative optimization between the model and the defense backdoor. The specific process is as follows:

[0072] The cloud server loads the received safe distillation dataset. and initialize the pre-trained large model. Its parameters are denoted as .

[0073] The cloud server performs large-scale model fine-tuning based on LoRA, learning LoRA parameters through the following optimization objective function. :

[0074] (15)

[0075] in, To fine-tune the loss function; Represents cross-entropy loss; Indicates based on pre-trained parameters and LoRA parameters The model output; Images in the safe distillation dataset , For the corresponding tag.

[0076] The following hyperparameters and configurations were used during fine-tuning: learning rate Single device batch size The gradient accumulation step count is 16; the number of training rounds is... LoRA rank LoRA scaling parameters The target module is all linear layers; the maximum sequence length is 2048; the data type is bfloat16; and the optimizer is Adam.

[0077] During training, evaluation and verification are performed every 50 steps, and checkpoints are saved (up to 2) to ensure stable convergence of the fine-tuning process.

[0078] After the cloud server completed fine-tuning, the optimized LoRA parameters were obtained. And output the finely tuned large model. Its parameters are and The combination. Save. The output directory yields a reliable large model that has been cleaned with a secure distillation dataset and is free of backdoor threats.

[0079] Step 2: The cloud server evaluates the model's performance and defense effectiveness through multi-dimensional testing, and outputs a complete verification report and a deployable model. The specific process is as follows:

[0080] The cloud server is used to evaluate model performance using a benign test sample set. Calculate the accuracy of the model on the original task. .

[0081] in, The number of samples that are correctly predicted. This represents the total number of test samples.

[0082] The cloud server was used to evaluate the effectiveness of backdoor defense by injecting various types of malicious triggers into test samples to simulate backdoor attack scenarios and calculating the attack success rate. .

[0083] in, The number of samples that successfully triggered malicious behavior. The total number of attack samples is used to quantitatively verify the model's ability to suppress backdoor attacks.

[0084] The cloud server undergoes a comprehensive security index assessment, incorporating the Defense Effectiveness Ratio (DER) metric.

[0085] (16)

[0086] in, The percentage decrease in benign accuracy compared to an undefended model under the same attack method. The DER index represents the decrease in the model's success rate against backdoor attacks after the introduction of a backdoor defense mechanism compared to when no defense mechanism is used; a larger DER value indicates that the model has a stronger backdoor defense capability while maintaining high performance.

[0087] The cloud server integrates all test results, generates a final evaluation report, and outputs a deployable model. The report includes a detailed analysis of key indicators such as BA, ASR, and DER, as well as a comparative evaluation with other mainstream defense methods. Based on the comprehensive evaluation results, the cloud server determines the optimal model configuration and outputs a final large model with high security and reliability for actual deployment.

[0088] The beneficial effects of this invention are:

[0089] (1) This invention proposes a privacy-preserving federated distillation and backdoor defense method for large model fine-tuning. By using dataset distillation based on neural feature function matching and adaptive noise defense through multi-feature fusion, it effectively solves multiple challenges faced by large model fine-tuning under the federated learning framework, including data privacy, communication overhead, model generalization, and backdoor security. This method significantly reduces the communication and computation burden of distributed training while strictly protecting the privacy of the client's original data. At the same time, it greatly improves the generalization performance of the model on specific downstream tasks and the robustness of backdoor defense, which is of great significance for promoting the safe and efficient deployment and application of large models in privacy-sensitive scenarios.

[0090] (2) The dataset distillation mechanism adopted in this invention achieves distributed compression from large-scale non-independent identically distributed original data to high-fidelity small datasets through neural feature function difference measurement and minimization optimization framework. This process not only preserves the core representation and diversity of the data, effectively overcoming the adverse effects of data heterogeneity on the model's generalization ability, but also fundamentally reduces communication overhead by uploading a very small scale of distilled data instead of the original data or large model parameters, providing a feasible technical path for federated fine-tuning of large models in resource-constrained edge environments.

[0091] (3) The adaptive noise backdoor defense mechanism based on multi-feature fusion designed in this invention innovatively integrates five major features: edge, contrast, texture, anomaly, and gradient, to achieve accurate localization and intelligent interference of potential backdoor triggering areas. By applying local and global noise in a coordinated manner, this mechanism can effectively disrupt backdoor triggering patterns and significantly suppress attack success rates, while adaptively adjusting the noise intensity to maximize the preservation of the original image quality and useful information. This achieves an excellent balance between defense effectiveness and model usability, providing a reliable guarantee for the secure training of large models in open distributed environments. Attached Figure Description

[0092] Figure 1 This is a flowchart of a privacy-preserving federated distillation and backdoor defense method for fine-tuning large models proposed in this invention;

[0093] Figure 2 This is a framework diagram of a privacy-preserving federated distillation and backdoor defense method for fine-tuning large models proposed in this invention.

[0094] Figure 3 Comparison of distillation results for datasets;

[0095] Figure 4 Comparison of backdoor removal effects. Detailed Implementation

[0096] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0097] This embodiment presents a privacy-preserving federated distillation and backdoor defense method for fine-tuning large models, such as... Figure 1 As shown, it specifically includes:

[0098] Step 1-1: Initialize the global pre-trained model parameters on the edge server and coordinate Each client participates in federated learning training.

[0099] In the During each round of federated learning, the edge server will display the current global model. Distribute to each client.

[0100] Each client Based on local multi-source heterogeneous data The received global model is trained locally, and the target is optimized. Get local model update .

[0101] Furthermore, the edge server calculates the robustness weights for each client model update. The calculation method is a weighted combination of two evaluation results: updated similarity and loss change. The specific steps are as follows:

[0102] (1) Calculate the weights based on the updated similarity Computing Client Model update The mean cosine similarity with all other client model updates, i.e. .

[0103] (2) Calculate the weights based on the change in loss. According to the client The weights are calculated based on the changes in the local training loss; the more significant the decrease in loss, the higher the weight. Specifically:

[0104] (1),

[0105] in, This is a hyperparameter greater than zero, used to adjust the sensitivity to changes in the magnitude of the loss. Indicates the first The client in the first The local loss value at the end of the local training round; Indicates the first The client in the first The local loss value at the end of the local training round.

[0106] (3) To and Normalization was performed separately to obtain and .

[0107] (4) Calculate the final robustness weights This is a linear combination of the two normalized weights mentioned above:

[0108] (2),

[0109] in, is a hyperparameter between 0 and 1, used to balance the importance of updating similarity and loss change in weight evaluation.

[0110] Furthermore, the edge server is based on the robustness weights. Data volume of each client Weighted aggregation is performed to update the global pre-trained model:

[0111] (3),

[0112] Repeat the above process for each federal training round. Reach the preset maximum number of training rounds Training ends when the time is right, and the final global pre-trained model is obtained. . save All global pre-trained models obtained during round training .

[0113] Steps 1-2: Each client builds a feature extractor based on the federated pre-trained model. Local raw multi-source heterogeneous data Feature representations are obtained by mapping to the latent space. ,in This represents the original data volume.

[0114] Client initializes small distillation dataset ,in This represents the amount of data after distillation. The initialization uses a noise initialization method: starting from a Gaussian distribution. Random sampling generates shapes as The initial synthetic data tensor, and constrain the pixel values ​​to Within the interval. Based on this, a distribution matching objective is constructed for the minimization optimization framework:

[0115] (4),

[0116] in, The neural feature function difference (NCFD) is defined as follows:

[0117] (5),

[0118] The output matching loss is defined as the KL divergence between the real data and the distilled data on the pre-trained model output distribution:

[0119] (6),

[0120] in, Indicates from the original dataset Real data samples were collected; Indicates the data from the distillation dataset Sampling of synthetic data samples; Sample network parameters for optimizing frequency parameters. The sampling distribution; It is a characteristic difference function; Frequency parameters The sampling distribution; The weight coefficients for outputting the matching loss have a range of values. .

[0121] Furthermore, the client uses an auxiliary sampling network. Dynamically optimize frequency parameters The sampling strategy aims to maximize the total loss function while optimizing the distillation dataset. To minimize the total loss function, an alternating optimization process is used to achieve the optimization of the original big data. To distillation dataset The distributed compression optimization process includes:

[0122] (1) Fixed distillation dataset Optimize sampling network To maximize total loss:

[0123] (7),

[0124] (2) Fixed sampling network Optimize the distillation dataset To minimize the total loss:

[0125] (8),

[0126] Aligning the amplitude and phase information of neural features in the complex plane, the characteristic function is decomposed as follows:

[0127] (9),

[0128] By adjusting the weighting coefficients Balanced amplitude term Diversity of control and phase terms To ensure authenticity, the distilled data is combined with output matching loss to ensure that the semantic space of the distilled data remains consistent with that of the original data.

[0129] The optimized small distillation dataset Uploaded to an edge server, completing the compression process from raw large-scale data to high-fidelity small-scale distillation data.

[0130] Step 2-1: The edge server performs multi-dimensional feature extraction and fusion on the aggregated distillation data image, extracting edge intensity features. Contrast characteristics Texture complexity features Outlier characteristics and gradient features After normalization, the results are linearly fused according to preset weights to generate a comprehensive suspicion score. :

[0131] (10)

[0132] The weighting coefficients are respectively , , , , , , , , , These are the normalized values ​​for the corresponding features.

[0133] Furthermore, the edge server performs spatial clustering optimization on suspicious areas, using a method based on Euclidean distance. Clustering algorithms merge adjacent regions, where and For the coordinates of the region center, when Clustering and merging are performed at the pixel level, calculating the center coordinates, average suspicious score, and bounding box of each cluster, and then based on the maximum number of interference regions. and maximum repair area Filtering pixels, where settings and This is based on the fact that backdoor triggers are usually concealed and do not occupy too many pixels, while ensuring the protection of the original image quality.

[0134] Edge servers receive optimized information on suspicious areas. ,in The coordinates of the region center For the average suspicious score, This provides bounding box information, enabling precise localization for subsequent noise interference processing.

[0135] Step 2-2: The edge server bases its actions on suspicious area information. Select an adaptive repair strategy; for each suspicious area... The server based on suspicious scores The dimensions of the repair area are calculated using the following formula:

[0136] (11),

[0137] The server selects the repair method based on the configuration parameters: when configured as local_mean, it uses the local mean filling method; otherwise, it uses the global mean filling method to ensure that suspicious areas receive appropriate repair processing.

[0138] Furthermore, the edge server applies local noise interference to the suspicious area, generating Gaussian noise. In the repaired area Internal injection, where noise intensity is based on suspicion score To perform adaptive adjustments, the following formula is used for calculation:

[0139] (12),

[0140] The baseline parameters set for the edge server are as follows: , .

[0141] The edge server applies global noise perturbation to the entire image, generating global Gaussian noise. It is applied to the entire image, where the global noise intensity is based on the overall suspiciousness score of the image. To perform adaptive adjustments, the following formula is used for calculation:

[0142] (13)

[0143] Through the coordinated interference of the aforementioned local and global noise, the server ensures that the backdoor triggering mode is effectively disrupted.

[0144] Furthermore, the edge server performs image quality protection by limiting the processed pixel values ​​to a certain range. Within the specified range, constraints are applied using the following formula:

[0145] (14)

[0146] Through this constraint mechanism, the server ensures that the image quality is always maintained within an effective range.

[0147] Building a secure distillation dataset on edge servers ,in This represents an image that has undergone noise reduction processing. The original labels are displayed; the server performs integrity verification on the dataset to ensure that all images have undergone defense processing, and finally uploads the securely distilled dataset to the cloud server for subsequent federated learning tasks.

[0148] Among them, the noise parameters set by the edge server , Based on experimental analysis of the noise sensitivity of the backdoor trigger, the repair method configuration is set according to the actual application requirements, so as to maintain image quality to the maximum extent while ensuring effective defense.

[0149] Step 3-1: The cloud server loads the received safe distillation dataset. and initialize the pre-trained large model. Its parameters are denoted as .

[0150] The cloud server performs large-scale model fine-tuning based on LoRA, learning LoRA parameters through the following optimization objective function. :

[0151] (15)

[0152] in, To fine-tune the loss function; Represents cross-entropy loss; Indicates based on pre-trained parameters and LoRA parameters The model output; Images in the safe distillation dataset , For the corresponding tag.

[0153] Furthermore, the following hyperparameters and configurations were used during the fine-tuning process: learning rate Single device batch size The gradient accumulation step count is 16; the number of training rounds is... LoRA rank LoRA scaling parameters The target module is all linear layers; the maximum sequence length is 2048; the data type is bfloat16; and the optimizer is Adam.

[0154] During training, evaluation and verification are performed every 50 steps, and checkpoints are saved (up to 2) to ensure stable convergence of the fine-tuning process.

[0155] After the cloud server completed fine-tuning, the optimized LoRA parameters were obtained. And output the finely tuned large model. Its parameters are and The combination. Save. The output directory yields a reliable large model that has been cleaned with a secure distillation dataset and is free of backdoor threats.

[0156] Step 3-2: The cloud server performs model performance evaluation using a benign test sample set. Calculate the accuracy of the model on the original task. .

[0157] in, The number of samples that are correctly predicted. This represents the total number of test samples.

[0158] Furthermore, the cloud server performs a backdoor defense effectiveness evaluation by injecting various types of malicious triggers into test samples to simulate backdoor attack scenarios and calculating the attack success rate. .

[0159] in, The number of samples that successfully triggered malicious behavior. The total number of attack samples is used to quantitatively verify the model's ability to suppress backdoor attacks.

[0160] Furthermore, the cloud server undergoes a comprehensive security index assessment, introducing the Defense Effectiveness Ratio (DER) metric:

[0161] (16)

[0162] in, The percentage decrease in benign accuracy compared to an undefended model under the same attack method. The DER index represents the decrease in the model's success rate against backdoor attacks after the introduction of a backdoor defense mechanism compared to when no defense mechanism is used; a larger DER value indicates that the model has a stronger backdoor defense capability while maintaining high performance.

[0163] Furthermore, the cloud server integrates all test results, generates a final evaluation report, and outputs a deployable model. The report includes a detailed analysis of key indicators such as BA, ASR, and DER, as well as a comparative evaluation with other mainstream defense methods. Based on the comprehensive evaluation results, the cloud server determines the optimal model configuration and outputs a final large model with high security and reliability for actual deployment.

[0164] To verify the effectiveness of the proposed method, a series of experiments were designed and executed, and the evaluation was carried out mainly from the following three aspects: (1) privacy protection effect of dataset distillation; (2) ability of adaptive backdoor defense processing to eliminate backdoor triggers; (3) comprehensive comparison of the performance and security of the final fine-tuning model.

[0165] 1. Validation of the privacy protection effect of dataset distillation:

[0166] This experiment aims to verify that the dataset distillation process can generate new images visually different from the original data while preserving the core semantic features of the original images, thereby achieving effective privacy protection. This invention was tested on a set of real-world image datasets containing multiple categories.

[0167] like Figure 3As shown, the distillation dataset obtained after the first stage of processing in this invention can visually clearly represent the category information of the original image (e.g., cat, dog, car, etc.), and human observers can easily recognize its semantic content. However, at the pixel level, the distilled images differ significantly from the original images and do not possess one-to-one reproducibility. This characteristic ensures that the original user's private data is not directly leaked, while retaining key features for model training in the distilled data, achieving a balance between privacy protection and data utility.

[0168] 2. Verification of the effectiveness of adaptive backdoor defense:

[0169] This experiment aims to demonstrate the effectiveness of the second stage of this invention—adaptive noise backdoor defense processing based on multi-feature fusion—in eliminating common backdoor triggers (taking BadNet as an example). This invention simulates a malicious client, injecting a backdoor trigger with a specific pattern into locally distilled data, and then observes the changes before and after the defense processing.

[0170] Experimental results show that the multi-feature fusion detection and adaptive noise interference mechanism employed in this invention can accurately locate and effectively disrupt the BadNet backdoor triggers embedded in the image. After processing, the visual saliency of the trigger pattern is significantly reduced, and its backdoor activation behavior is effectively suppressed. Simultaneously, because the noise is adaptively and locally applied, the overall visual quality and useful information of the image are preserved to the maximum extent, demonstrating the superior balance between disrupting backdoor patterns and maintaining data quality achieved by this method.

[0171] 3. Comprehensive evaluation of model performance and security:

[0172] To comprehensively evaluate the impact of the overall process of this invention on the performance and security of the final large model, this invention performs five rounds of LoRA fine-tuning on the deepseek-ai / Janus-Pro-1B large model in the cloud using different training datasets, and compares its performance metrics. This invention sets up two baseline methods for comparison with its own method:

[0173] 1. Baseline Method 1: Fine-tuning using a clean distillation dataset (ideal case).

[0174] 2. Baseline Method 2: Fine-tuned using a distillation dataset contaminated with multiple BadNet backdoors (without defense).

[0175] 3. The method of this invention: fine-tuning is performed using a safe distillation dataset that has been contaminated by the same BadNet backdoor but has been processed by the second stage of this invention.

[0176] Evaluation metrics include Benign Accuracy (BA, %), Attack Success Rate (ASR, %), and Defense Effectiveness Ratio (DER).

[0177] Table 1

[0178]

[0179] The experimental results in Table 1 clearly show that, compared with the model trained using clean data, the benign accuracy (BA) of the final model using the method of this invention decreases only negligibly (approximately 0.7% on average), indicating that the defensive processing of this invention has minimal impact on the model's performance on the original task and maintains excellent generalization ability. Compared with the model trained directly using backdoor-contaminated data, the method of this invention can significantly reduce the backdoor attack success rate (ASR)—under various backdoor trigger attacks, the ASR is suppressed from nearly 100% to an average of approximately 4.3%, a reduction of over 90 percentage points. Further comprehensive evaluation using the defense effectiveness ratio (DER) index shows that the method of this invention performs excellently in various attack scenarios, averaging approximately 1.35, and particularly in the "random cross" attack, the DER value reaches as high as 1.525, fully demonstrating that the method of this invention possesses strong backdoor defense capabilities while maintaining high performance, achieving an excellent balance between defense effectiveness and model performance. In summary, this invention successfully achieves a balance between high performance and high security during the fine-tuning of large models, fundamentally curbing the threat of backdoor attacks without significantly compromising the usefulness of the model.

[0180] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A privacy protection federated distillation and backdoor defense method for large model fine-tuning, characterized in that: The method comprises the following steps: Step one, federated pre-training based on contribution weight, local neural feature function matching data set distillation and core representation extraction; Step one comprises the following steps: Step 1-1, the edge server and the client train the global pre-training model through federated learning, and aggregate the data through a dynamic weighting aggregation strategy based on the client update quality evaluation; Step 1-1 comprises the following steps: Step 1-1-1: Edge server initializes global pre-trained model parameters and coordinates federated learning training with K clients participating; Step 1-1-2: In the first round of federated learning process, the edge server distributes the current global model to each client In the federated learning process, the edge server distributes the current global model to each client ;​ Step 1-1-3: Each client Based on local multi-source heterogeneous data To the received global model Local training by optimizing the target Get local model update ; wherein, represents the global model parameter of the first round of federated learning; represents the global model parameter issued by the edge server at the beginning of the first round of federated learning; represents the local updated model parameter obtained by the first client after the first round of local training; represents the local updated model parameter obtained by the first client after the first round of local training; represents the local loss function defined by the first client based on its local data set Step 1-1-4: The edge server calculates the robustness weight of each client model update The calculation method is a weighted combination of two evaluation results based on update similarity and loss change, and the specific steps are as follows: (1) Compute weight based on update similarity : Compute client 's model update 's average cosine similarity with all other client model updates , i.e. ; (2) Calculate the weight based on the loss change : Calculate the weight according to the local training loss change of the client , the more significant the loss drop, the higher the weight, specifically: （1）， in, It is a hyperparameter greater than zero; Indicates the first The client in the first The local loss value at the end of the local training round; Indicates the first The client in the first The local loss value at the end of the local training round; (3) the weight of the similarity and the weight of the loss change are normalized respectively to obtain the normalized weight of the similarity and the normalized weight of the loss change ; (4) calculating the final robustness weight a normalized weight for similarity and a normalized weight for loss change a linear combination of: （2）， wherein, is a hyperparameter between 0 and 1; Step 1-1-5: Edge server updates global pre-trained model based on robustness weight With each client data volume , weighted aggregation is performed to update the global pre-trained model: （3）， wherein, is a hyper-parameter between 0 and 1; denotes the overall robustness weight of the th client in the th round, which is obtained by weighting fusion after normalization based on the update similarity and loss change calculation in step 1-1-4; denotes the overall robustness weight of the th client in the th round, which is obtained by weighting fusion after normalization based on the update similarity and loss change calculation in step 1-1-4; is a normalization denominator; Step 1-1-6: Repeat the execution of Step 1-1-2 to Step 1-1-5, and end the training when the federal training epoch reaches a preset maximum training epoch, to obtain a final global pre-training model ​​​​​ Step 1-2: Each client maps the local data to the hidden space using the pre-training model, compresses the data distribution based on the NCFD metric through the maximum and minimum optimization framework, generates distillation data and uploads it to the edge server; Step two, adaptive noise backdoor defense processing based on multi-feature fusion; Step three, large model fine-tuning and backdoor defense effect verification.

2. The method of claim 1, wherein, The step two comprises the following steps: Step 2-1: After aggregating the distillation data, the edge server calculates the comprehensive suspicious score by fusing the edge, contrast, texture, anomaly and gradient five features, and locates the potential backdoor trigger area with an adaptive threshold; Step 2-2: The edge server clusters and optimizes the identified suspicious areas, applies adaptive multi-type noise combination interference, and finally constructs a secure distillation data set for uploading.

3. The method of claim 1, wherein, The step two comprises the following steps: Step 3-1: The cloud server loads the secure distillation data set and fine-tunes the pre-training large model based on the parameter fine-tuning technology; Step 3-2: The cloud server evaluates the model performance and defense effect through multi-dimensional testing, outputs a complete verification report and a deployable model.

4. The method of claim 1, wherein: Step 1-2 specifically comprises the following steps: Step 1-2-1: Each client builds a feature extractor based on the federated pre-trained model , maps the local raw multi-source heterogeneous data to the latent space to get the feature representation , where is the amount of raw data; where, represents the th real input sample in the raw data set ; Step 1-2-2: Client initializes small distillation dataset ; wherein, is the data volume after distillation; denotes a distilled data set of the th synthesized sample in the data set Initialization with noise initialization: from Gaussian distribution Random sampling generates initial synthetic data tensors with shape and pixel values are constrained in interval, based on which the distribution matching objective of the maximum-minimum optimization framework is constructed: （4）， wherein, is the neural characteristic function difference, NCFD, defined as: （5）， To output match loss, define as the KL divergence of the real data and the distilled data on the pre-training model output distribution: （6）， wherein, represents real data samples sampled from the original dataset ; represents synthetic data samples sampled from the distilled dataset ; is a sampling network parameter for optimizing the sampling distribution of the frequency parameter ; is a feature difference function; is a sampling distribution of the frequency parameter ; is a weight coefficient of the output matching loss, and the value range is ; Steps 1-2-3: The client uses the auxiliary sampling network Dynamically optimize frequency parameters The sampling strategy aims to maximize the total loss function while optimizing the distillation dataset. To minimize the total loss function, an alternating optimization process is used to achieve the desired result from the original dataset. To distillation dataset The distributed compression optimization process includes: (1) Fixed distillation dataset , optimizing the sampling network to maximize the total loss: （7）， (2) Fixed sampling network , optimizing distillation dataset to minimize total loss: （8）， Step 1-2-4: Align the amplitude and phase information of the neural features in the complex plane, and the feature function is decomposed as: （9）， By adjusting the weight coefficient Balancing amplitude terms Control of diversity and phase terms The authenticity of the guarantee, while combining the output matching loss makes the distilled data consistent with the original data in the semantic space; Step 1-2-5: Upload the optimized distillation dataset to the edge server, complete the compression process from the original large-scale data to the high-fidelity small distillation data. ​ 5. The method of claim 2, wherein: Step 2-1 specifically comprises the following steps: Step 2-1-1: The edge server performs multi-dimensional feature extraction and fusion on the aggregated distilled data image, extracts edge strength features , contrast features , texture complexity features , outlier features , and gradient features , and after normalization, linearly fuses them according to the preset weights to generate a comprehensive suspiciousness score : （10）， wherein, , , , , are weight coefficients, , , , , are normalized values of the corresponding features, respectively; Step 2-1-2: Spatial clustering optimization is performed on suspicious regions by the edge server, using a clustering algorithm based on Euclidean distance to merge adjacent regions, where and are the region center coordinates, when pixels, the center coordinates, average suspicious score and bounding box of each cluster are calculated, and the maximum number of interference regions and the maximum repair area pixels are screened; Step 2-1-3: The edge server gets the optimized suspicious region information wherein is the region center coordinate, is the average suspicious score, is the bounding box information.

6. The method of claim 5, wherein: Step 2-2 specifically comprises the following steps: Step 2-2-1: The edge server calculates suspicious region information according to the suspicious region information selects an adaptive repair strategy; for each suspicious region , the server calculates a suspicious score according to the suspicious region information calculates the repair region size, specifically using the following formula: （11）， The server selects a repair method according to the configuration parameters: when the configuration is local_mean, the local mean padding method is used; otherwise, the global mean padding method is used; Step 2-2-2: The edge server applies local noise jamming to the suspicious region, generating Gaussian noise , and injects into the repaired region , where the noise intensity is based on the suspicious score Adaptive adjustment is made, specifically calculated as follows: （12）， wherein, and a reference parameter set for the edge server; Step 2-2-3: The edge server applies global noise disturbance to the whole image, generating global Gaussian noise and applied to the whole image, where the global noise intensity is based on the image overall suspiciousness score Adaptive adjustment is made, specifically calculated using the following formula: （13）； Step 2-2-4: The edge server performs image quality protection by limiting the processed pixel values to the range [0, 255] using the following equation: Y = min(max(Y, 0), 255) （14）； Step 2-2-5: Edge server builds secure distillation dataset wherein denotes the image after noise interference processing, denotes the original label; wherein the noise parameter is set by the edge server , Based on the experimental analysis of the sensitivity of the backdoor trigger to noise.

7. The method of claim 3, wherein: Step 3-1 specifically comprises the following steps: Step 3-1-1: The cloud server loads the received secure distilled dataset and initializes the pre-trained large model with parameters denoted by ; Step 3-1-2: The cloud server performs LoRA-based large model fine-tuning to learn LoRA parameters by optimizing the following objective function : （15）， wherein, is a fine-tuned loss function; denotes a cross-entropy loss; denotes a model output based on pre-trained parameters and LoRA parameters ; is an image in a safety distillation dataset , is a corresponding label; Step 3-1-3: After the cloud server completes fine-tuning, the optimized LoRA parameters are obtained , and the fine-tuned large model is output , with parameters and combined, saved to the output directory, obtaining a reliable large model that has been purified by security distillation dataset and does not contain backdoor threats.

8. The method of claim 7, wherein: Step 3-2 specifically comprises the following steps: Step 3-2-1: The cloud server performs model performance evaluation using the benign test sample set Accuracy of the computational model on the original task ; wherein, the number of samples for correct prediction, is the total number of test samples; Step 3-2-2: The cloud server performs backdoor defense effect evaluation, injects various types of malicious triggers into the test sample, simulates a backdoor attack scene, and calculates the attack success rate ; wherein, the number of samples that successfully triggered malicious behavior, is the total number of attack samples; Step 3-2-3: The cloud server performs comprehensive security index evaluation and introduces the defense effectiveness ratio DER index: （16）， wherein, is the decrease of benign accuracy compared with the model without defense under the same attack mode, is the decrease of the success rate of the model against backdoor attacks after introducing the backdoor defense mechanism compared with the model without defense; the greater the DER index value is, the stronger the backdoor defense capability of the model is while maintaining high performance. Step 3-2-4: The cloud server generates a final evaluation report and outputs a deployable model based on all test results; the cloud server determines the optimal model configuration based on the comprehensive evaluation results and outputs the final large model for actual deployment.

Images