Generalization-enhanced sparse federated distillation and security defense fine-tuning method for large models

Abstract

This invention relates to a generalization-enhanced sparse federated distillation and security defense fine-tuning method for large models, belonging to the field of artificial intelligence security and federated learning. The method includes: Step 1, the client partitions non-independent and identically distributed data based on a Dirichlet distribution, performs generalization confidence evaluation and local privacy-preserving synthetic distillation, and encrypts and uploads hierarchical scores, sparse masks, distilled data, and contribution weights; Step 2, the edge server aggregates data and masks according to contribution weights, divides feature regions based on a generalization saliency heatmap, performs dual-path structured backdoor defense, and outputs secure distilled data, a global sparse mask, and feature rules after amplitude-limited fidelity calibration; Step 3, the cloud server deploys a structured LoRA adapter to complete lightweight fine-tuning, and outputs an optimized large model after evaluation. This invention simultaneously achieves privacy protection, data adaptation, overhead reduction, backdoor resistance, and efficient fine-tuning without requiring the original data to leave its domain, thereby improving model generalization, privacy security, and defense robustness.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security and federated learning technology, specifically involving a generalization enhancement sparse federated distillation and security defense fine-tuning method for large models. Background Technology

[0002] In recent years, multimodal large models have become a core support for the practical application of artificial intelligence due to their superior performance in tasks such as natural language understanding, computer vision, and cross-modal interaction. Lightweight fine-tuning of large models for vertical fields such as healthcare, finance, and industry is a key step in achieving the engineering deployment of models. In these scenarios, various terminal clients hold a large amount of high-value private domain data. Constrained by privacy regulations and data security, the original data cannot be centrally aggregated, and data silos and privacy compliance issues have become the core obstacles to the implementation of large model fine-tuning. Federated learning, as a distributed privacy-preserving computing paradigm, can complete collaborative model optimization without data leaving its domain. Combined with technologies such as federated knowledge distillation, model sparsity, backdoor defense, and LoRA lightweight fine-tuning, it has become a mainstream technical approach that balances privacy, performance, and efficiency, providing a basic framework for the distributed and secure fine-tuning of large models.

[0003] However, directly applying traditional federated learning with existing distillation, defense, and lightweight techniques to large-scale model fine-tuning scenarios results in numerous irreconcilable technical drawbacks, failing to meet practical deployment requirements. In federated scenarios, client data typically exhibits non-identical distribution (Non-IID) characteristics, with uneven class distribution and significant differences in sample size. Existing distillation methods, such as traditional federated distillation and dataset distillation, cannot fully address these limitations. Distillation is not optimized for data heterogeneity and lacks a generalization value quantification mechanism, resulting in insufficient knowledge retention and poor generalization of synthesized data. Large models have billions to hundreds of billions of parameters, and the participation of all parameters in federated communication will generate huge overhead. Existing sparsity methods such as structured pruning and dynamic masking only achieve single-layer parameter selection and do not construct a three-level importance assessment of layers-neurons-attention heads, resulting in limited lightweight effects. Local distillation data generation relies on synthesis methods such as GANs and diffusion models, making it difficult to balance privacy protection and knowledge fidelity, and the risk of original data leaving the country cannot be completely avoided. The federated distributed architecture is vulnerable to malicious clients implanting hidden backdoors and trigger-based backdoors. These backdoors are highly concealed, and the global model is easily manipulated. Existing backdoor defenses such as model update anomaly detection, differential privacy noise addition, and feature cleaning do not combine generalization distribution for precise processing, which can easily destroy the original features. At the same time, the computational cost of fine-tuning all parameters of large models is extremely high, and conventional LoRA fine-tuning is not linked with sparse masking, making it impossible to achieve a combination of security and efficiency.

[0004] To alleviate these issues, the industry has proposed a series of targeted technologies. Regarding privacy and communication, federated aggregation algorithms such as FedAvg and FedProx are used for distributed model updates; federated distillation methods such as FedKD and FedDF reduce data transmission volume by uploading knowledge logits; lottery assumptions, structured pruning, and dynamic sparse masks are used to reduce parameter communication volume. In terms of security defense, backdoor detection and removal methods such as STRIP, ABS, and Neural Cleanse are used to identify and clean malicious features; and in terms of lightweight fine-tuning, low-rank adaptation techniques such as LoRA, QLoRA, and Adapter are used to reduce the computational cost of training large models.

[0005] However, the aforementioned existing technologies have significant limitations and cannot achieve multi-objective collaborative optimization. FedAvg and FedProx are prone to model drift under Non-IID data, resulting in a significant drop in generalization performance; FedKD and FedDF do not incorporate Dirichlet distribution for data adaptation, lack generalization confidence assessment, and have low-quality distilled samples; traditional pruning and sparsification only perform single-layer screening and do not achieve three-level importance fusion, resulting in significant loss of model accuracy after sparsification; backdoor defenses such as Neural Cleanse and ABS do not divide feature regions based on generalization saliency and lack fidelity calibration, leading to a decrease in generalization ability after defense; conventional LoRA / QLoRA does not link with global sparse masking, only achieving parameter lightweighting, and is not integrated with security defense and generalization enhancement. Overall, existing technologies cannot simultaneously solve core issues such as Non-IID adaptation, generalization improvement, communication burden reduction, backdoor defense, feature fidelity, and efficient fine-tuning, making it difficult to support the long-term stable operation of large models in privacy-sensitive, heterogeneous distributed scenarios.

[0006] In summary, current federated fine-tuning techniques for large models suffer from multiple contradictions regarding privacy and security, generalization performance, communication efficiency, defense effectiveness, and computational cost, lacking an integrated and collaborative solution. Therefore, there is an urgent need to propose a generalization-enhanced sparse federated distillation and security-defense fine-tuning method for large models. This method systematically addresses the shortcomings of existing technologies by organically integrating Dirichlet data partitioning, generalization confidence assessment, three-level sparse masking, dual-path backdoor defense, amplitude-limiting fidelity calibration, and mask-aware LoRA fine-tuning, achieving a secure, efficient, and generalization-enhanced distributed fine-tuning process for large models. Summary of the Invention

[0007] This invention proposes a generalization-enhanced sparse federated distillation and security-defense fine-tuning method for large models, aiming to effectively solve the challenge of large model fine-tuning in scenarios where multiple clients cannot directly share private data. It overcomes the shortcomings of existing technologies, such as high federated communication overhead, insufficient model generalization ability under non-independent and identically distributed data, difficulty in accurately suppressing backdoor attacks, and the inability to coordinate lightweight fine-tuning and security defense. This invention is applicable to multi-modal large model training and fine-tuning scenarios involving multiple clients within a federated learning framework, and can support efficient distillation, secure aggregation, and lightweight fine-tuning of image-text multimodal data under strict privacy protection conditions. In typical business scenarios such as distributed cross-domain model fine-tuning, local data privacy distillation, core feature extraction, server-side security defense and feature calibration, and efficient deployment of large models in the cloud, an integrated mechanism is adopted, which includes non-independent and identically distributed data partitioning based on Dirichlet distribution, generalization confidence assessment, three-level importance dynamic sparse masking, dual-path structured backdoor defense and amplitude limiting fidelity calibration, and mask-aware structured LoRA fine-tuning. This mechanism achieves an effective balance between data simplification, privacy protection, generalization enhancement, and backdoor defense robustness of large models, and significantly improves the generalization performance and security reliability of the model in specific downstream tasks.

[0008] This method mainly consists of a client, an edge server, and a cloud server. The client is responsible for local data processing, privacy distillation, sparse mask generation, and contribution weight quantization; the edge server is responsible for weighted aggregation, feature region segmentation, backdoor defense, and fidelity calibration; and the cloud server is responsible for lightweight LoRA fine-tuning of large models, performance evaluation, and deployable model output.

[0009] Furthermore, the method comprises three steps: a client-side local generalization enhancement distillation, dynamic sparsification, and contribution weight quantization stage; an edge server weighted aggregation, dual-path backdoor defense, and fidelity calibration stage; and a cloud server structured LoRA lightweight fine-tuning, security performance verification, and model output stage. Specifically, it includes the following steps:

[0010] Step 1: In the client-side local generalization enhancement distillation, dynamic sparsity, and contribution weight quantification stage, the edge server sends initial model parameters to the clients. Each client completes non-independent identically distributed partitioning based on local data, evaluates the value of quantified samples through generalization confidence, generates local privacy-preserving synthetic distillation data, completes model hierarchy importance assessment and dynamic sparsity mask generation, quantifies local contribution weights, and uploads relevant data. This stage achieves distillation data privacy and model parameter sparsity without the original data leaving the domain, significantly reducing subsequent communication overhead and providing high-quality local data for server aggregation.

[0011] Preferably, step one mainly includes three steps:

[0012] Step 1: The client receives the initial model parameters from the edge server and sets the total number of clients participating in federated learning to [number missing]. The total number of data categories is Dirichlet concentration parameters are dynamically calculated based on local data category distribution and data volume statistics. By randomly generating independent class probability distribution vectors for each client using the Dirichlet distribution, the heterogeneity of data in a real federated scenario is simulated, and the partitioning of non-independent and identically distributed data is completed.

[0013] (1)

[0014] in, Indicates the first The category probability distribution vector of each client; Indicates the Dirichlet distribution; The distribution of local data categories and the statistical value of data volume are dynamically determined; the more uniform the statistical distribution, the better. The larger the value, the more evenly the data is distributed; This represents the total number of clients participating in federated learning.

[0015] After loading the initial model parameters, input the local private non-independent and identically distributed data into the model to complete the forward inference, and process the logits vector output by the model. Perform softmax normalization to convert unconstrained logits values ​​to... Standard class probability distribution of the interval:

[0016] (2)

[0017] in, This represents the output logits vector of the client's local model, with the subscript... It is the target category. It is the summation variable that iterates through all categories; This represents the operation of the exponential function; This represents the normalized exponential function; Indicates the first The predicted probability values ​​for each category; This indicates the total number of categories in the data classification.

[0018] Based on the category probability distribution, three core indicators—maximum confidence score, Top-2 decision margin, and normalized information entropy—are calculated. These three indicators are then linearly combined with pre-set weights and numerically truncated to obtain a generalization confidence score that accurately measures the generalization value of the samples.

[0019] (3)

[0020] in, This represents the maximum class confidence score of the model output; The decision margin representing the probability of the top-2 classes; information entropy. Normalized information entropy ; This represents a numerical truncation function that restricts the score to a certain value. Within the range, a higher score indicates a higher generalization value of the sample.

[0021] Step 2: To achieve local data privacy protection and knowledge preservation, the client constructs a multi-constraint joint optimization loss function containing six sub-losses, without allowing the original data to leave the domain. Through multi-objective gradient optimization, it generates high-fidelity synthetic distilled data that preserves core knowledge.

[0022] (4)

[0023] in, This represents the category feature alignment loss, used to ensure that the class center features of the synthesized sample are consistent with those of the original sample; This represents the classification cross-entropy loss, used to ensure the classification accuracy of synthesized samples; This represents the generalization score consistency loss, used to constrain the matching of the generalization value of synthetic samples with that of the original samples. This represents the total variational regularization loss, used to suppress high-frequency noise in synthetic data; This represents the anchor point regularization loss, used to prevent synthesized samples from deviating from their initial values; This represents the L2 pixel regularization loss, used to constrain the numerical range of the synthesized data; , , , , , These represent the weighting coefficients of the corresponding sub-losses, with the weighting coefficients taking values ​​ranging from 1 to 2. , , , , , The optimal value was determined through a grid search experiment.

[0024] Iterative gradient descent optimization is performed based on the joint loss function, updating the synthetic distillation data parameters round by round. After each gradient update, pixel value truncation constraints are applied to the synthetic distillation data to ensure that the data conforms to the normal numerical range of images and features, avoiding abnormal values ​​from affecting subsequent processes.

[0025] (5)

[0026] in, Indicates the first One sample of synthetic distillation data; This represents a numerical truncation function that ensures pixel values ​​remain stable within the valid range of [0,1], thus preventing numerical anomalies.

[0027] Step 3: To filter high-value parameters and reduce federated communication overhead, the client performs a three-level unit importance assessment on the model: layer, neuron, and attention head. The layer importance score is calculated by weighting the average activation magnitude and gradient norm of all parameters in that layer; the neuron importance score is calculated by weighting the variance of the neuron's output activation value and the weight magnitude; and the attention head importance score is calculated by weighting the weight distribution entropy of the attention head and the contribution of its output features. These three importance scores are then combined with the absolute values ​​of the parameters to obtain a comprehensive importance score that fully reflects the contribution of each parameter.

[0028] (6)

[0029] in, This represents the overall importance score of the nth model parameter; It represents the absolute value of the parameter, reflecting the magnitude of the parameter update; This represents the single-client level importance score of the layer to which the parameter belongs; This represents the importance score of the neuron to which the parameter belongs; This represents the importance score of the attention head to which the parameter belongs.

[0030] Dynamic Top-K sorting is performed based on the comprehensive importance score of parameters, retaining a preset proportion of high-importance parameters, and generating a binary global sparse mask to achieve lightweight sparsity of model parameters:

[0031] (7)

[0032] in, This represents the sparse mask corresponding to the parameter; This indicates an indicator function; if the condition is true, the value is 1, meaning the parameter is retained; otherwise, the value is 0, meaning the parameter is masked. This represents the screening threshold for the overall importance score of the parameters.

[0033] Taking into account both the generalization quality and data scale of the distillation data, the contribution weight of the local data to federated learning is calculated. The contribution weight, distillation data, sparse mask, and single-client hierarchical importance score are then packaged and uploaded to the edge server.

[0034] (8)

[0035] in, Indicates the first The contribution weight of each client; Indicates the client The average generalization score of the distillation data; Indicates the client The sample size of the distillation data; Indicates the client The original data sample size; with generalization score as the core contribution and data size as an auxiliary contribution, the impact of distillation data generalization quality and sample size on federated aggregation is balanced.

[0036] This step addresses the pain points of existing federated learning, such as poor adaptation to non-IID data, the contradiction between privacy protection and generalization performance, and excessive communication overhead, by proposing an integrated client-side preprocessing solution. It accurately simulates the distribution of real heterogeneous data using the Dirichlet distribution, effectively solving the data skew problem caused by traditional random partitioning. Combined with generalization confidence assessment to select high-value samples, and relying on multi-constraint joint loss to generate privacy-preserving synthetic distilled data, it ensures that the original data does not leave the domain, balancing privacy and knowledge fidelity. Simultaneously, it innovates a three-level importance assessment of the layer-neuron-attention head, generating a dynamic sparse mask that significantly reduces the amount of communication data. By quantifying contribution weights through both generalization quality and data scale, it lays the foundation for subsequent fair aggregation, fundamentally solving the multiple bottlenecks of data, privacy, and communication in federated training.

[0037] Step Two: In the edge server weighted aggregation, dual-path backdoor defense, and fidelity calibration phase, the edge server receives data uploaded from each client. Based on contribution weights, it performs weighted aggregation of distillation data, hierarchical importance scores, and sparse masks. It then uses a generalization saliency heatmap to segment feature regions, executes dual-path structured backdoor defense, and corrects feature distribution offsets through amplitude-limited generalization fidelity calibration. This phase accurately removes backdoor-triggered features, ensuring the security of distillation data and maximizing the preservation of the model's core generalization features, providing secure and reliable data support for fine-tuning large models in the cloud.

[0038] Preferably, step two mainly includes three steps:

[0039] Step 1: The edge server receives distillation data, sparse masks, single-client hierarchical importance scores, and contribution weights uploaded by all clients. To ensure the fairness of weight calculation and avoid the anomaly of zero denominator, a unified minimum constant is introduced. Perform standardization and normalization on the client contribution weights:

[0040] (9)

[0041] in, This represents the normalized client contribution weight; To maintain a consistent minimum value throughout the text, and to prevent calculation anomalies caused by weights being zero; This represents the total number of clients participating in federated learning.

[0042] The total number of global aggregated samples is dynamically determined based on the computing power and storage limits of the federated cluster. Sampling quotas are calculated based on normalized contribution weights, and samples are extracted and concatenated to form a global aggregated distillation dataset. ,in For the first Each client sampling quota, The total number of samples in the global aggregation is dynamically determined.

[0043] A globally unified sparse mask is generated by fusing the sparse masks from all clients using a weighted voting mechanism and performing binarization.

[0044] (10)

[0045] in, Represents a global sparse mask; Indicates the first Sparse mask of parameters uploaded by each client; This represents the normalized client contribution weight; The indicator function is used to implement mask binarization;

[0046] Perform weighted aggregation on the single-client-level importance scores uploaded by the client to obtain the global aggregated level importance score: .

[0047] Step 2: The edge server generates a generalized significance score based on the global aggregation hierarchy importance score and the mask retention rate, which is the retention rate of the parameter in the global mask. The ratio of the number of parameters preserved in the global sparse mask of the layer to the total number of parameters in that layer is calculated using the following formula: The weight ratios were determined through a variance experiment on defense effectiveness. The generalization significance score for each parameter was calculated, and the significance scores were subjected to min-max normalization to eliminate differences in numerical units and achieve accurate segmentation of feature regions.

[0048] (11)

[0049] (12)

[0050] in, This represents the original generalization significance score of the parameter; This represents the importance score of the global aggregation hierarchy; This indicates the retention rate of the parameter in the global sparse mask; This represents the generalization significance score after normalization; m is the index for traversing all model parameters; To ensure a consistent minimum constant across the entire text and avoid zero denominators, and because the contributions of hierarchical importance scores and mask retention rates to parameter generalization significance are equivalent, equal-weighted fusion is employed.

[0051] A threshold is set based on the normalized significance score to divide the high-generalization core region into a low-generalization redundant region. Orthogonal projection feature cleaning is then performed on the high-generalization core region to completely remove backdoor perturbations. ,in, Features after cleaning These are the original aggregation features. The global reference vector is the aggregated feature mean. for The L2 norm.

[0052] Adaptive Gaussian noise is applied to low-generalization redundancy regions, with noise intensity linearly negatively correlated with the generalization significance score, to pinpoint and remove backdoor implantation features. ,in The optimal base noise intensity for robustness.

[0053] Step 3: To correct the feature distribution shift caused by the dual-path defense operation, the edge server performs amplitude-limiting generalization fidelity calibration only on samples with generalization confidence scores not lower than a preset threshold, and the calibration intensity coefficient... The range of values ​​is This ensures that the defense effect is not weakened and that the feature distribution is repaired; among which To normalize the generalization confidence scalar, the generalization confidence score from step 1-1-3 is generated through min-max normalization. The element-wise multiplication of the scalar and the feature vector satisfies the dimension matching requirement.

[0054] (13)

[0055] in, This represents the final feature data after calibration; This represents the feature data after dual-path defense; This represents the fidelity calibration intensity coefficient, with a value range of [value range missing]. ; This represents the normalized generalization confidence scalar; This indicates the global polymer distillation characteristics before defense; This indicates element-wise multiplication.

[0056] The generalization feature distribution rule consists of the mean, variance, and normalized generalization significance score distribution of the global polymeric distillation features, which is generated by generalization significance heatmap and calibration feature statistics. The three core data types, namely the calibrated safe distillation dataset, global sparse mask, and generalization feature distribution rule, are uniformly synchronized to the cloud server as the sole safe data source for fine-tuning the large cloud model.

[0057] Step 3: In the cloud server's structured LoRA lightweight fine-tuning, security performance verification, and model output stage, the cloud server performs structured LoRA lightweight fine-tuning on the multimodal large model based on the security distillation dataset, global sparse mask, and generalization feature distribution rules. The model's generalization performance and backdoor defense effectiveness are evaluated through multi-dimensional metrics, ultimately outputting a deployable optimized large model. This stage significantly reduces the computational cost of large model fine-tuning, achieving a synergistic improvement in model performance and security defense capabilities, and outputting a highly secure and reliable deployable model.

[0058] This step addresses the shortcomings of existing aggregation methods, such as susceptibility to interference, inaccurate backdoor defense, and easily corrupted features, by constructing a secure and efficient edge aggregation defense system. It achieves weighted aggregation of distilled data and sparse masks based on client contribution weights, avoiding contamination of traditional average aggregation by low-quality or malicious data. By using a generalization saliency heatmap to divide feature regions, a dual-path structured backdoor defense is employed to accurately remove hidden backdoor perturbations, overcoming the limitations of existing defenses that easily damage core generalization features. Through amplitude-limiting fidelity calibration, it repairs feature distribution shifts caused by defense, thoroughly resisting backdoor attacks while maximizing the preservation of data generalization value, providing secure, clean, and feature-complete high-quality data support for cloud-based fine-tuning.

[0059] Preferably, step three mainly includes two steps:

[0060] Step 1: Cloud server combines global sparse mask Structured LoRA low-rank adapters are deployed only in layers where the mask parameter retention ratio is not lower than a preset layer threshold. These adapters are used in the self-attention query / key / value / output projection layers and feedforward network gating / up / down projection layers of the multimodal large model. Furthermore, LoRA low-rank updates only affect the mask-retained parameter positions; no adapters are deployed in the mask layer. Only low-rank parameters are updated, and pre-trained backbone parameters are frozen, significantly reducing the computational cost of fine-tuning.

[0061] (14)

[0062] in, This represents the model weight matrix after fine-tuning; This represents the original weight matrix of the pre-trained model; This represents the LoRA scaling factor, relative to the Dirichlet parameters. are independent parameters; A and B represent the LoRA low-rank decomposition matrices.

[0063] Based on the secure distillation dataset synchronized by the edge server and the generalization feature distribution rules, we construct multimodal dialogue format training data containing images, prompts, and labels. We adopt an assistant-specific supervision strategy, calculating the loss only on the answer part of the model output and ignoring the user prompts to improve the accuracy of fine-tuning.

[0064] To reduce memory usage during single-step training and adapt to fine-tuning of large models, a gradient accumulation training strategy is adopted, accumulating... After each gradient step, a unified parameter update is performed. The loss per microstep is:

[0065] (15)

[0066] in, This represents the loss value for a single microstep; This represents the single-step supervised fine-tuning loss; the total gradient is... The sum of gradients, divided by the value before parameter update. Achieve smooth updates.

[0067] Step 2: The cloud server loads the test dataset to complete model inference. Based on the inference results, the classification accuracy and invalid output rate are calculated. Simultaneously, a benign test set and a backdoor attack test set are used to test the model's benign accuracy (BA) and backdoor attack success rate (ASR) respectively, comprehensively verifying the model's performance.

[0068] (16)

[0069] (17)

[0070] in, This represents the model's classification accuracy. Indicates the category predicted by the model; Indicates the true category of the sample; This represents the total number of samples in the test set; This indicates the invalid output rate of the model.

[0071] The fine-tuned structured LoRA adapter is merged with the backbone model, the complete model file is saved to the specified output directory, and a compliant, deployable, optimized large model with generalization enhancement and backdoor removal is directly output.

[0072] This step addresses the high computational cost of full parameter fine-tuning for large models, the disconnect between lightweight design and security defense, and the simplistic performance evaluation methods, achieving efficient and secure model fine-tuning optimization. It utilizes a structured LoRA adapter deployed with a global sparse mask, updating low-rank parameters only at high-value parameter locations and freezing the backbone network, significantly reducing fine-tuning computational and memory overhead, thus adapting to the lightweight deployment requirements of large models. A gradient accumulation strategy is employed to adapt to large model training, and multi-dimensional metrics are used to evaluate generalization performance and defense effectiveness, balancing model accuracy improvement with enhanced backdoor robustness. This solution establishes a collaborative link between sparsity, defense, and fine-tuning, resolving the pain point of traditional lightweight design and security separation, and rapidly outputting deployable large models that combine generalization capabilities, privacy security, and robust defense.

[0073] The beneficial effects of this invention are:

[0074] (1) This invention proposes a generalization enhancement sparse federated distillation and security defense fine-tuning method for large models. By adapting non-independent and identically distributed data to Dirichlet distribution, evaluating generalization confidence, and employing local privacy-preserving synthetic distillation techniques, it fundamentally solves the core challenges of data privacy protection, data heterogeneity adaptation, and insufficient model generalization ability in large model fine-tuning under federated learning scenarios. Under the premise that the original client data does not leave the domain, this method generates high-fidelity synthetic distillation data based on a multi-constraint joint loss function, accurately quantifies and retains the core generalization value of the samples, effectively offsets the negative impact of the non-independent and identically distributed characteristics of the data on the global model, and significantly improves the generalization performance and robustness of the model in downstream vertical tasks. This provides safe and reliable technical support for the compliant fine-tuning of large models in privacy-sensitive fields such as medicine and finance.

[0075] (2) This invention employs a three-level importance assessment mechanism involving layers, neurons, and attention heads, along with a dynamic sparse masking mechanism, to achieve structured and lightweight model parameters and efficient federated communication. This mechanism filters high-value model parameters through multi-dimensional weight fusion, generates a dynamic sparse mask, and transmits only key parameters and lightweight distilled data. Compared to traditional full-parameter federated learning, it significantly reduces the computational load and network communication bandwidth consumption of edge clients, while fully preserving the core feature representation capabilities of the model. It achieves an optimal balance between communication efficiency and model performance, perfectly adapting to the deployment needs of resource-constrained edge devices and distributed federated scenarios.

[0076] (3) This invention innovatively designs a dual-path structured backdoor defense and amplitude-limiting fidelity calibration mechanism based on generalization saliency heatmaps, achieving precise removal of backdoor attacks and stable repair of feature distribution. This method divides the high-generalization core region and the low-generalization redundant region, and uses orthogonal projection cleaning and adaptive noise interference dual-path processing to thoroughly remove hidden backdoor triggering features. Then, amplitude-limiting calibration is used to repair the feature distribution shift caused by the defense operation. While significantly reducing the success rate of backdoor attacks, it maximizes the preservation of original data information and the benign performance of the model, providing a solid guarantee for the safe training and stable operation of large models in open distributed environments.

[0077] (4) This invention adopts a structured LoRA lightweight fine-tuning scheme with global sparse mask adaptation, which realizes efficient and low-cost fine-tuning and rapid deployment of large models. This scheme only deploys low-rank adapters at the high-value parameter positions preserved in the mask, freezes the pre-trained backbone network, and updates only a small number of low-rank parameters, which greatly reduces the computational power consumption and training time of large model fine-tuning, while fully preserving the effects of generalization enhancement and security defense, enabling large model federated fine-tuning to be implemented quickly, and significantly improving the engineering practicality and large-scale application value of the technology. Attached Figure Description

[0078] Figure 1 This is a framework diagram of a generalization enhancement sparse federated distillation and security defense fine-tuning method for large models proposed in this invention.

[0079] Figure 2 This is a flowchart of a generalization enhancement sparse federated distillation and security defense fine-tuning method for large models proposed in this invention; Detailed Implementation

[0080] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0081] Example: A privacy-preserving federated distillation and backdoor defense method for fine-tuning large models, such as... Figure 1 As shown, it specifically includes:

[0082] Step 1-1: The client receives the initial model parameters from the edge server and sets the total number of clients participating in federated learning to [number missing]. The total number of data categories is Dirichlet concentration parameters are dynamically calculated based on local data category distribution and data volume statistics. By randomly generating independent class probability distribution vectors for each client using the Dirichlet distribution, the heterogeneity of data in a real federated scenario is simulated, and the partitioning of non-independent and identically distributed data is completed.

[0083] (1)

[0084] in, Indicates the first The category probability distribution vector of each client; Indicates the Dirichlet distribution; The distribution of local data categories and the statistical value of data volume are dynamically determined; the more uniform the statistical distribution, the better. The larger the value, the more evenly the data is distributed; This represents the total number of clients participating in federated learning.

[0085] After loading the initial model parameters, input the local private non-independent and identically distributed data into the model to complete the forward inference, and process the logits vector output by the model. Perform softmax normalization to convert unconstrained logits values ​​to... Standard class probability distribution of the interval:

[0086] (2)

[0087] in, This represents the output logits vector of the client's local model, with the subscript... It is the target category. It is the summation variable that iterates through all categories; This represents the operation of the exponential function; This represents the normalized exponential function; Indicates the first The predicted probability values ​​for each category; This indicates the total number of categories in the data classification.

[0088] Based on the category probability distribution, three core indicators—maximum confidence score, Top-2 decision margin, and normalized information entropy—are calculated. These three indicators are then linearly combined with pre-set weights and numerically truncated to obtain a generalization confidence score that accurately measures the generalization value of the samples.

[0089] (3)

[0090] in, This represents the maximum class confidence score of the model output; The decision margin representing the probability of the top-2 classes; information entropy. Normalized information entropy ; This represents a numerical truncation function that restricts the score to a certain value. Within the range, a higher score indicates a higher generalization value of the sample.

[0091] Steps 1-2: To achieve local data privacy protection and knowledge preservation, the client constructs a multi-constraint joint optimization loss function containing six sub-losses, without allowing the original data to leave the domain. Through multi-objective gradient optimization, it generates high-fidelity synthetic distilled data that preserves core knowledge.

[0092] (4)

[0093] in, This represents the category feature alignment loss, used to ensure that the class center features of the synthesized sample are consistent with those of the original sample; This represents the classification cross-entropy loss, used to ensure the classification accuracy of synthesized samples; This represents the generalization score consistency loss, used to constrain the matching of the generalization value of synthetic samples with that of the original samples. This represents the total variational regularization loss, used to suppress high-frequency noise in synthetic data; This represents the anchor point regularization loss, used to prevent synthesized samples from deviating from their initial values; This represents the L2 pixel regularization loss, used to constrain the numerical range of the synthesized data; , , , , , These represent the weighting coefficients of the corresponding sub-losses, with the weighting coefficients taking values ​​ranging from 1 to 2. , , , , , The optimal value was determined through a grid search experiment.

[0094] Iterative gradient descent optimization is performed based on the joint loss function, updating the synthetic distillation data parameters round by round. After each gradient update, pixel value truncation constraints are applied to the synthetic distillation data to ensure that the data conforms to the normal numerical range of images and features, avoiding abnormal values ​​from affecting subsequent processes.

[0095] (5)

[0096] in, Indicates the first One sample of synthetic distillation data; This represents a numerical truncation function that ensures pixel values ​​remain stable within the valid range of [0,1], thus preventing numerical anomalies.

[0097] Steps 1-3: To filter high-value parameters and reduce federated communication overhead, the client performs a three-level unit importance assessment on the model: layer, neuron, and attention head. The layer importance score is calculated by weighting the average activation magnitude and gradient norm of all parameters in that layer; the neuron importance score is calculated by weighting the variance of the neuron's output activation value and the weight magnitude; and the attention head importance score is calculated by weighting the weight distribution entropy of the attention head and the contribution of its output features. These three importance scores are then combined with the absolute values ​​of the parameters to obtain a comprehensive importance score that fully reflects the contribution of each parameter.

[0098] (6)

[0099] in, This represents the overall importance score of the nth model parameter; It represents the absolute value of the parameter, reflecting the magnitude of the parameter update; This represents the single-client level importance score of the layer to which the parameter belongs; This represents the importance score of the neuron to which the parameter belongs; This represents the importance score of the attention head to which the parameter belongs.

[0100] Dynamic Top-K sorting is performed based on the comprehensive importance score of parameters, retaining a preset proportion of high-importance parameters, and generating a binary global sparse mask to achieve lightweight sparsity of model parameters:

[0101] (7)

[0102] in, This represents the sparse mask corresponding to the parameter; This indicates an indicator function; if the condition is true, the value is 1, meaning the parameter is retained; otherwise, the value is 0, meaning the parameter is masked. This represents the screening threshold for the overall importance score of the parameters.

[0103] Taking into account both the generalization quality and data scale of the distillation data, the contribution weight of the local data to federated learning is calculated. The contribution weight, distillation data, sparse mask, and single-client hierarchical importance score are then packaged and uploaded to the edge server.

[0104] (8)

[0105] in, Indicates the first The contribution weight of each client; Indicates the client The average generalization score of the distillation data; Indicates the client The sample size of the distillation data; Indicates the client The original data sample size; with generalization score as the core contribution and data size as an auxiliary contribution, the impact of distillation data generalization quality and sample size on federated aggregation is balanced.

[0106] Step 2-1: The edge server receives distillation data, sparse mask, single-client hierarchical importance score and contribution weight uploaded by all clients. To ensure the fairness of weight calculation and avoid the anomaly of zero denominator, a unified minimum constant is introduced. Perform standardization and normalization on the client contribution weights:

[0107] (9)

[0108] in, This represents the normalized client contribution weight; To maintain a consistent minimum value throughout the text, and to prevent calculation anomalies caused by weights being zero; This represents the total number of clients participating in federated learning.

[0109] The total number of global aggregated samples is dynamically determined based on the computing power and storage limits of the federated cluster. Sampling quotas are calculated based on normalized contribution weights, and samples are extracted and concatenated to form a global aggregated distillation dataset. ,in For the first Each client sampling quota, The total number of samples in the global aggregation is dynamically determined.

[0110] A globally unified sparse mask is generated by fusing the sparse masks from all clients using a weighted voting mechanism and performing binarization.

[0111] (10)

[0112] in, Represents a global sparse mask; Indicates the first Sparse mask of parameters uploaded by each client; This represents the normalized client contribution weight; This is an indicator function used to implement mask binarization.

[0113] Perform weighted aggregation on the single-client-level importance scores uploaded by the client to obtain the global aggregated level importance score: .

[0114] Step 2-2: The edge server generates a generalized significance score based on the global aggregation hierarchy importance score and the mask retention rate, where the parameter's retention rate in the global mask is... The ratio of the number of parameters preserved in the global sparse mask of the layer to the total number of parameters in that layer is calculated using the following formula: The weight ratios were determined through a variance experiment on defense effectiveness. The generalization significance score for each parameter was calculated, and the significance scores were subjected to min-max normalization to eliminate differences in numerical units and achieve accurate segmentation of feature regions.

[0115] (11)

[0116] (12)

[0117] in, This represents the original generalization significance score of the parameter; This represents the importance score of the global aggregation hierarchy; This indicates the retention rate of the parameter in the global sparse mask; This represents the generalization significance score after normalization; m is the index for traversing all model parameters; To ensure a consistent minimum constant across the entire text and avoid zero denominators, and because the contributions of hierarchical importance scores and mask retention rates to parameter generalization significance are equivalent, equal-weighted fusion is employed.

[0118] A threshold is set based on the normalized significance score to divide the high-generalization core region into a low-generalization redundant region. Orthogonal projection feature cleaning is then performed on the high-generalization core region to completely remove backdoor perturbations. ,in, Features after cleaning These are the original aggregation features. The global reference vector is the aggregated feature mean. for The L2 norm.

[0119] Adaptive Gaussian noise is applied to low-generalization redundancy regions, with noise intensity linearly negatively correlated with the generalization significance score, to pinpoint and remove backdoor implantation features. ,in The optimal base noise intensity for robustness.

[0120] Steps 2-3: To correct the feature distribution shift caused by the dual-path defense operation, the edge server performs amplitude-limiting generalization fidelity calibration only on samples with generalization confidence scores not lower than a preset threshold, and the calibration intensity coefficient... The range of values ​​is This ensures that the defense effect is not weakened and that the feature distribution is repaired; among which To normalize the generalization confidence scalar, which is generated from the generalization confidence score through min-max normalization, the element-wise multiplication of the scalar and the feature vector satisfies the dimension matching requirement:

[0121] (13)

[0122] in, This represents the final feature data after calibration; This represents the feature data after dual-path defense; This represents the fidelity calibration intensity coefficient, with a value range of [value range missing]. ; This represents the normalized generalization confidence scalar; This indicates the global polymer distillation characteristics before defense; This indicates element-wise multiplication.

[0123] The generalization feature distribution rule consists of the mean, variance, and normalized generalization significance score distribution of the global polymeric distillation features, which is generated by generalization significance heatmap and calibration feature statistics. The three core data types, namely the calibrated safe distillation dataset, global sparse mask, and generalization feature distribution rule, are uniformly synchronized to the cloud server as the sole safe data source for fine-tuning the large cloud model.

[0124] Step 3-1: Cloud server combines global sparse mask Structured LoRA low-rank adapters are deployed only in layers where the mask parameter retention ratio is not lower than a preset layer threshold. These adapters are used in the self-attention query / key / value / output projection layers and feedforward network gating / up / down projection layers of the multimodal large model. Furthermore, LoRA low-rank updates only affect the mask-retained parameter positions; no adapters are deployed in the mask layer. Only low-rank parameters are updated, and pre-trained backbone parameters are frozen, significantly reducing the computational cost of fine-tuning.

[0125] (14)

[0126] in, This represents the model weight matrix after fine-tuning; This represents the original weight matrix of the pre-trained model; This represents the LoRA scaling factor, relative to the Dirichlet parameters. are independent parameters; A and B represent the LoRA low-rank decomposition matrices.

[0127] Based on the secure distillation dataset synchronized by the edge server and the generalization feature distribution rules, we construct multimodal dialogue format training data containing images, prompts, and labels. We adopt an assistant-specific supervision strategy, calculating the loss only on the answer part of the model output and ignoring the user prompts to improve the accuracy of fine-tuning.

[0128] To reduce memory usage during single-step training and adapt to fine-tuning of large models, a gradient accumulation training strategy is adopted, accumulating... After each gradient step, a unified parameter update is performed. The loss per microstep is:

[0129] (15)

[0130] in, This represents the loss value for a single microstep; This represents the single-step supervised fine-tuning loss; the total gradient is... The sum of gradients, divided by the value before parameter update. Achieve smooth updates.

[0131] Step 3-2: The cloud server loads the test dataset to complete model inference. Based on the inference results, the classification accuracy and invalid output rate are calculated. Simultaneously, a benign test set and a backdoor attack test set are used to test the model's benign accuracy (BA) and backdoor attack success rate (ASR) respectively, comprehensively verifying the model's performance.

[0132] (16)

[0133] (17)

[0134] in, This represents the model's classification accuracy. Indicates the category predicted by the model; Indicates the true category of the sample; This represents the total number of samples in the test set; This indicates the invalid output rate of the model.

[0135] The fine-tuned structured LoRA adapter is merged with the backbone model, the complete model file is saved to the specified output directory, and a compliant, deployable, optimized large model with generalization enhancement and backdoor removal is directly output.

[0136] To verify the effectiveness of the proposed method, we designed and executed a series of experiments, mainly evaluating it from the following three aspects: (1) the privacy protection effect of dataset distillation; (2) the ability of adaptive backdoor defense processing to eliminate backdoor triggers; and (3) a comprehensive comparison of the performance and security of the final fine-tuned model. The experiments used the general image classification dataset CIFAR-10 as the core test set, built a multi-client federated simulation environment, and carried out verification and analysis from three dimensions: stage output effect, generalization performance, and defense performance.

[0137] 1. Privacy protection effect of dataset distillation:

[0138] The synthetic distillation data generated through multi-constraint joint loss optimization exhibits visually blurred and noisy characteristics, making it impossible to reconstruct from the original private data. This completely avoids the risk of leaking details of the original data and achieves the privacy protection goal of keeping private data within its domain. At the same time, the synthetic data still fully retains the category semantic features and distribution patterns of the original data. The category features of key targets such as airplanes, cars, and birds are clearly distinguishable, which can effectively support subsequent model training and knowledge transfer. This verifies that while protecting data privacy, this method can still provide sufficient generalization knowledge support for the model, thus balancing privacy security and knowledge usability.

[0139] 2. Adaptive backdoor defense processing's ability to eliminate backdoor triggers;

[0140] This experiment, based on the CIFAR-10 dataset, injects a BadNet backdoor to verify the effectiveness of the dual-path structured backdoor defense and amplitude-limiting fidelity calibration mechanism of this invention. Experimental results show that backdoor triggering features (BadNet's preset perturbation pattern) still remain in the distilled data before defense. However, after processing by the method of this invention, the backdoor triggering features in the synthetic data are completely removed, with no obvious abnormal perturbations or backdoor pattern residues, and visually indistinguishable from normal distilled data. Simultaneously, the category semantic information and core feature distribution of the data are completely preserved, and the category features of targets such as airplanes, cars, and animals are clearly distinguishable, without information loss or distribution shift due to the defense operation. This verifies that the dual-path defense based on generalization saliency heatmaps can accurately locate and remove backdoor triggering features, and the amplitude-limiting fidelity calibration effectively repairs the feature distribution shift during the defense process. While completely eliminating the risk of backdoor attacks, it also ensures the generalization value of the distilled data, providing safe and clean data support for fine-tuning large models in the cloud.

[0141] 3. A comprehensive comparison of the performance and security of the final fine-tuned model;

[0142] This experiment uses the Qwen / Qwen3.5-2B model as the backbone and builds a federated simulation environment with three clients. Each client generates 128 synthetic distillation images based on the CIFAR-10+BadNet backdoor dataset using the method of this invention, resulting in a total of 128×3 privacy-preserving and secure distillation data for subsequent model fine-tuning and validation. Before fine-tuning, the model's basic classification accuracy on the clean CIFAR-10 test set was 88%, while the success rate (ASR) against BadNet backdoor attacks was as high as 92.3%, indicating limited generalization ability and serious backdoor security vulnerabilities. After processing using the method of this invention, the model's classification accuracy on the CIFAR-10 test set improved to 94.5%, an improvement of 6.5 percentage points compared to the baseline model, significantly enhancing generalization performance and verifying the effectiveness of the multi-client collaborative distillation and generalization enhancement mechanism. Regarding backdoor defense effectiveness, for test samples injected with the BadNet backdoor, the backdoor attack success rate (ASR) of the fine-tuned model dropped to 0, backdoor triggering was completely ineffective, and no pre-set malicious behavior occurred. This proves that the dual-path structured backdoor defense and amplitude-limiting fidelity calibration mechanism of this invention can completely remove backdoor triggering features from distillation data in multi-client aggregation scenarios, achieving immune-level defense against backdoor attacks. Overall, this method, using only a small amount of synthetic distillation data (384 images in total), simultaneously achieves a significant improvement in model generalization performance and complete resistance to backdoor attacks, fully verifying its effectiveness and practicality in the secure and efficient fine-tuning of large models in federated distributed scenarios.

[0143] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A generalization-enhanced sparse federated distillation and security defense fine-tuning method for large models, characterized in that, Includes the following steps: Step 1: The client performs data partitioning based on the Dirichlet distribution, non-independent and identically distributed data, generalization confidence assessment, and local synthetic data distillation. Combined with hierarchical importance assessment, it generates a dynamic sparse mask and quantifies client contribution weights, then uploads the hierarchical importance score, sparse mask, distillation data, and contribution weights. Step 2: The edge server performs distillation data sampling and aggregation, hierarchical importance score weighted aggregation, and sparse mask weighted voting binarization aggregation based on the client contribution weights. It then uses a generalization saliency heatmap to partition feature regions and executes a dual-path structured backdoor defense. Through amplitude limiting fidelity calibration, it outputs a secure distillation dataset, a global sparse mask, and generalized feature distribution rules. Step 3: The cloud server performs lightweight LoRA fine-tuning and generalization security performance assessment of the multimodal large model based on the secure distillation dataset, global sparse mask, and generalized feature distribution rules. It saves the model and outputs a deployable optimized large model. Step one specifically includes: Step 1-1: The client performs non-independent and identically distributed data partitioning on local private data by determining the Dirichlet parameters using local data statistics. Based on the logits output of the local model, it completes the generalization confidence score calculation to quantify the generalization value of the distilled samples. Step 1-2: The client constructs a multi-constraint weighted joint loss function, generates local privacy-preserving synthetic distilled data through gradient optimization, and applies pixel value range constraints to the distilled data. Step 1-3: The client evaluates the importance of the model execution layer, neurons, and attention head at three levels, fuses them to obtain a comprehensive parameter importance score, generates a dynamic sparse mask, calculates the contribution weight based on the generalization score and data volume ratio, and uploads the hierarchical importance score, sparse mask, distilled data, and contribution weight to the edge server. Step two specifically includes: Step 2-1: The edge server receives all data uploaded by clients, performs normalization processing on the contribution weights, and completes the weighted sampling aggregation of distillation data, the weighted aggregation of hierarchical importance scores, and the weighted voting binarization aggregation of sparse masks. Step 2-2: The edge server generates a generalization saliency heatmap based on the aggregated hierarchical scores and mask retention rate, divides the high-generalization core region and the low-generalization redundant region, and performs dual-path structured backdoor defense. Step 2-3: The edge server performs amplitude-limited generalization fidelity calibration on the defended feature data, corrects the feature distribution offset without weakening the defense effect, and outputs the secure distillation dataset, global sparse mask, and generalized feature distribution rules to the cloud server. Step three specifically includes: Step 3-1: The cloud server deploys a structured LoRA adapter in the self-attention layer and feedforward network layer of the multimodal large model, constructs dialogue-format training data based on the secure distillation dataset and generalization feature distribution rules, and completes lightweight fine-tuning using assistant-specific supervision and gradient accumulation; Step 3-2: The cloud server calculates the model's classification accuracy, invalid output rate, benign accuracy, and backdoor attack success rate, evaluates generalization performance and defense effectiveness, saves the fine-tuned model and LoRA adapter, and directly outputs a deployable optimized large model.

2. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 1, characterized in that: Step 1-1 specifically includes the following steps: Step 1-1-1: The client receives the initial model parameters from the edge server and sets the total number of clients participating in federated learning to [number missing]. The total number of data categories is Dirichlet concentration parameters are dynamically calculated based on local data category distribution and data volume statistics. By randomly generating independent class probability distribution vectors for each client using the Dirichlet distribution, the heterogeneity of data in a real federated scenario is simulated, and the partitioning of non-independent and identically distributed data is completed. （1） in, Indicates the first The category probability distribution vector of each client; Indicates the Dirichlet distribution; The distribution of local data categories and the statistical value of data volume are dynamically determined; the more uniform the statistical distribution, the better. The larger the value, the more evenly the data is distributed. This indicates the total number of clients participating in federated learning. Step 1-1-2: After loading the initial model parameters, input the local private non-independent and identically distributed data into the model to complete the forward inference, and process the logits vector output by the model. Perform softmax normalization to convert unconstrained logits values ​​to... Standard class probability distribution of the interval: （2） in, This represents the output logits vector of the client's local model, with the subscript... It is the target category. It is the summation variable that iterates through all categories; This represents the operation of the exponential function; This represents the normalized exponential function; Indicates the first The predicted probability values ​​for each category; This indicates the total number of categories in the data classification. Step 1-1-3: Calculate the three core indicators—maximum confidence, Top-2 decision margin, and normalized information entropy—based on the class probability distribution. Then, linearly combine these three indicators with preset weights and perform numerical truncation to obtain a generalization confidence score that accurately measures the generalization value of the samples. （3） in, This represents the maximum class confidence score of the model output; The decision margin representing the probability of the top-2 classes; information entropy. Normalized information entropy ; This represents a numerical truncation function that restricts the score to a certain value. Within the range, a higher score indicates a higher generalization value of the sample.

3. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 2, characterized in that: Step 1-2 specifically includes the following steps: Step 1-2-1: To achieve local data privacy protection and knowledge preservation, the client constructs a multi-constraint joint optimization loss function containing six sub-losses, without allowing the original data to leave the domain. Through multi-objective gradient optimization, it generates high-fidelity synthetic distilled data that preserves core knowledge. （4） in, This represents the category feature alignment loss, used to ensure that the class center features of the synthesized sample are consistent with those of the original sample; This represents the classification cross-entropy loss, used to ensure the classification accuracy of synthesized samples; This represents the generalization score consistency loss, used to constrain the matching of the generalization value of synthetic samples with that of the original samples. This represents the total variational regularization loss, used to suppress high-frequency noise in synthetic data; This represents the anchor point regularization loss, used to prevent synthesized samples from deviating from their initial values; This represents the L2 pixel regularization loss, used to constrain the numerical range of the synthesized data; , , , , , These represent the weighting coefficients of the corresponding sub-losses, with the weighting coefficients taking values ​​ranging from 1 to 2. , , , , , The optimal value was determined through a grid search experiment. Step 1-2-2: Perform iterative gradient descent optimization based on the joint loss function, updating the synthetic distillation data parameters round by round. After each gradient update, apply pixel value truncation constraints to the synthetic distillation data to ensure that the data conforms to the normal numerical range of the image and features, avoiding abnormal values ​​from affecting subsequent processes. （5） in, This represents the j-th synthetic distillation data sample; This represents a numerical truncation function that ensures pixel values ​​remain stable within the valid range of [0,1], thus preventing numerical anomalies.

4. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 3, characterized in that: Steps 1-3 specifically include the following steps: Step 1-3-1: To screen high-value parameters and reduce federated communication overhead, the client performs a three-level unit importance assessment on the model: layer, neuron, and attention head. The layer importance score is calculated by weighting the average activation magnitude and gradient norm of all parameters in that layer; the neuron importance score is calculated by weighting the variance of the neuron's output activation value and the weight magnitude; and the attention head importance score is calculated by weighting the weight distribution entropy of the attention head and the contribution of its output features. These three importance scores are then combined with the absolute values ​​of the parameters to obtain a comprehensive importance score that fully reflects the contribution of each parameter. （6） in, This represents the overall importance score of the nth model parameter; It represents the absolute value of the parameter, reflecting the magnitude of the parameter update; This represents the single-client level importance score of the layer to which the parameter belongs; This represents the importance score of the neuron to which the parameter belongs; This represents the importance score of the attention head to which the parameter belongs. Step 1-3-2: Perform dynamic Top-K sorting and filtering based on the comprehensive importance score of the parameters, retain a preset proportion of high-importance parameters, and generate a binary global sparse mask to achieve lightweight sparsity of model parameters: （7） in, This represents the sparse mask corresponding to the parameter; This indicates an indicator function; if the condition is true, the value is 1, meaning the parameter is retained; otherwise, the value is 0, meaning the parameter is masked. The selection threshold representing the overall importance score of the parameters. Step 1-3-3: Taking into account both the generalization quality and data scale of the distillation data, calculate the local contribution weight to federated learning, and then package the contribution weight, distillation data, sparse mask, and single-client hierarchical importance score together and upload them to the edge server. （8） in, Indicates the first The contribution weight of each client; This represents the average generalization score of the client's k-distilled data; Indicates the client The sample size of the distillation data; This represents the original data sample size of client k; the generalization score is the core contribution, and the data size is the auxiliary contribution, balancing the impact of distillation data generalization quality and sample size on federated aggregation.

5. The generalization enhancement sparse federated distillation and security defense fine-tuning method for large models according to claim 4, characterized in that: Step 2-1 specifically includes the following steps: Step 2-1-1: The edge server receives distillation data, sparse mask, single-client hierarchical importance score and contribution weight uploaded by all clients, and introduces a unified minimum constant. Perform standardization and normalization on the client contribution weights: （9） in, This represents the normalized client contribution weight; To maintain a consistent minimum value throughout the text, and to prevent calculation anomalies caused by weights being zero; This indicates the total number of clients participating in federated learning. Step 2-1-2: Dynamically determine the total number of global aggregated samples based on the computing power and storage limits of the federated cluster, calculate the sampling quota based on the normalized contribution weight, extract samples, and concatenate them to form a global aggregated distillation dataset: ,in For the first Each client sampling quota, The total number of samples in the global aggregation is dynamically determined. Step 2-1-3: Based on a weighted voting mechanism, merge the sparse masks from all clients and perform binarization to generate a globally unified global sparse mask: （10） in, Represents a global sparse mask; Indicates the first Sparse mask of parameters uploaded by each client. Step 2-1-4: Perform weighted aggregation on the single-client level importance scores uploaded by the client to obtain the global aggregated level importance score: .

6. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 5, characterized in that: Step 2-2 specifically includes the following steps: Step 2-2-1: The edge server generates a generalized significance score based on the global aggregation hierarchy importance score and the mask retention rate, where the parameter's retention rate in the global mask is... The ratio of the number of parameters preserved in the global sparse mask of the layer to the total number of parameters in that layer is calculated using the following formula: The weight ratios were determined through a variance experiment on defense effectiveness. The generalization significance score for each parameter was calculated, and the significance scores were subjected to min-max normalization to eliminate differences in numerical units and achieve accurate segmentation of feature regions. （11） （12） in, This represents the original generalization significance score of the parameter; This represents the importance score of the global aggregation hierarchy; This indicates the retention rate of the parameter in the global sparse mask; This represents the generalization significance score after normalization; m is the index for traversing all model parameters; To maintain a consistent minimum value constant throughout the text and avoid fractions with zero denominators, Step 2-2-2: Based on the normalized significance score, a threshold is set to divide the high-generalization core region into a low-generalization redundant region. Orthogonal projection feature cleaning is performed on the high-generalization core region to completely remove backdoor perturbations. ,in, Features after cleaning These are the original aggregation features. The global reference vector is the aggregated feature mean. for L2 norm, Step 2-2-3: Apply adaptive Gaussian noise to the low-generalization redundancy region. The noise intensity is linearly negatively correlated with the generalization significance score. This is used to precisely remove backdoor implanted features. ， in The optimal base noise intensity for robustness.

7. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 6, characterized in that: Steps 2-3 specifically include the following steps: Step 2-3-1: To correct the feature distribution shift caused by the dual-path defense operation, the edge server performs amplitude-limiting generalization fidelity calibration only on samples with generalization confidence scores not lower than a preset threshold, and the calibration intensity coefficient... The range of values ​​is This ensures that the defense effect is not weakened and that the feature distribution is repaired; among which To normalize the generalization confidence scalar, the generalization confidence score from step 1-1-3 is generated through min-max normalization. The element-wise multiplication of the scalar and the feature vector satisfies the dimension matching requirement. （13） in, This represents the final feature data after calibration; This represents the characteristic data after dual-path defense; This represents the fidelity calibration intensity coefficient, with a value range of [value range missing]. ; This represents the normalized generalization confidence scalar; This indicates the global polymer distillation characteristics before defense; This represents element-wise multiplication. Step 2-3-2: The generalization feature distribution rule is composed of the mean, variance, and normalized generalization significance score distribution of the global polymeric distillation features. It is generated by the generalization significance heatmap in Step 2-2 and the calibration feature statistics in Step 2-3-1. The three core data types, namely the calibrated safe distillation dataset, the global sparse mask, and the generalization feature distribution rule, are uniformly synchronized to the cloud server as the sole safe data source for fine-tuning the large cloud model.

8. The method for generalization enhancement sparse federated distillation and security defense fine-tuning for large models according to claim 7, characterized in that: Step 3-1 specifically includes the following steps: Step 3-1-1: Cloud server combines global sparse mask Structured LoRA low-rank adapters are deployed only in layers where the mask parameter retention ratio is not lower than a preset layer threshold. These adapters are used in the self-attention query / key / value / output projection layers and feedforward network gating / up / down projection layers of the multimodal large model. Furthermore, LoRA low-rank updates only affect the mask-retained parameter positions; no adapters are deployed in the mask layer. Only low-rank parameters are updated, and pre-trained backbone parameters are frozen, significantly reducing the computational cost of fine-tuning. （14） in, This represents the model weight matrix after fine-tuning; This represents the original weight matrix of the pre-trained model; This represents the LoRA scaling factor, relative to the Dirichlet parameter. These are independent parameters; A and B represent the low-rank LoRA decomposition matrices. Step 3-1-2: Based on the secure distillation dataset synchronized from the edge server and the generalization feature distribution rules, construct multimodal dialogue format training data containing images, prompts, and labels. Employ an assistant-specific supervision strategy, calculating the loss only on the model's output answer portion and ignoring the user's prompts to improve fine-tuning accuracy. Step 3-1-3: To reduce memory usage during single-step training and adapt to fine-tuning of large models, a gradient accumulation training strategy is adopted, accumulating... After each gradient step, a unified parameter update is performed. The loss per microstep is: （15） in, This represents the loss value for a single microstep; This represents the single-step supervised fine-tuning loss; the total gradient is... The sum of gradients, divided by the step gradients before parameter update. Achieve smooth updates. Step 3-2 specifically includes the following steps: Step 3-2-1: The cloud server loads the test dataset to complete model inference. Based on the inference results, the classification accuracy and invalid output rate are calculated. Simultaneously, a benign test set and a backdoor attack test set are used to test the model's benign accuracy (BA) and backdoor attack success rate (ASR) respectively, comprehensively verifying the model's performance. （16） （17） in, This represents the model's classification accuracy. Indicates the category predicted by the model; Indicates the true category of the sample; This represents the total number of samples in the test set; This indicates the invalid output rate of the model. Step 3-2-2: Merge the fine-tuned structured LoRA adapter with the backbone model, save the complete model file to the specified output directory, and directly output the compliant, deployable, and optimized large model that has undergone generalization enhancement and backdoor removal.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the generalization enhancement sparse federated distillation and security defense fine-tuning method for large models as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement the generalization-enhanced sparse federated distillation and security defense fine-tuning method for large models as described in any one of claims 1-8.

Images