A two-stage federated distillation and large model fine-tuning method based on differential privacy

Abstract

This invention relates to a two-stage federated distillation and large model fine-tuning method based on differential privacy, belonging to the fields of artificial intelligence security, federated learning, and differential privacy technology. The method includes: Step 1, federated pre-training based on complexity adaptive budget management and momentum orthogonal subspace directional noise addition; Step 2, local dataset distillation based on frequency domain-aware complex plane anisotropy noise addition; and Step 3, cloud-based large model fine-tuning and end-to-end privacy compliance assessment based on noise prior-driven adaptive rank. This invention achieves precise allocation and tracking of the privacy budget in both the federated pre-training and data distillation stages by introducing mathematically provable RDP differential privacy protection. It completes high-fidelity generation of the distilled dataset and secure large model fine-tuning while strictly protecting the privacy of the client's original data, which is of great significance for promoting the secure and efficient deployment of large models in distributed privacy-sensitive environments.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence security, federated learning and differential privacy technology, specifically involving a method for fine-tuning a large privacy-preserving model based on two-stage federated distillation and differential privacy. Background Technology

[0002] With the milestone progress made in large-scale pre-trained models in fields such as natural language processing and computer vision, fine-tuning models using high-quality domain data has become a core path to improve the professionalism of artificial intelligence systems in industries such as healthcare and finance.

[0003] To achieve distributed learning without directly sharing raw data, the federated learning paradigm has been introduced into large-scale model fine-tuning scenarios. However, traditional federated learning directly processes model updates with hundreds of billions of parameters, resulting in extremely high communication overhead and computational burden on the endpoints. To address this, federated dataset distillation technology has emerged, which attempts to reduce transmission pressure by generating small-scale synthetic image representations on the client side and uploading these compressed "high-value data" to the cloud for efficient parameter fine-tuning (such as LoRA).

[0004] Despite its significant advantages in efficiency, this framework still faces serious challenges in balancing security and utility. While existing federated distillation mechanisms avoid the physical movement of the original data, attackers can still use intercepted synthetic representations or gradient information to carry out Membership Inference Attacks (MIA) or gradient inversion attacks, thereby precisely stealing sensitive information from the underlying data.

[0005] To address the aforementioned privacy risks, differential privacy (DP) with high-intensity noise injection is currently the mainstream defense method. However, most existing noise-adding strategies employ globally isotropic random perturbations, failing to mathematically distinguish the different contributions of data features to model utility. This indiscriminate noise addition often masks key semantic components of the image, leading to a significant decline in the fidelity of the distilled dataset and directly weakening the generalization accuracy of downstream large models during fine-tuning.

[0006] Furthermore, existing solutions often treat privacy protection and model fine-tuning as two independent processes, lacking a holistic perspective for coordinated optimization. In complex, long-chain tasks, the static allocation of privacy budgets at each stage cannot cope with dynamic computational demands. More importantly, cloud-based fine-tuning processes (such as fixed-rank LoRA) are typically "blind" to privacy noise, making it difficult to adaptively filter high-variance perturbations injected in earlier stages. This can easily lead to the model mistaking privacy noise for valid features and fitting them, further deteriorating its robustness.

[0007] Faced with the aforementioned technical bottlenecks such as privacy leakage risks, noise reduction utility collapse, and lack of cross-stage linkage, there is an urgent need for a systematic solution that can integrate dynamic privacy budget planning, frequency domain feature-aware noise reduction, and noise prior-driven adaptive fine-tuning. This invention is based on this pressing need, aiming to break the zero-sum game between privacy protection and model utility, and provide key technical support for the secure and efficient deployment of large models in distributed environments. Summary of the Invention

[0008] This invention proposes a two-stage federated distillation and large model fine-tuning method based on differential privacy, aiming to effectively solve the challenge of large model fine-tuning in scenarios where multiple clients cannot directly share private data. It overcomes the shortcomings of existing technologies, such as huge communication overhead, the collapse of model generalization ability due to traditional differential privacy, and threats to deep privacy such as member inference attacks (MIA) and gradient inversion. This invention is applicable to scenarios involving large model training and efficient parameter fine-tuning with multiple clients under a federated learning framework, and can support efficient compression and secure aggregation of heterogeneous data under strict privacy constraints. In typical long business processes such as federated model pre-training, local data distillation and core feature extraction, and cloud-based large model security fine-tuning, this invention innovatively adopts a cross-stage dynamic budget coordination mechanism based on Rényi differential privacy (RDP), a dataset distillation mechanism based on spatial-frequency dual-domain decoupling and complex frequency domain anisotropic noise addition, and a noise prior-driven adaptive rank fine-tuning strategy. This breaks the zero-sum game between privacy protection and model utility, and achieves a perfect balance between massive data simplification, strict privacy compliance, and extremely high downstream task fine-tuning fidelity for large models. It significantly improves the security, reliability, and generalization performance of models in distributed heterogeneous environments.

[0009] Edge clients are responsible for the secure distillation of local data and the extraction of core privacy features; edge servers, as federated coordination and security hubs, are responsible for dynamically managing the privacy budget and performing targeted noise aggregation of global pre-trained models; cloud servers are responsible for aggregating distilled data, performing adaptive fine-tuning of large models, and final privacy compliance verification.

[0010] In addition, the method includes three steps: federated pre-training based on complexity adaptive budget planning and momentum orthogonal subspace directional noise addition; local dataset distillation and security representation extraction based on frequency domain-aware complex plane anisotropic noise addition and pure gradient decoupling; and cloud-based large model fine-tuning and end-to-end privacy compliance assessment based on noise prior-driven adaptive rank.

[0011] Step 1: In the federated pre-training phase based on complexity adaptive budget management and momentum orthogonal subspace directional noise addition, the edge server and each client collaborate to securely construct the globally pre-trained model. Specifically, the system first dynamically manages the Rényi Differential Privacy (RDP) budget according to the computational complexity of cross-stage tasks, providing a globally optimal privacy budget allocation for the entire chain of tasks. Each client trains locally using private data and performs strict global threshold pruning on the model weight update amount. Subsequently, during aggregation updates, the edge server uses historical momentum to extract principal component directions representing public knowledge, rigorously decomposes the aggregation gradient, and innovatively applies directional Gaussian noise only to the orthogonal private subspace carrying individual-specific privacy. This phase, while ensuring that the original data never leaves the local machine, accurately removes and protects sensitive information through the momentum orthogonal directional noise addition mechanism. While maximizing the preservation of public semantic features, it provides a reliable pre-trained model with high generalization ability and strong privacy guarantee for the subsequent data distillation phase.

[0012] Preferably, step one mainly includes two steps:

[0013] Step 1: The edge server initializes the globally pre-trained model, performs differential privacy budget allocation based on iterative complexity and utility-priority asymmetric constraints, and plans a constant noise injection amount; each client performs local training and performs global threshold pruning on the model weight updates. The specific process is as follows:

[0014] Edge server initializes global pre-trained model parameters and coordinate Each client participates in federated learning training; the system's preset total differential privacy budget is... Extracting the computational complexity of the pre-training phase and the computational complexity of the data distillation stage ;

[0015] in, The total number of clients participating in federated learning training. The total number of training rounds. For client-side downsampling rate, To optimize the number of rounds, Number of images generated per class This represents the total number of categories;

[0016] The system calculates the dynamic allocation factor. ,in A pre-defined computational complexity balancing coefficient is set for the system, and utility-priority asymmetric constraints are applied to ensure... The budget is positioned within a pre-defined skewed range to ensure that the majority of the budget is allocated to the distillation stage; the pre-training budget is allocated based on the differential privacy serial combination theorem. Differential privacy relaxation item Total number of training rounds Client downsampling rate The minimum constant dimensionless noise multiplier satisfying the Rényi differential privacy (RDP) constraint is pre-programmed through numerical binary search. And convert it into physical noise injection amount ;

[0017] In the During each round of federated learning, the edge server will display the current global model. Distribute to each client;

[0018] Each client Based on local multi-source heterogeneous data The received global model is trained locally; after training, the difference between the local model weight state and the received initial global model weight state is calculated to obtain the cross-layer global pseudo-gradient update amount. :

[0019] (1)

[0020] in, This indicates the ID of the specific client participating in federated learning (values ​​are 1, 2, ...). ); Indicates the first Global model parameters issued by the edge server at the start of a round of federated learning; Indicates the first The model parameters obtained by each client after local training is completed;

[0021] The system calculates the overall global pseudo gradient update amount. Norm, and perform global threshold pruning on the update amount of all layers with the same scaling factor:

[0022] (2)

[0023] in, This is the amount of local updates after trimming; The global gradient pruning threshold preset for the system; The global norm 2 of the global pseudo-gradient update amount; after pruning, each client will update the pruned local update amount. Uploaded to an edge server for subsequent aggregation.

[0024] Step 2: The edge server collects the pruning update data, performs federated aggregation, extracts the principal component directions of the undamped momentum, injects differential privacy Gaussian noise into the orthogonal subspace, updates the global model, and dynamically records the actual privacy consumption. The specific process is as follows:

[0025] Edge servers collect data from participating clients. Uploaded cropped local update volume Federated aggregation is performed using the arithmetic mean to obtain the aggregate update. :

[0026] (3)

[0027] in, This represents the total number of clients participating in this aggregation round.

[0028] Subsequently, the edge server utilizes the undamped momentum buffer state from the previous round. Construct a unit vector representing the main direction of public knowledge. The aggregated update quantity is decomposed into common components parallel to the main direction. Private components orthogonal to the main direction :

[0029] (4)

[0030] (5)

[0031] Dominant Gaussian noise is applied to the orthogonal private component carrying individual-specific privacy features to obtain the directional noise-added update. :

[0032] (6)

[0033] Among them, Gaussian noise matrix , It is the identity matrix. The constant physical noise injection amount pre-planned in step 1;

[0034] The edge server uses a momentum buffer to update the directional noise level. Accumulated smoothing is performed, and the global pre-trained model is updated. Its momentum update mechanism strictly follows the standard form of undamped accumulation:

[0035] (7)

[0036] (8)

[0037] in, The momentum coefficient is preset for the server side, and the initial momentum... Set it to a zero tensor with the same dimension as the model parameters; The updated momentum state; This is for updating the parameters of the globally pre-trained model before entering the next round;

[0038] The process of local training on the client and aggregated updates on the edge server is repeated. After each training round, the edge server uses the Dynamic Rényi Differential Privacy (RDP) accounting mechanism to record the actual accumulated privacy consumption in that round and extracts and saves the updated global model snapshot after that round. ;

[0039] When federal training rounds Reach the preset total number of training rounds Training ends when the time is right, and the final global pre-trained model is obtained. and save all The global model snapshot set is transmitted to the subsequent data distillation stage for distillation initialization by generating diverse teacher model feature extraction states through linear interpolation;

[0040] At the end of the federated pre-training phase, a dynamic adjustment rule for the privacy budget is triggered: the edge servers calculate the actual privacy budget consumed in the first phase through the dynamic Rényi Differential Privacy (RDP) accounting mechanism. If the budget carry-forward conditions are met, the system will allocate the remaining budget balance. Dynamically carry over to the data distillation phase; and the updated second phase after the carryover can utilize the privacy budget. Strictly meet the constraints: To ensure that the total privacy overhead across the entire chain absolutely does not exceed the total budget. This maximizes the utility of feature matching under hard security constraints.

[0041] Step Two: In the local dataset distillation and secure representation extraction stage based on frequency-domain-aware complex plane anisotropic noise addition and clean gradient decoupling, each participating client independently constructs its local secure distillation dataset and initializes the synthesized image through a hybrid interpolation mechanism with spatial-frequency dual-domain decoupling to enhance the feature diversity and structural fidelity of the initial data. Each client then uses a pre-trained model to map the local raw data to the latent space. Under the minimization optimization framework constructed using the Neural Feature Function Difference (NCFD) metric, it calculates and extracts the feature matching gradient in the complex frequency domain. Through innovative implementation of amplitude-phase anisotropic differential privacy noise addition and clean calibration gradient decoupling updates, efficient distributed compression from large-scale raw sensitive data to a high-fidelity small distillation dataset is achieved. This stage, while ensuring that the raw data never leaves the local environment, completely blocks gradient inverse calculation and privacy leakage paths by forcibly executing defensive decoupling operations. This not only significantly reduces the communication and computational overhead of subsequent large model fine-tuning but also provides the main model with a secure core representation that combines high semantic utility with robust privacy defense.

[0042] Preferably, step two mainly includes two steps:

[0043] Step 1: Each client extracts two adjacent snapshots of the pre-trained model and performs linear interpolation to construct a feature extractor, mapping the local raw data to the latent space to extract the true feature distribution; a hybrid interpolation strategy based on spatial-frequency dual-domain decoupling is used to initialize the synthesized image, and a minimization data distribution compression framework based on the neural feature function difference (NCFD) metric is constructed. The specific process is as follows:

[0044] Each client extracts snapshots of two adjacent rounds of the pre-trained model and performs linear interpolation to construct a feature extractor. The feature extractor is used to extract local raw multi-source heterogeneous data. Mapping to the latent space yields the true feature distribution representation. ;in The original data volume, Represents the original dataset The first in One real input sample.

[0045] Client initializes local small distillation dataset ,in This refers to the amount of data after distillation. Represents the distillation dataset The first in A synthetic sample.

[0046] A hybrid interpolation strategy based on space-frequency dual-domain decoupling is employed to initialize the synthesized image tensor: Discrete Fourier transform is performed on the sampled real images to extract the low-frequency amplitude spectrum of the first set of real images and the high-frequency phase spectrum of the second set of real images. Amplitude-phase fusion and inverse Fourier transform are then performed across images to generate the initial synthesized data tensor, and the pixel values ​​are constrained to... Within the range.

[0047] Based on the above feature extraction and initialization results, a minimization-maximum data distribution compression framework based on the Neural Feature Function Difference (NCFD) metric is constructed, with the distribution matching optimization objective being:

[0048] (9)

[0049] in, Represents mathematical expectation. Represents the actual input sample Sampled from the local raw dataset (That is, it follows the true data distribution); Indicates synthetic sample Sampled from locally generated distillation dataset (i.e., follows a synthetic data distribution); The overall representation is the joint mathematical expectation of the overall loss function (including the difference in neural feature functions and the output matching loss) within the square brackets on the real dataset and the distilled dataset; For sampling networks, The neural feature function difference (NCFD) is defined by including the feature difference function. The characteristic difference function Expand into feature maps and In frequency The square of the magnitude of the difference of the empirical characteristic functions under the following conditions:

[0050] (10)

[0051] in, The imaginary unit is used; this function measures the global consistency of the latent space data distribution within the minimax optimization framework. The output matching loss is defined as the KL divergence between the real data and the distilled data on the pre-trained model output distribution:

[0052]

[0053] in, Represents relative entropy. This represents the probability distribution of the model's feature mappings in the semantic space; The weighting coefficient for controlling the proportion of output matching loss has a range of values. .

[0054] Aligning the amplitude and phase information of neural features in the complex plane, the feature function is decomposed into:

[0055] (12)

[0056] in, To use the imaginary unit, an adjustment weight coefficient is introduced. ,and By adjusting the weighting coefficient The balance is determined by the amplitude term. The diversity of control and the phase terms To ensure authenticity, the distilled data is combined with output matching loss to ensure that the semantic space of the distilled data remains consistent with that of the original data.

[0057] Step 2: Under the minimax data distribution compression framework, the client calculates the synthetic data feature matching gradient corresponding to the single-class single-graph (IPC) level, and performs amplitude-phase anisotropic differential privacy noise addition in the complex frequency domain; then, the noisy feature gradient is decoupled and updated from the clean calibration gradient, a defensive disengagement operation is forcibly performed, and a privacy-preserving secure distillation dataset is generated and uploaded. The specific process is as follows:

[0058] The client performs alternating adversarial iterations within the minimax data distribution compression framework; first, the distillation dataset is fixed. By optimizing the auxiliary sampling network To maximize the total loss function:

[0059] (13)

[0060] Fixed sampling network Calculate the differences in synthetic data with respect to neural feature functions. The feature matching gradient is then used to reshape the synthesized image tensor after decoding and dimension expansion into a multidimensional structure containing decoding dimension expansion multipliers, based on the decoding scaling factor. A strict summation pooling operation is then performed on the expanded dimensions to obtain feature matching gradient components that precisely correspond to the single-class single-image (IPC) level. .

[0061] Match the features to gradient components Mapped to the complex plane frequency domain, and performed independently for the gradient component corresponding to each IPC. Norm difference privacy clipping:

[0062] (14)

[0063] in, The feature gradient clipping threshold is the threshold value used in the data distillation stage. Maximize the true feature vector of a single sample Norm Boundary With batch size Mathematically precisely set as ;

[0064] Subsequently, amplitude-phase anisotropic differential privacy noise is implemented in the complex frequency domain: privacy budgeting during the data distillation stage. Below the preset privacy utility threshold At that time, the anisotropy mechanism is triggered, separating the clipped gradient components into amplitude-matched gradients and phase-matched gradients. Variance is independently injected into the amplitude-matched gradient, which represents the macroscopic semantics. Low-intensity Gaussian noise, with an independent injection variance of phase-matched gradients to carry individual-specific privacy. High-intensity Gaussian noise; in satisfying Furthermore, under the premise that the combination satisfies the total budget constraint, the reconstructed noisy feature gradient in the complex frequency domain is obtained. .

[0065] The synthesized image is computed while preserving the forward computation graph without consuming additional privacy budget. Regarding output matching loss Pure calibration gradient The clean calibration gradient, free from noise contamination, is decoupled from the noisy feature gradient and accumulated to complete a single parameter update.

[0066] (15)

[0067] in, To update the step size; and after the update, to force a defensive detach operation on the model parameters that the computation graph depends on, cutting off the backpropagation path of gradient flow to the frozen pre-trained feature extractor, thereby preventing memory overflow and completely blocking the privacy leakage risk caused by using synthetic data gradients to back-infer the internal state of the pre-trained model.

[0068] Repeat the above alternating optimization and decoupled update process until the preset number of distillation rounds is reached, completing the efficient distributed compression of local data, and finally generating a privacy-preserving secure distillation dataset. And upload it to the edge server.

[0069] Step 3: In the cloud-based large model fine-tuning and end-to-end privacy compliance assessment stage based on noise prior-driven adaptive rank, the cloud server independently completes the efficient parameter fine-tuning and global security verification of the large model. It guides the dynamic adaptive compression of the fine-tuning optimization space by analyzing the accumulated differential privacy noise prior variance attached to the aggregated secure distillation data. The cloud server then introduces a low-rank adaptive (LoRA) bypass structure at specific layers of the large model. By constructing an inverse proportional mapping function between the rank dimension and noise intensity, a rank truncation mechanism is triggered. The feature filtering characteristics of the low-rank space are used to suppress the contamination of weights by high-frequency privacy perturbations, achieving secure updates from high-noise distillation data to high-fidelity parameters of the large model. This stage, while providing end-to-end dynamic Rényi differential privacy (RDP) compliance verification and resistance to membership inference attacks (MIA), completely breaks through the dilemma of high-intensity noise addition and model utility collapse, ultimately outputting a reliable large model with both extremely high security defense robustness and excellent downstream task generalization ability.

[0070] Preferably, step three mainly includes two steps:

[0071] Step 1: The cloud server loads the secure distillation dataset, adaptively adjusts the rank dimension of the low-rank matrix based on the prior knowledge of accumulated privacy noise in the received data, and uses efficient parameter fine-tuning techniques to securely fine-tune the pre-trained large model. The specific process is as follows:

[0072] The cloud server loads the received safe distillation dataset. And analyze its associated cumulative difference privacy noise prior variance. Simultaneously initialize the pre-trained large model. Let its original weight parameters be denoted as And keep it frozen during the fine-tuning process.

[0073] in, This refers to the synthetic image data in the safe distillation dataset (i.e., images that have undergone local feature compression and differential privacy noise enhancement). The category labels corresponding to the synthesized image; these two, together as a data pair, constitute a secure distillation dataset for secure fine-tuning of large models in the cloud. .

[0074] A low-rank adaptive (LoRA) bypass structure is introduced into a specific layer of a large model, and the LoRA parameters that need to be learned are denoted as... Based on the aforementioned privacy noise prior variance Adaptively adjust the rank dimension of a low-rank matrix, specifically by constructing the rank dimension. With respect to the prior standard deviation of the privacy noise Monotonically decreasing mapping function Perform adaptive adjustment of rank;

[0075] in, The robustness coefficient is preset for the system. To maintain the minimum rank lower bound of the basic representational power of large models, Indicates a round-down operation; preset noise prior threshold. When the noise intensity obtained from the analysis satisfies And the calculated The value reaches the lower bound At that time, the rank truncation mechanism is forcibly triggered to compress the optimization space, so as to use the feature filtering properties of the low-rank space to suppress the contamination of the large model weights by high variance noise.

[0076] The cloud server performs large-model security fine-tuning based on adaptive rank LoRA, learning LoRA parameters through the following optimization objective function. :

[0077]

[0078] in, To fine-tune the loss function; Represents cross-entropy loss; Indicates based on pre-trained parameters and LoRA parameters For synthetic images The model output.

[0079] After the cloud server completes fine-tuning, the optimized LoRA parameters are obtained. And output the finely tuned large model. Its parameters are and The combination, save The output directory yields a reliable large model that has been purified by secure distillation of the dataset and possesses high privacy and security.

[0080] Step 2: The cloud server aggregates the dynamic privacy accounting overhead from both phases to perform end-to-end compliance verification, and evaluates the performance and defense effectiveness of the fine-tuned large model through multi-dimensional testing, outputting a complete verification report and a deployable model. The specific process is as follows:

[0081] The cloud server performs an end-to-end privacy compliance assessment, summarizing the dynamic Rényi Differential Privacy (RDP) accounting overhead from the two phases of federated pre-training and data distillation, and converting it into a system-level... Consumption metrics are used to verify whether the overall data meets the preset upper bound for privacy and security.

[0082] Cloud servers are used to evaluate model performance using a benign test sample set. The calculated model's balanced accuracy on the original task :

[0083] (17)

[0084] in, The number of samples that are correctly predicted. This represents the total number of test samples.

[0085] The cloud server undergoes a comprehensive security index assessment, incorporating the Defense Performance Ratio (DER) metric.

[0086] (18)

[0087] in, The baseline attack success rate when the model faces member inference attacks; The percentage decrease in benign accuracy compared to the baseline model without differential privacy protection under the same testing environment; The percentage decrease in attack success rate for members compared to the undefended baseline model; a higher DER value indicates that the model has stronger privacy protection capabilities while maintaining high performance fine-tuning utility.

[0088] The cloud server integrates all compliance and testing results to generate a complete assessment report that includes privacy compliance verification; based on the comprehensive assessment results, it determines the optimal model configuration and outputs the final large model for actual deployment.

[0089] The beneficial effects of this invention are:

[0090] (1) This invention proposes a two-stage federated distillation and large model fine-tuning method based on differential privacy. By using RDP budget planning based on complexity adaptation, precise anisotropic noise addition in the complex frequency domain, and an adaptive rank fine-tuning strategy driven by noise prior, this method effectively solves multiple challenges faced by distributed large model fine-tuning, such as strict privacy compliance, end-side computing power bottlenecks, huge communication overhead, and deep privacy leakage. Under the premise of absolutely protecting the original client data from leaving the local machine and satisfying the strict end-to-end privacy security upper bound, this method significantly reduces the communication and computing burden of long-link training, while greatly improving the generalization fidelity of the model in downstream tasks and its robustness against membership inference attacks (MIA). This method has extremely important engineering and theoretical significance for promoting the secure and efficient deployment of large models in privacy-sensitive and data-heterogeneous scenarios.

[0091] (2) The dataset distillation mechanism based on frequency domain awareness and pure gradient decoupling designed in this invention achieves efficient distributed compression from large-scale heterogeneous raw sensitive data to high-fidelity small-scale secure distillation datasets through spatial-frequency dual-domain hybrid interpolation initialization and neural feature function difference (NCFD) measurement. This process not only accurately strips sensitive features and preserves macroscopic semantic diversity by utilizing the complex plane anisotropic noise mechanism (i.e., applying differentiated privacy noise to amplitude and phase), effectively overcoming the utility collapse problem caused by traditional global equal-intensity differential privacy noise, but also fundamentally cuts off the privacy leakage path of gradient inverse calculation and greatly reduces communication overhead by uploading only a very small amount of distillation data and forcibly performing defensive detach operations. This provides the optimal technical path for collaborative optimization of large models in resource-constrained and privacy-critical edge environments.

[0092] (3) This invention innovatively constructs a cross-stage collaborative adaptive privacy defense and model fine-tuning mechanism. In the pre-training stage, it pioneered the use of momentum orthogonal subspace directional noise addition to accurately lock and mask private gradient components carrying individual-specific privacy. In the cloud fine-tuning stage, it introduces an adaptive low-rank rank fine-tuning (LoRA) architecture based on noise prior. By analyzing the differential privacy noise prior variance accumulated in the previous stage, it constructs an inverse proportional mapping function and forcibly triggers a rank truncation mechanism. It utilizes the feature filtering characteristics of the low-rank space to intelligently suppress the pollution of high-intensity privacy perturbations on the weights of the large model. This mechanism breaks the barrier between privacy protection and model optimization in traditional multi-stage tasks. While effectively blocking member inference attacks and significantly reducing the risk of privacy leakage, it adaptively compresses the optimization space to maximize the preservation of the original representational ability and useful semantics of the large model, thus achieving an unprecedented excellent balance between stringent defense effectiveness and the usability of large model fine-tuning.

[0093] This invention effectively breaks down the dual limitations of "data silos" and "computing power barriers." By securely offloading the fine-tuning process of large models with high computing power requirements to the cloud, edge clients (such as medical institutions, financial branches, or IoT terminals) only need to handle lightweight feature extraction and minimal distillation data transmission, greatly reducing the hardware threshold, energy consumption requirements, and network bandwidth deployment costs for distributed participating nodes. Simultaneously, the end-to-end quantifiable and provable differential privacy compliance mechanism provides a solid technical foundation for the commercialization of large models in highly regulated fields such as cross-institutional collaboration and sensitive data analysis. This enables enterprises to legally and compliantly utilize massive amounts of heterogeneous data from multiple parties, empowering the rapid iteration and large-scale application of industry-specific large models, while strictly adhering to current data security and privacy protection regulations. Attached Figure Description

[0094] Figure 1This is a flowchart of a two-stage federated distillation and large model fine-tuning method based on differential privacy proposed in this invention;

[0095] Figure 2 This is a framework diagram of a two-stage federated distillation and large model fine-tuning method based on differential privacy proposed in this invention. Detailed Implementation

[0096] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0097] Example: See Figure 1 , Figure 2 A two-stage federated distillation and large model fine-tuning method based on differential privacy is proposed. The experimental environment is configured as follows: using the nccl distributed backend, with 8 workers in a multi-process architecture, and implementing distributed collaborative computing based on a deep learning framework. Figure 1 and Figure 2 As shown, it specifically includes:

[0098] Step 1-1: Initialize the global pre-trained model parameters on the edge server In this embodiment, a convolutional neural network (ConvNet) with a depth of 3, a width multiplier of 1.0, and equipped with group normalization is used as the basic architecture, and coordinated... Each client participates in federated learning training; the system's preset total differential privacy budget is... Extracting the computational complexity of the pre-training phase and the computational complexity of the data distillation stage ;

[0099] in The total number of training rounds. The client downsampling rate (i.e., randomly selecting 100 clients from 200 clients to participate in the aggregation in each round). To optimize the number of rounds, Number of images generated per class This represents the total number of categories (using the CIFAR-10 dataset).

[0100] The system calculates the dynamic allocation factor. ,in A pre-defined computational complexity balancing coefficient is set for the system, and utility-priority asymmetric constraints are applied to ensure... The budget is positioned within a pre-defined skewed range to ensure that the majority of the budget is allocated to the distillation stage; the pre-training budget is allocated based on the differential privacy serial combination theorem. Differential privacy relaxation item Total number of training rounds Client downsampling rate The minimum constant dimensionless noise multiplier satisfying the Rényi differential privacy (RDP) constraint is pre-programmed through numerical binary search. And convert it into physical noise injection amount ;

[0101] In the During each round of federated learning, the edge server will display the current global model. Distribute to each client;

[0102] Each client Based on local multi-source heterogeneous data The received global model is trained locally for 5 epochs. After training, the difference between the local model weight state and the received initial global model weight state is calculated to obtain the cross-layer global pseudo-gradient update. :

[0103] (1)

[0104] in, Indicates the first Global model parameters issued by the edge server at the start of a round of federated learning; Indicates the first The model parameters obtained by each client after local training is completed;

[0105] The system calculates the overall global pseudo gradient update amount. Norm, and perform global threshold pruning on the update amount of all layers with the same scaling factor:

[0106] (2)

[0107] in, This is the amount of local updates after trimming; The global gradient pruning threshold preset for the system; The global norm 2 of the global pseudo-gradient update amount; after pruning, each client will update the pruned local update amount. Uploaded to an edge server for subsequent aggregation.

[0108] Edge servers collect data from participating clients. Uploaded cropped local update volume Federated aggregation is performed using the arithmetic mean to obtain the aggregate update. :

[0109] (3)

[0110] in, This represents the total number of clients participating in this aggregation round.

[0111] Subsequently, the edge server utilizes the undamped momentum buffer state from the previous round. Construct a unit vector representing the main direction of public knowledge. The aggregated update quantity is decomposed into common components parallel to the main direction. Private components orthogonal to the main direction :

[0112] (4)

[0113] (5)

[0114] Dominant Gaussian noise is applied to the orthogonal private component carrying individual-specific privacy features to obtain the directional noise-added update. :

[0115] (6)

[0116] Among them, Gaussian noise matrix , It is the identity matrix. The constant physical noise injection amount pre-planned in step 1-1;

[0117] The edge server uses a momentum buffer to update the directional noise level. Accumulated smoothing is performed, and the global pre-trained model is updated. Its momentum update mechanism strictly follows the standard form of undamped accumulation:

[0118] (7)

[0119] (8)

[0120] in, The momentum coefficient preset for the server side (in this embodiment, the momentum coefficient is set). ), and initial momentum Set it to a zero tensor with the same dimension as the model parameters; The updated momentum state; This is for updating the parameters of the globally pre-trained model before entering the next round;

[0121] The process of local training on the client and aggregated updates on the edge server is repeated. After each training round, the edge server uses the Dynamic Rényi Differential Privacy (RDP) accounting mechanism to record the actual accumulated privacy consumption in that round and extracts and saves the updated global model snapshot after that round. ;

[0122] When federal training rounds Reach the preset total number of training rounds Training ends when the number of federal global communication rounds is set in this embodiment. ), thus obtaining the final global pre-trained model. and save all The global model snapshot set (20 teacher model snapshots are generated in this embodiment) is transmitted to the subsequent data distillation stage for distillation initialization by generating diverse teacher model feature extraction states through linear interpolation;

[0123] At the end of the federated pre-training phase, a dynamic adjustment rule for the privacy budget is triggered: the edge servers calculate the actual privacy budget consumed in the first phase through the dynamic Rényi Differential Privacy (RDP) accounting mechanism. If the budget carry-forward conditions are met, the system will allocate the remaining budget balance. Dynamically carry over to the data distillation phase; and the updated second phase after the carryover can utilize the privacy budget. Strictly meet the constraints: To ensure that the total privacy overhead across the entire chain absolutely does not exceed the total budget. (In this embodiment, the global core privacy parameter is set to) This maximizes the utility of feature matching under hard security constraints.

[0124] Step 2-1: Each client extracts two adjacent snapshots of the pre-trained model and performs linear interpolation to construct a feature extractor. The feature extractor is used to extract local raw multi-source heterogeneous data. Mapping to the latent space yields the true feature distribution representation. ;in The original data volume, Represents the original dataset The first in One real input sample (in this embodiment, the original data uses the CIFAR-10 dataset, with a total number of categories). Image size is (The actual data loading batch size is set to 1024).

[0125] Client initializes local small distillation dataset ,in This refers to the amount of data after distillation. Represents the distillation dataset The first in One synthetic sample (in this embodiment, the number of images generated per class is set) );

[0126] A hybrid interpolation strategy based on space-frequency dual-domain decoupling is employed to initialize the synthesized image tensor: Discrete Fourier transform is performed on the sampled real images to extract the low-frequency amplitude spectrum of the first set of real images and the high-frequency phase spectrum of the second set of real images. Amplitude-phase fusion and inverse Fourier transform are then performed across images to generate the initial synthesized data tensor, and the pixel values ​​are constrained to... Within the range;

[0127] Based on the above feature extraction and initialization results, a minimization-maximum data distribution compression framework based on the Neural Feature Function Difference (NCFD) metric is constructed, with the distribution matching optimization objective being:

[0128]

[0129] in, Represents mathematical expectation. Represents the actual input sample Sampled from the local raw dataset (That is, it follows the true data distribution); Indicates synthetic sample Sampled from locally generated distillation dataset (i.e., follows a synthetic data distribution); The overall representation is the joint mathematical expectation of the overall loss function (including the difference in neural feature functions and the output matching loss) within the square brackets on the real dataset and the distilled dataset; For sampling networks, The neural feature function difference (NCFD) is defined by including the feature difference function. The characteristic difference function Expand into feature maps and In frequency The square of the magnitude of the difference of the empirical characteristic functions under the following conditions:

[0130] (10)

[0131] in, The imaginary unit is used; this function measures the global consistency of the latent space data distribution within the minimax optimization framework. The output matching loss is defined as the KL divergence between the real data and the distilled data on the pre-trained model output distribution:

[0132]

[0133] in, Represents relative entropy. This represents the probability distribution of the model's feature mappings in the semantic space; To control the weighting coefficient of the output matching loss ratio (in this embodiment, the loss weight is set). );

[0134] Aligning the amplitude and phase information of neural features in the complex plane, the feature function is decomposed into:

[0135] (12)

[0136] in, To use the imaginary unit, an adjustment weight coefficient is introduced. (Loss weights are set in this embodiment) ), through the adjustment of the weighting coefficient The balance is determined by the amplitude term. The diversity of control and the phase terms To ensure authenticity, the distilled data is combined with output matching loss to ensure that the semantic space of the distilled data remains consistent with that of the original data.

[0137] Step 2-2: The client performs alternating adversarial iterations within the minimax data distribution compression framework; first, the distillation dataset is fixed. By optimizing the auxiliary sampling network To maximize the total loss function:

[0138] (13)

[0139] Fixed sampling network Calculate the differences in synthetic data with respect to neural feature functions. Feature matching gradient; for the synthesized image tensor after decoding and dimension expansion, based on the decoding scaling factor (network width multiplier in this embodiment)... The feature matching gradient is reshaped into a multidimensional structure containing decoding dimension expansion multipliers, and a strict summation pooling operation is performed on the dimension expansion, thereby obtaining feature matching gradient components that precisely correspond to the single-class single-graph (IPC) level. ;

[0140] Match the features to gradient components Mapped to the complex plane frequency domain, and performed independently for the gradient component corresponding to each IPC. Norm difference privacy clipping:

[0141] (14)

[0142] in, The feature gradient clipping threshold is the threshold value used in the data distillation stage. Maximize the true feature vector of a single sample Norm Boundary With batch size Mathematically precisely set as ;

[0143] Subsequently, amplitude-phase anisotropic differential privacy noise is implemented in the complex frequency domain: privacy budgeting during the data distillation stage. Below the preset privacy utility threshold At that time, the anisotropy mechanism is triggered, separating the clipped gradient components into amplitude-matched gradients and phase-matched gradients. Variance is independently injected into the amplitude-matched gradient, which represents the macroscopic semantics. Low-intensity Gaussian noise, with an independent injection variance of phase-matched gradients to carry individual-specific privacy. High-intensity Gaussian noise; in satisfying Furthermore, under the premise that the combination satisfies the total budget constraint, the reconstructed noisy feature gradient in the complex frequency domain is obtained. ;

[0144] The synthesized image is computed while preserving the forward computation graph without consuming additional privacy budget. Regarding output matching loss Pure calibration gradient The clean calibration gradient, free from noise contamination, is decoupled from the noisy feature gradient and accumulated to complete a single parameter update.

[0145] (15)

[0146] in, To update the step size (in this embodiment, the pixel learning rate of the synthesized image is set). Configure image momentum coefficient ); and after the update, a defensive detach operation is forced on the model parameters that the computation graph depends on, cutting off the backpropagation path of gradient flow to the frozen pre-trained feature extractor. This prevents memory overflow and completely blocks the privacy leakage risk caused by using synthetic data gradients to reverse-engineer the internal state of the pre-trained model.

[0147] Repeat the above alternating optimization and decoupling update process until the preset number of distillation rounds is reached (in this embodiment, the total number of distillation iterations is set). (This process is repeated several times) to achieve efficient distributed compression of local data, ultimately generating a secure distillation dataset with privacy protection. And upload it to the edge server.

[0148] The cloud server loads the received safe distillation dataset. And analyze its associated cumulative difference privacy noise prior variance. Simultaneously initialize the pre-trained large model. Let its original weight parameters be denoted as And keep it frozen during fine-tuning;

[0149] in, This refers to the synthetic image data in the safe distillation dataset (i.e., images that have undergone local feature compression and differential privacy noise enhancement). The category labels corresponding to the synthesized image; these two, together as a data pair, constitute a secure distillation dataset for secure fine-tuning of large models in the cloud. .

[0150] A low-rank adaptive (LoRA) bypass structure is introduced into a specific layer of a large model, and the LoRA parameters that need to be learned are denoted as... Based on the aforementioned privacy noise prior variance Adaptively adjust the rank dimension of a low-rank matrix, specifically by constructing the rank dimension. With respect to the prior standard deviation of the privacy noise Monotonically decreasing mapping function Perform adaptive adjustment of rank;

[0151] in, The robustness coefficient preset for the system (set in this embodiment) ), To maintain the basic representational power of the large model, a minimum rank lower bound is set (in this embodiment, a basic minimum rank is defined). , Indicates a round-down operation; preset noise prior threshold. When the noise intensity obtained from the analysis satisfies And the calculated The value reaches the lower bound At that time, the rank truncation mechanism is forcibly triggered to compress the optimization space, so as to use the feature filtering properties of the low-rank space to suppress the contamination of the large model weights by high variance noise.

[0152] The cloud server performs large-model security fine-tuning based on adaptive rank LoRA, learning LoRA parameters through the following optimization objective function. :

[0153]

[0154] in, To fine-tune the loss function; Represents cross-entropy loss; Indicates based on pre-trained parameters and LoRA parameters For synthetic images The model output.

[0155] After the cloud server completes fine-tuning, the optimized LoRA parameters are obtained. And output the finely tuned large model. Its parameters are and The combination, save The output directory yields a reliable large model that has been purified by secure distillation of the dataset and possesses high privacy and security.

[0156] Step 3-2: The cloud server performs an end-to-end privacy compliance assessment, summarizing the Dynamic Rényi Differential Privacy (RDP) accounting overhead from the two phases of federated pre-training and data distillation, and converting it into a system-level... Consumption metrics are used to verify whether the overall data meets the preset privacy and security upper bound (in this embodiment, privacy parameters are strictly limited by...). and );

[0157] Cloud servers are used to evaluate model performance using a benign test sample set. The calculated model's balanced accuracy on the original task :

[0158] (17)

[0159] in, The number of samples that are correctly predicted. This represents the total number of test samples;

[0160] The cloud server undergoes a comprehensive security index assessment, incorporating the Defense Performance Ratio (DER) metric.

[0161] (18)

[0162] in, The baseline attack success rate when the model faces member inference attacks; The percentage decrease in benign accuracy compared to the baseline model without differential privacy protection under the same testing environment; The percentage decrease in member inference attack success rate compared to the undefended baseline model; a higher DER value indicates that the model has stronger privacy protection capabilities while maintaining high performance fine-tuning utility.

[0163] The cloud server integrates all compliance and testing results to generate a complete assessment report that includes privacy compliance verification; based on the comprehensive assessment results, it determines the optimal model configuration and outputs the final large model for actual deployment.

[0164] To verify the privacy defense effectiveness and downstream task model performance of the proposed differential privacy-based two-stage federated distillation and large model fine-tuning method under different privacy budget constraints, the following experiments were designed and executed.

[0165] The experiment utilizes a deep learning framework and distributed computing architecture for end-to-end verification and performance evaluation. Janus-Pro-1B is used as the cloud-based pre-trained large language model, and a low-rank adaptive (LoRA) bypass structure is introduced into specific layers of this large model. Efficient parameter fine-tuning techniques are employed to update the model weights. For comparison, a configuration without distillation-stage differential privacy noise injection (total budget 8.0, with distillation privacy budget of 0.0) serves as the baseline; configurations applying different intensities of differential privacy noise (total budgets 12.0 and 16.0) are used to verify the security defense features of this invention.

[0166] The evaluation test results of the model accuracy after fine-tuning are shown in Table 1 for the different privacy budget allocations and noise addition conditions mentioned above:

[0167] Table 1

[0168]

[0169] Based on the experimental results in Table 1, the following is a detailed analysis of the intrinsic technical correlation between the stage allocation of the differential privacy budget, the noise figure, and the accuracy of large model fine-tuning:

[0170] First, the privacy budget configuration and noise injection intensity directly constrain the final utility of large model fine-tuning. When the system's total privacy budget is configured as 8.0, and all of it is allocated to the pre-training stage (i.e., the distilled privacy budget is 0.0, corresponding to an injected noise coefficient σ of 0), the fine-tuned model achieves a maximum accuracy of 90.04%. This set of data establishes the upper limit of baseline utility without perturbation of synthetic data, indicating that the model itself possesses excellent fine-tuning fitting capabilities.

[0171] Secondly, when the system fully introduces a high-intensity differential privacy protection mechanism during the distillation stage, the model performance exhibits a reasonable and expected performance degradation as the budget tightens. Specifically, when the total privacy budget is set to 12.0 (pre-training privacy budget 8.0, distillation privacy budget 4.0), the system needs to inject high-intensity perturbations into the feature matching gradient, requiring a noise coefficient σ as high as 3.76. Despite being severely contaminated by high-variance noise, the large model still maintains a fine-tuning accuracy of 67.49% after suppressing noise through the feature filtering characteristics of low-rank space using the mechanism unique to this invention. When the total privacy budget is moderately relaxed to 16.0 (pre-training budget remains at 8.0, distillation privacy budget increases to 8.0), the system's tolerable privacy consumption increases, causing the required injected noise coefficient σ to decrease significantly to 2.1973, improving data fidelity. Correspondingly, the model accuracy after fine-tuning recovers to 68.08%.

[0172] The experimental data above fully demonstrates that the method of this invention can achieve precise quantitative linkage between noise injection intensity and system privacy consumption through cross-stage dynamic calculation and reasonable allocation of privacy budget. While implementing rigorous end-to-end compliance assessments, the system ensures that the model fine-tuning accuracy exhibits controllable dynamic changes with the strictness of the privacy budget. This breaks down the technical barriers between deep privacy protection and model usability in practical applications, achieving a highly controllable system-level balance between the two.

[0173] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A two-stage federated distillation and large model fine-tuning method based on differential privacy, characterized in that, Includes the following steps: Step 1: Federated pre-training based on complexity adaptive budget planning and momentum orthogonal subspace directional noise addition; Step 2: Distillation of the local dataset based on frequency domain-aware complex plane anisotropic noise addition; Step 3: Fine-tuning of the cloud-based large model based on noise prior-driven adaptive rank and end-to-end privacy compliance assessment; Step one includes the following steps: Step 1-1: The edge server initializes the global pre-trained model, performs differential privacy budget allocation based on iterative complexity and utility-priority asymmetric constraints, and plans a constant noise injection amount; each client performs local training and performs global threshold pruning on the model weight update amount; Steps 1-2: The edge server collects the pruning update amount, performs federated aggregation, extracts the principal component direction of the undamped momentum, injects differential privacy Gaussian noise into the orthogonal subspace, updates the global model, and dynamically records the actual privacy consumption. Step two includes the following steps: Step 2-1: Each client extracts two adjacent snapshots of the pre-trained model and performs linear interpolation to build a feature extractor, mapping the local raw data to the latent space to extract the true feature distribution; a hybrid interpolation strategy based on spatial-frequency dual-domain decoupling is used to initialize the synthesized image, and a maximum-minimum data distribution compression framework based on the neural feature function difference NCFD metric is constructed. Step 2-2: Under the maximum-minimum data distribution compression framework, the client calculates the synthetic data feature matching gradient corresponding to the IPC level of a single class and a single graph, and performs amplitude-phase anisotropic differential privacy noise addition in the complex frequency domain; then, the noise-added feature gradient is decoupled and updated from the clean calibration gradient, a defensive disengagement operation is forcibly performed, and a privacy-protected secure distillation dataset is generated and uploaded. Step three includes the following steps: Step 3-1: Load the secure distillation dataset onto the cloud server, adaptively adjust the rank dimension of the low-rank matrix based on the prior of accumulated privacy noise from the received data, and use efficient parameter fine-tuning techniques to securely fine-tune the pre-trained large model; Step 3-2: The cloud server aggregates the dynamic privacy accounting overhead of the two stages to perform end-to-end compliance verification, and evaluates the performance and defense effectiveness of the fine-tuned large model through multi-dimensional testing, and outputs a complete verification report and a deployable model.

2. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 1, characterized in that, Step 1-1 specifically includes: Step 1-1-1: Initialize the global pre-trained model parameters on the edge server and coordinate Each client participates in federated learning training; the system's preset total differential privacy budget is... Extracting the computational complexity of the pre-training phase and the computational complexity of the data distillation stage ; in, The total number of clients participating in federated learning training. The total number of training rounds. For client-side downsampling rate, To optimize the number of rounds, Number of images generated per class This represents the total number of categories; The system calculates the dynamic allocation factor. ,in A pre-defined computational complexity balancing coefficient is set for the system, and utility-priority asymmetric constraints are applied to ensure... The budget is positioned within a pre-defined skewed range to ensure that the majority of the budget is allocated to the distillation stage; the pre-training budget is allocated based on the differential privacy serial combination theorem. Differential privacy relaxation item Total number of training rounds Client downsampling rate The minimum constant dimensionless noise multiplier satisfying the Rényi differential privacy RDP constraint is pre-programmed through numerical binary search. And convert it into physical noise injection amount ; Step 1-1-2: In the first During each round of federated learning, the edge server will display the current global model parameters. Distribute to each client; Step 1-1-3: Each client Based on local multi-source heterogeneous data The received global model is trained locally; after training, the difference between the local model weight state and the received initial global model weight state is calculated to obtain the cross-layer global pseudo-gradient update amount. : （1） in, This represents the ID of the specific client participating in federated learning, with values ​​ranging from 1, 2, ... ; Indicates the first Global model parameters issued by the edge server at the start of a round of federated learning; Indicates the first The model parameters obtained by each client after local training is completed; Step 1-1-4: The system calculates the overall global pseudo-gradient update. Norm, and perform global threshold pruning on the update amount of all layers with the same scaling factor: （2） in, This is the amount of local updates after trimming; The global gradient pruning threshold preset for the system; The global norm 2 of the global pseudo-gradient update is used; after pruning, each client will update its local data with the pruned values. Uploaded to an edge server for subsequent aggregation.

3. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 2, characterized in that, Steps 1-2 specifically include: Step 1-2-1: The edge server collects data from each participating client. Uploaded cropped local update volume Federated aggregation is performed using the arithmetic mean to obtain the aggregate update. : （3） in, This represents the total number of clients participating in this aggregation round. Subsequently, the edge server utilizes the undamped momentum buffer state from the previous round. Construct a unit vector representing the main direction of public knowledge. The aggregated update quantity is decomposed into common components parallel to the main direction. Private components orthogonal to the main direction : （4） （5） Dominant Gaussian noise is applied to the orthogonal private component carrying individual-specific privacy features to obtain the directional noise-added update. : （6） Among them, Gaussian noise matrix , It is the identity matrix. The constant physical noise injection amount pre-planned in step 1-1; Step 1-2-2: The edge server updates the directional noise level using the momentum buffer. Accumulated smoothing is performed, and the global pre-trained model is updated. Its momentum update mechanism strictly follows the standard form of undamped accumulation: （7） （8） in, The momentum coefficient is preset for the server side, and the initial momentum... Set it to a zero tensor with the same dimension as the model parameters; The updated momentum state; This is for updating the parameters of the globally pre-trained model before entering the next round; Steps 1-2-3: Repeat the client-side local training and edge server aggregation update process. After each training round, the edge server uses the dynamic Rényi differential privacy RDP accounting mechanism to record the actual accumulated privacy consumption in that round and extracts and saves the updated global model snapshot after that round. ; Steps 1-2-4: During the federal training rounds Reach the preset total number of training rounds Training ends when the time is right, and the final global pre-trained model is obtained. and save all The global model snapshot set is transmitted to the subsequent data distillation stage for distillation initialization by generating diverse teacher model feature extraction states through linear interpolation; At the end of the federated pre-training phase, a dynamic adjustment rule for the privacy budget is triggered: the edge server calculates the actual privacy budget consumed in the first phase through the dynamic Rényi differential privacy RDP accounting mechanism. If the budget carry-forward conditions are met, the system will allocate the remaining budget balance. Dynamically carry over to the data distillation phase; and the updated second phase after the carryover can utilize the privacy budget. Strictly meet the constraints: To ensure that the total privacy overhead across the entire chain absolutely does not exceed the total budget. This maximizes the utility of feature matching under hard security constraints.

4. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 3, characterized in that, Step 2-1 specifically includes the following steps: Step 2-1-1: Each client extracts two adjacent snapshots of the pre-trained model and performs linear interpolation to construct a feature extractor. ; Using feature extractors to extract local raw multi-source heterogeneous data Mapping to the latent space yields the true feature distribution representation. ;in The original data volume, Represents the original dataset The first in One real input sample; Step 2-1-2: Client initializes local small distillation dataset ,in This refers to the amount of data after distillation. Represents the distillation dataset The first in One synthetic sample; A hybrid interpolation strategy based on space-frequency dual-domain decoupling is employed to initialize the synthesized image tensor: Discrete Fourier transform is performed on the sampled real images to extract the low-frequency amplitude spectrum of the first set of real images and the high-frequency phase spectrum of the second set of real images. Amplitude-phase fusion and inverse Fourier transform are then performed across images to generate the initial synthesized data tensor, and the pixel values ​​are constrained to... Within the range; Step 2-1-3: Based on the feature extraction and initialization results, construct a minimization-maximum data distribution compression framework based on the NCFD metric of neural feature function differences. Its distribution matching optimization objective is: （9）， in, Represents the mathematical expectation; Represents the actual input sample Sampled from the local raw dataset That is, it follows the true data distribution; Indicates synthetic sample Sampled from locally generated distillation dataset That is, it follows the distribution of synthetic data; The overall representation is the joint mathematical expectation of the overall loss function within the square brackets, which includes the difference in neural feature functions and the output matching loss, calculated on the real dataset and the distilled dataset. For sampling networks, The neural feature function difference (NCFD) is defined by including the feature difference function. ; Characteristic difference function Expand into feature maps and In frequency The square of the magnitude of the difference of the empirical characteristic functions under the following conditions: （10） in, The imaginary unit is used; this function measures the global consistency of the latent space data distribution within the minimax optimization framework. The output matching loss is defined as the KL divergence between the real data and the distilled data on the pre-trained model output distribution: （11） in, Represents relative entropy. This represents the probability distribution of the model's feature mappings in the semantic space; The weighting coefficient for controlling the proportion of output matching loss has a range of values. ; Step 2-1-4: Align the amplitude and phase information of the neural features in the complex plane, and decompose the feature function into: （12） in, To use the imaginary unit, an adjustment weight coefficient is introduced. ,and By adjusting the weighting coefficient The balance is determined by the amplitude term. The diversity of control and the phase terms To ensure authenticity, the distilled data is combined with output matching loss to ensure that the semantic space of the distilled data remains consistent with that of the original data.

5. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 4, characterized in that, Step 2-2 specifically includes the following steps: Step 2-2-1: The client executes an alternating adversarial iteration within a minimax data distribution compression framework; first, the distillation dataset is fixed. By optimizing the auxiliary sampling network To maximize the total loss function: （13） Step 2-2-2: Fix the sampling network Calculate the differences in synthetic data with respect to neural feature functions. The feature matching gradient is obtained by reshaping the feature matching gradient into a multidimensional structure containing decoding expansion multipliers based on the decoding scaling factor, and strictly performing a summation pooling operation on the expansion dimension to obtain the feature matching gradient components that accurately correspond to the IPC level of a single class and a single image. ; Step 2-2-3: Match features with gradient components Mapped to the complex plane frequency domain, and performed independently for the gradient component corresponding to each IPC. Norm difference privacy clipping: （14） in, The feature gradient clipping threshold is used during the data distillation stage. Maximize the true feature vector of a single sample Norm Boundary With batch size Mathematically precisely set as ; Subsequently, amplitude-phase anisotropic differential privacy noise is implemented in the complex frequency domain: privacy budgeting during the data distillation stage. Below the preset privacy utility threshold At that time, the anisotropy mechanism is triggered, separating the clipped gradient components into amplitude-matched gradients and phase-matched gradients. Variance is independently injected into the amplitude-matched gradient, which represents the macroscopic semantics. Low-intensity Gaussian noise, with an independent injection variance of phase-matched gradients to carry individual-specific privacy. High-intensity Gaussian noise; in satisfying Furthermore, under the premise that the combination satisfies the total budget constraint, the reconstructed noisy feature gradient in the complex frequency domain is obtained. ; Step 2-2-4: Without consuming additional privacy budget, preserve the forward computation graph and compute the synthesized image. Regarding output matching loss Pure calibration gradient The clean calibration gradient, free from noise contamination, is decoupled from the noisy feature gradient and accumulated to complete a single parameter update. （15） in, To update the step size; and after the update, to force a defensive detach operation on the model parameters that the computation graph depends on, cutting off the backpropagation path of gradient flow to the frozen pre-trained feature extractor; Step 2-2-5: Repeat the above alternating optimization and decoupling update process until the preset number of distillation rounds is reached, completing the efficient distributed compression of local data and finally generating a secure distillation dataset with privacy protection. And upload it to the edge server.

6. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 5, characterized in that, Step 3-1 specifically includes the following steps: Step 3-1-1: The cloud server loads the received safe distillation dataset. And analyze its associated cumulative difference privacy noise prior variance. Simultaneously initialize the pre-trained large model. Let its original weight parameters be denoted as And keep it frozen during fine-tuning; in, The synthetic image data in the safe distillation dataset consists of images that have undergone local feature compression and differential privacy noise enhancement. The category labels corresponding to the synthesized image; these two, together as a data pair, constitute a secure distillation dataset for secure fine-tuning of large models in the cloud. , Step 3-1-2: Introduce a low-rank adaptive LoRA bypass structure in a specific layer of the large model. The LoRA parameters to be learned are denoted as... Based on privacy noise prior variance Adaptively adjust the rank dimension of a low-rank matrix, specifically by constructing the rank dimension. Prior standard deviation of privacy noise Monotonically decreasing mapping function Perform adaptive adjustment of rank; in, The robustness coefficient is preset for the system. To maintain the minimum rank lower bound of the basic representational power of large models, Indicates a round-down operation; preset noise prior threshold. When the noise intensity obtained from the analysis satisfies And the calculated The value reaches the lower bound At that time, the rank truncation mechanism is forcibly triggered to compress the optimization space, so as to use the feature filtering properties of the low-rank space to suppress the contamination of the large model weights by high variance noise. Step 3-1-3: The cloud server performs large model security fine-tuning based on adaptive rank LoRA, learning LoRA parameters through the following optimization objective function. : in, To fine-tune the loss function; Represents cross-entropy loss; Indicates based on pre-trained parameters and LoRA parameters For synthetic images The model output; Step 3-1-4: After the cloud server completes fine-tuning, the optimized LoRA parameters are obtained. And output the finely tuned large model. Its parameters are and The combination, save The output directory yields a reliable large model that has been purified by secure distillation of the dataset and possesses high privacy and security.

7. The two-stage federated distillation and large model fine-tuning method based on differential privacy according to claim 6, characterized in that, Step 3-2 specifically includes the following steps: Step 3-2-1: The cloud server performs an end-to-end privacy compliance assessment, summarizing the dynamic Rényi differential privacy RDP accounting overhead from the two phases of federated pre-training and data distillation, and converting it into a system-level... Consumption metrics are used to verify whether the overall data meets the preset upper bound for privacy and security. Step 3-2-2: Evaluate model performance on the cloud server using a benign test sample set. The calculated model's balanced accuracy on the original task : （17） in, The number of samples that are correctly predicted. This represents the total number of test samples; Step 3-2-3: Conduct a comprehensive security index assessment of the cloud server, introducing the Defense Performance Ratio (DER) metric: （18） in, This represents the baseline attack success rate when the model faces member inference attacks. The percentage decrease in benign accuracy compared to the baseline model without differential privacy protection under the same testing environment; The percentage decrease in member inference attack success rate compared to the undefended baseline model; a higher DER value indicates that the model has stronger privacy protection capabilities while maintaining high performance fine-tuning utility. Step 3-2-4: The cloud server integrates all compliance and test results to generate a final, complete assessment report that includes privacy compliance verification; based on the comprehensive assessment results, the optimal model configuration is determined, and the final large model is output for actual deployment.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a two-stage federated distillation and large model fine-tuning method based on differential privacy as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, the computer instructions implement a two-stage federated distillation and large model fine-tuning method based on differential privacy as described in any one of claims 1-7.

Images