A large model fine-tuning method based on generative distillation and cross-modal semantic verification

Abstract

The application discloses a large model fine-tuning method based on generative distillation and cross-modal semantic verification, and belongs to the technical field of large model security and federal edge collaborative learning, and comprises the following steps: completing generative training with semantic pre-alignment and latent space compactness constraint locally on a client side, stripping original sensitive information and outputting synthetic distillation images and synthetic labels, uploading the synthetic distillation images and the synthetic labels to an edge server; performing image-text similarity calculation, threshold screening and backdoor sample elimination on the aggregated distillation data, generating data credibility labels and uploading the purified data to the cloud; performing dynamic rank allocation and directional forgetting fine-tuning on the cloud server, performing positive correlation rank adaptive parameter updating according to the data credibility, introducing a forgetting term to offset residual backdoor features for low credibility data, and realizing large model security fine-tuning and backdoor defense; and the application can improve the privacy security, data reliability and defense robustness of large model fine-tuning in a distributed edge scene.

Description

Technical Field

[0001] This invention belongs to the field of large model security and federated edge collaborative learning technology, specifically involving a large model fine-tuning method based on generative distillation and cross-modal semantic verification. Background Technology

[0002] In recent years, multimodal large-scale models centered on graph-text joint understanding have developed rapidly. Leveraging core capabilities such as cross-modal reasoning, content generation, and semantic matching, they have achieved large-scale applications in numerous privacy-sensitive vertical fields, including smart healthcare, intelligent finance, industrial quality inspection, intelligent security, and smart cities. To adapt general-purpose pre-trained multimodal large-scale models to the business logic and data characteristics of specific scenarios, fine-tuning the model based on scenario-specific private domain graph-text data is a core technical step in enhancing the model's scenario adaptability, improving task inference accuracy, and unlocking the model's practical value. This type of private domain data is independently held by different enterprises, institutions, and end-users, encompassing key information such as user behavior data, business-specific samples, and core technical materials. Its data privacy and security attributes are prominent. Against the backdrop of increasingly stringent global data privacy regulations, the cross-domain transmission, centralized aggregation, and direct sharing of private domain data are severely restricted. Data silos have become a core bottleneck hindering large-scale models from acquiring private domain-specific knowledge and upgrading their generalization capabilities for vertical scenarios.

[0003] However, conducting distributed fine-tuning of multimodal large-scale models under the rigid constraint of keeping the original private domain data within its domain faces several intractable technical challenges: First, the multimodal large-scale model has a massive number of parameters, while edge clients have limited computing power, storage, and bandwidth resources. Directly performing full-parameter fine-tuning would generate extremely high computational load and communication overhead, failing to meet the efficiency and cost requirements of actual deployment. Second, the openness and decentralized nature of the distributed fine-tuning architecture make it highly vulnerable to malicious backdoor attacks. Attackers can implant hidden triggers in local private domain data and transmit malicious features to the global model through data distillation, causing abnormal model output and disrupting business logic. Third, the synthetic samples generated by distilling the original private domain data may have problems such as image-text semantic separation, feature distribution distortion, and uneven quality, which cannot provide stable and reliable data support for fine-tuning of large models; Fourth, the existing efficient parameter fine-tuning strategies lack dynamic adaptive adjustment capabilities, cannot implement differentiated optimization for data with different credibility and security levels, and are difficult to balance model task accuracy and backdoor defense effectiveness; Fifth, privacy protection and model security defense are mutually exclusive, and the existing solutions cannot achieve a coordinated balance between ensuring that the original private domain data does not leave the domain, that the distilled data is secure and reliable, that the large model is efficiently fine-tuned, and that backdoor attacks are actively defended.

[0004] To address the challenge of fine-tuning large models in scenarios where original private data does not leave the domain, the industry has conducted extensive research in areas such as distributed learning, privacy protection, and model security, resulting in four mainstream technical solutions: The first is federated learning, which achieves collaborative model optimization without sharing original data by training locally on the client, uploading model updates, and aggregating them globally on the server; the second is generative data distillation, which uses generative adversarial networks (GANs) to transform original private data into desensitized synthetic data, completing private knowledge transfer while protecting privacy; the third is efficient parameter fine-tuning, with lightweight fine-tuning techniques such as AdaLoRA reducing trainable parameters through low-rank adaptation, thus reducing the resource consumption of fine-tuning large models; and the fourth is backdoor defense, which performs anomaly detection and risk screening on the server side for model updates and training data, resisting backdoor attacks through feature removal, noise addition, and other methods.

[0005] While existing technologies can partially address the issues in single-stage applications, they all exhibit significant limitations in practical applications of private domain fine-tuning for multimodal large models, failing to meet the technical requirements of complex scenarios: Traditional federated learning struggles to adapt to large models with hundreds of billions of parameters, high-frequency model updates create immense communication pressure, and it cannot handle the issue of non-independent and identically distributed client data, easily leading to global model accuracy shifts and decreased generalization ability; Generative distillation based solely on GANs focuses only on fitting the feature distribution of synthetic and original data, without introducing a cross-modal semantic consistency verification mechanism, making synthetic samples prone to semantic misalignment between text and images, and unable to filter backdoor features implanted by malicious clients, directly using them for fine-tuning would introduce serious security risks; Efficient parameter fine-tuning methods such as AdaLoRA employ fixed-rank parameter update strategies, failing to combine data credibility and security level to achieve dynamic rank allocation, resulting in wasted computational resources and an inability to accurately suppress backdoor features in low-credibility data; Existing backdoor defenses are mostly passive detection on the server side, unable to perform pre-emptive security screening of distillation data, and some defense schemes achieve security protection by adding noise, but this significantly reduces the model's task accuracy and usability. In summary, none of the existing solutions can integrate privacy distillation, semantic verification, adaptive fine-tuning, and backdoor defense into a collaborative closed loop, making it difficult to achieve the optimal balance between privacy protection, data security, model performance, and defense capabilities.

[0006] In summary, the industry currently lacks an integrated solution that simultaneously satisfies private domain data privacy compliance, controllable semantics of synthetic distillation data, proactive defense against backdoor attacks, and efficient adaptive fine-tuning of large models. Significant technical bottlenecks remain in the secure fine-tuning of multimodal large models in privacy-sensitive vertical scenarios. Therefore, there is an urgent need to propose an innovative method that deeply integrates generative distillation, cross-modal semantic verification, and adaptive fine-tuning to systematically address the core technical challenges of coordinating privacy protection, data security, model efficiency, and defense robustness in large model fine-tuning within distributed private domain scenarios. Summary of the Invention

[0007] This invention proposes a large model fine-tuning method based on generative distillation and cross-modal semantic verification, aiming to effectively solve the problem of multimodal large model fine-tuning in scenarios where original private domain data cannot be shared among multiple clients. It overcomes the shortcomings of existing technologies, such as high privacy leakage risk, semantic fragmentation of distilled data, difficulty in actively defending against backdoor attacks, and low fine-tuning efficiency and adaptability. It is suitable for multimodal large model training and fine-tuning scenarios with multi-client collaboration under federated learning frameworks, and can support efficient distillation, semantic purification, and adaptive secure fine-tuning of image-text multimodal data under the security constraint of not leaving the private domain.

[0008] This invention is executed collaboratively by a three-tier architecture consisting of a client, an edge server, and a cloud server. The client is responsible for local GAN ​​latent space privacy distillation and synthetic data export. The edge server is responsible for aggregating the synthetic data and performing semantic verification of the synthetic data, risk sample removal, and credibility labeling based on the CLIP model. The cloud server is responsible for implementing AdaLoRA dynamic rank allocation, backdoor feature forgetting learning, and global model adaptive updates based on data credibility. Through a collaborative mechanism across the entire process, a comprehensive improvement in privacy protection, data security, model performance, and defense capabilities is achieved.

[0009] Furthermore, the method of this invention comprises three core stages: a private domain data desensitization generation stage based on GAN latent space privacy distillation; a distillation data security purification stage based on CLIP cross-modal semantic verification; and a large model security fine-tuning and backdoor defense stage based on AdaLoRA dynamic rank allocation and forgetting learning. Specifically, it includes the following steps:

[0010] Step 1: In the private domain data desensitization generation stage based on GAN latent space privacy distillation, each client initiates the local distillation process. The client trains a lightweight generative adversarial network (GAN) with semantic pre-alignment based on local private domain data, transforming pixel-level optimization of the private data into latent space optimization with latent space compactness constraints. Through joint iterative optimization using adversarial loss, feature matching loss, CLIP semantic pre-constraint loss, and latent space compactness constraint loss, synthetic distilled data that closely approximates the feature distribution and semantic features of the real private data is generated. The synthetic distilled image and synthetic label are exported and uploaded to the edge server. This process completes private domain knowledge extraction and desensitized data upload without leaving the original private domain, thus eliminating the risk of privacy leakage at the source.

[0011] Preferably, step one mainly includes two steps:

[0012] Step 1: Each client independently starts the distillation process locally. Based on the globally preset total number of synthetic samples and the total number of clients participating in the training, the number of synthetic samples required to be generated by a single client is determined. The specific process is as follows:

[0013] Each client initiates its local distillation process; the edge server does not participate in local client training. Each client loads its local private data according to its own identity, thus constructing its local private dataset. The local private dataset contains normal private domain samples and malicious samples that may be implanted with backdoors;

[0014] Based on the globally preset total number of synthetic samples and the total number of clients participating in training, the number of synthetic samples required per client is determined according to an equal allocation rule. The allocation of synthetic sample quantity adopts the following formula:

[0015] (1),

[0016] in, Indicates the first The number of synthetic samples required to be generated for each client; This indicates the total number of synthetic samples required globally. This represents the total number of clients participating in the federated distillation training; This indicates the floor function; This indicates an indicator function; it takes the value 1 when the condition inside the parentheses is true and 0 when the condition is false. This represents the modulo operation between the total number of global synthetic samples and the total number of clients.

[0017] Step 2: The client trains a lightweight Generative Adversarial Network (GAN) with semantic pre-alignment locally using private data. Pixel-level optimization of the private data is transformed into latent space optimization with latent space compactness constraints. Based on joint iterations of adversarial loss, feature matching loss, CLIP semantic pre-constraint loss, and latent space compactness constraint loss, the latent vectors and generator weights are optimized so that the feature distribution and semantic features of the generated image closely approximate the distribution of the real private data. The client discards the original private data and the GAN discriminator, exports the synthetic distilled image, synthetic labels, and latent vector data, and uploads the synthetic distilled image and synthetic labels to the edge server, ensuring that the original private domain data remains within its domain. The specific process is as follows:

[0018] The client initializes a lightweight generative adversarial network distillation model with semantic pre-alignment, the distillation model containing a generator. Discriminator With latent vector library Real private data samples With the generator using latent vectors The generated synthetic sample Input the data into the discriminator and simultaneously calculate the discriminant loss and semantic pre-constraint loss.

[0019] The client uses an adaptive gradient descent algorithm to iteratively update the discriminator model parameters based on the discriminator loss. The discriminator parameter update uses the following formula:

[0020] (2),

[0021] in, This represents the trainable parameters of the discriminator model; This represents the discriminator's training learning rate; This represents the gradient of the discriminator loss function with respect to the discriminator parameters;

[0022] The client calculates the generator adversarial loss, generator feature matching loss, CLIP semantic pre-constraint loss, and latent space compactness constraint loss respectively, and then weights and fuses the four types of losses to obtain the total training loss of the generator.

[0023] The generator adversarial loss is expressed by the following formula:

[0024] (3),

[0025] in, This indicates that the generator is adversarial against loss;

[0026] The generator feature matching loss is expressed by the following formula:

[0027] (4),

[0028] in, This represents the generator feature matching loss; This indicates the total number of feature layers output by the discriminator; Indicates the discriminator's first... The layer's feature output results; This represents the expected value calculation of the feature data; Represents L1 norm operations;

[0029] CLIP semantic pre-constraint loss is calculated using the following formula:

[0030] (5),

[0031] in, This represents the semantic pre-constraint loss; This represents the image feature extraction result of the CLIP model. This loss achieves semantic pre-alignment of the generated image by constraining the consistency of image embedding features between the original image and the synthesized image. This represents the cosine similarity operation;

[0032] The latent space compactness constraint loss is expressed by the following formula:

[0033] (6),

[0034] in, This represents the latent space compactness constraint loss, which is an auxiliary regularization term used only to prevent excessive discretization of the latent space and does not dominate generator training. Indicates the first One latent vector; Represents the mean vector of the latent vector library; Represents L2 norm operations; Indicates the total number of latent vectors;

[0035] The total generator loss is expressed by the following formula:

[0036] (7),

[0037] in, Represents the total loss function of the generator; This represents the weighting coefficient for adversarial loss; This represents the feature matching loss weight coefficient; Indicates the semantic pre-constraint weight coefficient; This represents the weight coefficient of the latent space compactness constraint;

[0038] Based on the generator's total loss, the client synchronously and iteratively updates the generator model parameters and latent vector library parameters. The updates to the generator and latent vector library are performed using the following formulas:

[0039] (8),

[0040] in, This represents the trainable parameters of the generator model; This represents the generator's training learning rate; This represents the gradient value of the generator's total loss with respect to the generator parameters; Represents the set of parameters in the latent vector library; This represents the training learning rate of the latent vector library; This represents the gradient value of the generator's total loss with respect to the parameters of the latent vector library;

[0041] After the client completes the generative adversarial network distillation training, it discards the original private data and discriminator model, exports the synthesized image and synthesized label, and uploads it to the edge server to complete the client's privacy distillation process.

[0042] Step 2: In the distillation data security purification stage based on CLIP cross-modal semantic verification, the edge server receives the synthetic distillation images and synthetic tags uploaded by each client and performs global aggregation. Then, it uses the CLIP multimodal model to complete the image-text semantic consistency detection between the synthetic distillation images and the corresponding text, calculates the semantic similarity score, and completes data filtering with a preset threshold. Backdoor samples and samples with low semantic consistency are accurately removed. Based on the semantic similarity of qualified samples, exclusive credibility tags are generated. The purified distillation data and credibility tags are bound and encrypted and then uploaded to the cloud server to build a secure distillation dataset with no backdoors and high semantic consistency.

[0043] Preferably, step two mainly includes two steps:

[0044] Step 1: The edge server receives the globally aggregated synthetic distilled image and synthetic labels from the client, calls the CLIP multimodal model to encode the image and text respectively, calculates the cosine similarity between the image embedding features and the text embedding features, and obtains the image-text semantic consistency score for a single sample. The specific process is as follows:

[0045] The edge server receives synthetic distillation image samples from each client. With synthetic tags After global aggregation, cross-modal semantic verification is performed; the synthesized distilled image samples are then analyzed. The input is a CLIP image encoder, which, after encoding and mapping, yields the image embedding feature vector corresponding to that sample. ; synthesize distillation image samples The matched text description is input to the CLIP text encoder, and after encoding and mapping processing, the corresponding text embedding feature vector is obtained. .

[0046] The edge server calculates the cosine similarity between the image embedding features and the text embedding features to obtain the image-text semantic consistency score for a single sample. The image-text semantic consistency score is calculated using the following formula:

[0047] (9),

[0048] in, Indicates the first The semantic consistency score of the text and image of each distillation sample; This represents the function for calculating cosine similarity. Indicates the first CLIP image embedding feature vectors for each sample; Indicates the first CLIP text embedding feature vectors for each sample; This represents the transpose of the image embedding feature vector; This represents the magnitude of the image embedding feature vector; This represents the magnitude of the text embedding feature vector.

[0049] Step 2: The edge server compares the semantic similarity score with a preset threshold, removes backdoor samples with similarity below the threshold and samples with low semantic consistency, generates credibility labels based on the semantic similarity of qualified samples, and packages the cleaned distilled image and credibility labels together and uploads them to the cloud server. The specific process is as follows:

[0050] The edge server uses the CLIP (Content-Based Image-Text Semantic Consistency Score) as a cross-modal semantic trust anchor. It precisely compares the semantic consistency score of each distillation sample with a preset judgment threshold, directly eliminating semantically fragmented samples caused by backdoor attacks and low-quality distillation samples with excessive semantic deviation. Only compliant samples with highly aligned image and text semantics are retained, thus obtaining secure distillation data. The secure data selection uses the following formula:

[0051] (10)

[0052] in, This represents a secure distillation dataset after cross-modal semantic filtering. Indicates the first One distillation image sample; Indicates the first The true label corresponding to each distillation image sample; Indicates the first CLIP semantic consistency score of each distillation sample; This represents the preset semantic consistency threshold, the value of which is adjusted according to the distillation image effect, and is used to determine whether a sample has semantic fragmentation or backdoor risk;

[0053] Based on the semantic consistency scores of the selected secure samples, the edge server establishes a positive correlation between semantic consistency scores and data credibility, generating a unique data credibility label. This label will serve as the core basis for subsequent AdaLoRA dynamic rank allocation on the cloud server side. The edge server binds the secure distillation images one by one with the corresponding credibility labels, constructing a complete secure distillation dataset containing semantic credibility annotations. This dataset is then uploaded to the cloud server via an encrypted communication channel, without ever touching or storing the client's original private data, thus strictly ensuring the privacy and security of private domain data.

[0054] Step 3: In the large-scale model security fine-tuning and backdoor defense stage based on AdaLoRA dynamic rank allocation and forgetting learning, the cloud server uses the AdaLoRA fine-tuner to establish dynamic rank allocation rules based on the security distillation data and credibility labels uploaded from the edge. Adaptive parameters are assigned to different data according to the positive correlation between credibility and rank. For low credibility data, forgetting terms are introduced to offset backdoor features, completing the large-scale model adaptive parameter update and security fine-tuning. The accuracy of the model on clean samples and the success rate of backdoor attacks are verified through a standard dataset. Finally, a secure large-scale model that can be directly deployed is output.

[0055] Preferably, step three mainly includes two steps:

[0056] Step 1: The cloud server establishes an AdaLoRA dynamic rank allocation rule based on the data credibility labels. Credibility and rank are positively correlated. High-credibility data is assigned high-rank parameters to enhance feature learning, while low-credibility data is assigned low-rank parameters to limit model updates. The specific process is as follows:

[0057] The cloud server receives the secure distillation dataset uploaded by the edge server, concatenates the images and labeled text to construct multimodal training samples, and generates the input encoding, attention mask and visual feature tensor required for model input.

[0058] The cloud server performs AdaLoRA dynamic rank routing allocation based on the semantic consistency score of the samples and a preset trust threshold. It assigns appropriate parameters with corresponding rank sizes to samples of different trust levels, achieving collaborative optimization of focusing on learning high-trust samples and suppressing backdoor features of low-trust samples. The dynamic rank routing adopts the following formula:

[0059] (11),

[0060] in, Indicates the rank-based branch of AdaLoRA; This represents the semantic consistency score between the sample text and images; Represents the trust threshold for rank assignment, satisfying ; This represents the high-rank fitting branch corresponding to a high-confidence sample; This represents the low-rank fitting branch corresponding to the low-confidence sample;

[0061] For multimodal large model fine-tuning scenarios, cloud servers calculate the regularization loss term for trainable parameters. They weight and fuse the multimodal task cross-entropy loss and the parameter regularization loss to construct a total fine-tuning loss that balances task performance, model stability, and backdoor defense capabilities. The parameter regularization term uses the following formula:

[0062] (12),

[0063] in, This represents the regularization term for the model parameters; This represents the set of trainable parameters for the model; Indicates trainable parameters; This represents the mean of the squares of the parameters.

[0064] The total loss function is expressed by the following formula:

[0065] (13)

[0066] in, This represents the total loss from model fine-tuning; This represents the cross-entropy loss for multimodal tasks; This represents the weight coefficient of the regularization term;

[0067] The cloud server accumulates gradients according to a preset number of gradient accumulation steps, applies clipping constraints to the accumulated gradients, and then updates the global model parameters. The gradient clipping uses the following formula:

[0068] (14)

[0069] in, Represents the gradient of the model parameters; Represents the gradient clipping function; The L2 norm of the gradient; This represents the maximum threshold of the gradient.

[0070] Step 2: The cloud server introduces a forgetting term into the loss function for low-confidence data, uses the low-rank AdaLoRA parameter to constrain the fitting of backdoor features, and combines the forgetting term to offset residual backdoor features, completing the parameter update and security adaptive fine-tuning of the global large model. The model performance and backdoor defense effectiveness are then verified. The specific process is as follows:

[0071] Based on sample credibility labels, the cloud server introduces a forgetting term into the total loss of the model fine-tuning for low-credibility data with semantic consistency scores below the trust threshold, constructing a joint loss function with a backdoor feature forgetting mechanism; the backdoor feature forgetting term uses the following formula:

[0072] (15)

[0073] in, This represents a subset of data with low credibility. Indicates low confidence samples Trigger sample obtained after injecting a preset backdoor trigger; This represents the forward propagation function of the current global model; Represents the cross-entropy loss function; Indicates the true label of the sample; Indicates the target label for a backdoor attack;

[0074] The joint loss is expressed by the following formula:

[0075] (16)

[0076] in, This represents the final fine-tuning loss after introducing the forgetting term; This represents the total loss from fine-tuning the original model; This represents the weight coefficient of the forgotten item; This represents the forgotten term of the backdoor features, used to suppress gradient propagation of residual backdoor features;

[0077] The cloud server calls the AdaLoRA low-rank adaptation parameter branch to restrict the fitting of hidden backdoor features in low-confidence data. It combines the forgetting term to achieve accurate cancellation of backdoor features. Based on the final joint loss, it completes gradient calculation, gradient accumulation and pruning, updates the global large model parameters, eliminates backdoor risks while retaining normal task features, and completes the safe adaptive fine-tuning of the global large model.

[0078] The cloud server uses a standard publicly available test dataset to verify the performance of the fine-tuned model, calculating the clean sample accuracy and backdoor attack success rate. The clean sample accuracy is calculated using the following formula:

[0079] (17)

[0080] in, This represents the accuracy of the model on clean samples. This represents the total number of samples in the standard test set; Indicates the model's predicted label; Indicates the true label of the sample; This indicates an indicator function; a value of 1 indicates a correct prediction, and a value of 0 indicates an incorrect prediction.

[0081] The success rate of backdoor attacks is calculated using the following formula:

[0082] (18)

[0083] in, Indicates the success rate of backdoor attacks; This represents the total number of samples in the backdoor test set; Indicates the predicted label of the backdoor sample; Indicates the target label;

[0084] After the cloud server completes the verification of model performance and defense effectiveness, it saves the optimal model checkpoints and outputs a large, finely tuned security model that can be directly deployed.

[0085] The beneficial effects of this invention are:

[0086] (1) This invention proposes a large model fine-tuning method based on generative distillation and cross-modal semantic verification. Through a collaborative closed-loop mechanism of GAN latent space privacy distillation, CLIP cross-modal semantic verification, and AdaLoRA adaptive fine-tuning, it effectively solves multiple technical problems faced by large model fine-tuning in distributed scenarios, such as privacy leakage of private domain data, semantic fragmentation of distilled data, difficulty in defending against backdoor attacks, and excessive consumption of fine-tuning resources. Under the premise of strictly adhering to the bottom line of original private domain data not leaving the domain, this method significantly reduces the computing and communication burden on the edge side, while significantly improving the generalization ability and backdoor defense robustness of large models in vertical private domain scenarios. It provides core technical support for the safe and efficient implementation of multimodal large models in privacy-sensitive fields such as medical care, finance, and government affairs.

[0087] (2) The GAN latent space privacy distillation mechanism adopted in this invention transforms pixel-level optimization of private domain data into latent space optimization. It generates desensitized synthetic data that approximates the original data distribution through lightweight GAN training, and uploads the synthetic image and synthetic label. This mechanism not only fully preserves the core features and distribution characteristics of the private domain data, effectively adapting to the heterogeneity problem of client data, but also eliminates the risk of privacy leakage from the source by uploading synthetic distilled images and synthetic labels to replace the original private domain data. Simultaneously, it greatly reduces the communication overhead of data transmission, providing a lightweight and feasible solution for resource-constrained edge clients to participate in the federated fine-tuning of large models.

[0088] (3) The CLIP cross-modal semantic verification and AdaLoRA dynamic rank adaptive defense mechanism designed in this invention uses the semantic consistency between text and images as a trust anchor to complete the pre-screening of distilled data, and realizes dynamic rank allocation and backdoor feature forgetting learning based on data credibility. This mechanism can accurately remove backdoor samples and data with low semantic consistency, limit the fitting of backdoor features by low-rank parameters, and cancel the residual backdoor features by forgetting terms. While completely eliminating the risk of backdoor attacks, it adopts high-rank focus learning for high-credibility and high-quality data, taking into account the efficiency of model fine-tuning and task performance, and achieving the optimal balance between the effectiveness of backdoor defense, model availability and fine-tuning efficiency. Attached Figure Description

[0089] Figure 1 This is a framework diagram of a large model fine-tuning method based on generative distillation and cross-modal semantic verification proposed in this invention;

[0090] Figure 2 This is a flowchart of a large model fine-tuning method based on generative distillation and cross-modal semantic verification proposed in this invention. Detailed Implementation

[0091] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0092] This embodiment presents a large model fine-tuning method based on generative distillation and cross-modal semantic verification, such as... Figure 1 As shown, it specifically includes:

[0093] Step 1-1: The client trains a lightweight generative adversarial network (GAN) locally using private data. The pixel-level optimization of the private data is transformed into latent space optimization. The latent vectors and generator weights are iteratively optimized to make the feature distribution of the generated image approximate the distribution of the real private data. The specific process is as follows:

[0094] Each client independently initiates the distillation process locally; the edge server does not participate in local client training. Each client loads its local private data according to its own identity, thus constructing its local private dataset. The local private dataset contains normal private domain samples and malicious samples that may be implanted with backdoors;

[0095] Based on the globally preset total number of synthetic samples and the total number of clients participating in training, the number of synthetic samples required per client is determined according to an equal allocation rule. The allocation of synthetic sample quantity adopts the following formula:

[0096] (1),

[0097] in, Indicates the first The number of synthetic samples required to be generated for each client; This indicates the total number of synthetic samples required globally. This represents the total number of clients participating in the federated distillation training; This indicates the floor function; This indicates an indicator function; it takes the value 1 when the condition inside the parentheses is true and 0 when the condition is false. This represents the modulo operation between the total number of global synthetic samples and the total number of clients.

[0098] Steps 1-2: The client discards the original private data and GAN discriminator, exports the synthetic distillation image and synthetic label, and uploads the synthetic distillation image and synthetic label to the edge server through a secure communication channel. This achieves privacy-preserving uploading of the original private domain data without it leaving the domain. The specific process is as follows:

[0099] The client initializes a lightweight generative adversarial network distillation model with semantic pre-alignment, the distillation model containing a generator. Discriminator With latent vector library Real private data samples With the generator using latent vectors The generated synthetic sample Input the data into the discriminator and simultaneously calculate the discriminant loss and semantic pre-constraint loss.

[0100] The client uses an adaptive gradient descent algorithm to iteratively update the discriminator model parameters based on the discriminator loss. The discriminator parameter update uses the following formula:

[0101] (2),

[0102] in, This represents the trainable parameters of the discriminator model; This represents the discriminator's training learning rate; This represents the gradient of the discriminator loss function with respect to the discriminator parameters;

[0103] The client calculates the generator adversarial loss, generator feature matching loss, CLIP semantic pre-constraint loss, and latent space compactness constraint loss respectively, and then weights and fuses the four types of losses to obtain the total training loss of the generator.

[0104] The generator adversarial loss is expressed by the following formula:

[0105] (3),

[0106] in, This indicates that the generator is adversarial against loss;

[0107] The generator feature matching loss is expressed by the following formula:

[0108] (4),

[0109] in, This represents the generator feature matching loss; This indicates the total number of feature layers output by the discriminator; Indicates the discriminator's first... The layer's feature output results; This represents the expected value calculation of the feature data; Represents L1 norm operations;

[0110] CLIP semantic pre-constraint loss is calculated using the following formula:

[0111] (5),

[0112] in, This represents the semantic pre-constraint loss; This represents the image feature extraction result of the CLIP model. This loss achieves semantic pre-alignment of the generated image by constraining the consistency of image embedding features between the original image and the synthesized image. This represents the cosine similarity operation;

[0113] The latent space compactness constraint loss is expressed by the following formula:

[0114] (6),

[0115] in, This represents the latent space compactness constraint loss, which is an auxiliary regularization term used only to prevent excessive discretization of the latent space and does not dominate generator training. Indicates the first One latent vector; Represents the mean vector of the latent vector library; Represents L2 norm operations; Indicates the total number of latent vectors;

[0116] The total generator loss is expressed by the following formula:

[0117] (7),

[0118] in, Represents the total loss function of the generator; This represents the weighting coefficient for adversarial loss; This represents the feature matching loss weight coefficient; Indicates the semantic pre-constraint weight coefficient; This represents the weight coefficient of the latent space compactness constraint;

[0119] Based on the generator's total loss, the client synchronously and iteratively updates the generator model parameters and latent vector library parameters. The updates to the generator and latent vector library are performed using the following formulas:

[0120] (8),

[0121] in, This represents the trainable parameters of the generator model; This represents the generator's training learning rate; This represents the gradient value of the generator's total loss with respect to the generator parameters; Represents the set of parameters in the latent vector library; This represents the training learning rate of the latent vector library; This represents the gradient value of the generator's total loss with respect to the parameters of the latent vector library;

[0122] After the client completes the generative adversarial network distillation training, it discards the original private data and discriminator model, exports the synthesized image and synthesized label, and uploads it to the edge server to complete the client's privacy distillation process.

[0123] Step 2-1: The edge server receives the synthetic distillation images and synthetic labels uploaded by each client, performs global aggregation, and then calls the CLIP multimodal model to encode the image and text respectively. It calculates the cosine similarity between the image embedding features and the text embedding features to obtain the image-text semantic consistency score for a single sample. The specific process is as follows:

[0124] The edge server receives synthetic distillation image samples from each client. With synthetic tags After global aggregation, cross-modal semantic verification is performed; the synthesized distilled image samples are then analyzed. The input is a CLIP image encoder, which, after encoding and mapping, yields the image embedding feature vector corresponding to that sample. ; synthesize distillation image samples The matched text description is input to the CLIP text encoder, and after encoding and mapping processing, the corresponding text embedding feature vector is obtained. .

[0125] The edge server calculates the cosine similarity between the image embedding features and the text embedding features to obtain the image-text semantic consistency score for a single sample. The image-text semantic consistency score is calculated using the following formula:

[0126] (9),

[0127] in, Indicates the first The semantic consistency score of the text and image of each distillation sample; This represents the function for calculating cosine similarity. Indicates the first CLIP image embedding feature vectors for each sample; Indicates the first CLIP text embedding feature vectors for each sample; This represents the transpose of the image embedding feature vector; This represents the magnitude of the image embedding feature vector; This represents the magnitude of the text embedding feature vector.

[0128] Step 2-2: The edge server compares the semantic similarity score with a preset threshold, removes backdoor samples with similarity below the threshold and samples with low semantic consistency, generates credibility labels based on the semantic similarity of qualified samples, and packages the purified distilled image and credibility labels and uploads them to the cloud server. The specific process is as follows:

[0129] The edge server uses the CLIP (Content-Based Image-Text Semantic Consistency Score) as a cross-modal semantic trust anchor. It precisely compares the semantic consistency score of each distillation sample with a preset judgment threshold, directly eliminating semantically fragmented samples caused by backdoor attacks and low-quality distillation samples with excessive semantic deviation. Only compliant samples with highly aligned image and text semantics are retained, thus obtaining secure distillation data. The secure data selection uses the following formula:

[0130] (10)

[0131] in, This represents a secure distillation dataset after cross-modal semantic filtering. Indicates the first One distillation image sample; Indicates the first The true label corresponding to each distillation image sample; Indicates the first CLIP semantic consistency score of each distillation sample; This represents the preset semantic consistency threshold, the value of which is adjusted according to the distillation image effect, and is used to determine whether a sample has semantic fragmentation or backdoor risk;

[0132] Based on the semantic consistency scores of the selected secure samples, the edge server establishes a positive correlation between semantic consistency scores and data credibility, generating a unique data credibility label. This label will serve as the core basis for subsequent AdaLoRA dynamic rank allocation on the cloud server side. The edge server binds the secure distillation images one by one with the corresponding credibility labels, constructing a complete secure distillation dataset containing semantic credibility annotations. This dataset is then uploaded to the cloud server via an encrypted communication channel, without ever touching or storing the client's original private data, thus strictly ensuring the privacy and security of private domain data.

[0133] Step 3-1: The cloud server establishes an AdaLoRA dynamic rank allocation rule based on the data credibility labels. Credibility and rank are positively correlated. High-credibility data is assigned high-rank parameters to enhance feature learning, while low-credibility data is assigned low-rank parameters to limit model updates. The specific process is as follows:

[0134] The cloud server receives the secure distillation dataset uploaded by the edge server, concatenates the images and labeled text to construct multimodal training samples, and generates the input encoding, attention mask and visual feature tensor required for model input.

[0135] The cloud server performs AdaLoRA dynamic rank routing allocation based on the semantic consistency score of the samples and a preset trust threshold. It assigns appropriate parameters with corresponding rank sizes to samples of different trust levels, achieving collaborative optimization of focusing on learning high-trust samples and suppressing backdoor features of low-trust samples. The dynamic rank routing adopts the following formula: (11),

[0136] in, Indicates the rank-based branch of AdaLoRA; This represents the semantic consistency score between the sample text and images; Represents the trust threshold for rank assignment, satisfying ; This represents the high-rank fitting branch corresponding to a high-confidence sample; This represents the low-rank fitting branch corresponding to the low-confidence sample;

[0137] For multimodal large model fine-tuning scenarios, cloud servers calculate the regularization loss term for trainable parameters. They weight and fuse the multimodal task cross-entropy loss and the parameter regularization loss to construct a total fine-tuning loss that balances task performance, model stability, and backdoor defense capabilities. The parameter regularization term uses the following formula:

[0138] (12),

[0139] in, This represents the regularization term for the model parameters; This represents the set of trainable parameters for the model; Indicates trainable parameters; This represents the mean of the squares of the parameters.

[0140] The total loss function is expressed by the following formula:

[0141] (13)

[0142] in, This represents the total loss from model fine-tuning; This represents the cross-entropy loss for multimodal tasks; This represents the weight coefficient of the regularization term;

[0143] The cloud server accumulates gradients according to a preset number of gradient accumulation steps, applies clipping constraints to the accumulated gradients, and then updates the global model parameters. The gradient clipping uses the following formula:

[0144] (14)

[0145] in, Represents the gradient of the model parameters; Represents the gradient clipping function; The L2 norm of the gradient; This represents the maximum threshold of the gradient.

[0146] Step 3-2: Based on the sample credibility labels, the cloud server introduces a forgetting term into the total loss of the model fine-tuning for low-credibility data whose semantic consistency score is lower than the trust threshold, constructing a joint loss function with a backdoor feature forgetting mechanism; the backdoor feature forgetting term adopts the following formula:

[0147] (15)

[0148] in, This represents a subset of data with low credibility. Indicates low confidence samples Trigger sample obtained after injecting a preset backdoor trigger; This represents the forward propagation function of the current global model; Represents the cross-entropy loss function; Indicates the true label of the sample; Indicates the target label for a backdoor attack;

[0149] The joint loss is expressed by the following formula:

[0150] (16)

[0151] in, This represents the final fine-tuning loss after introducing the forgetting term; This represents the total loss from fine-tuning the original model; This represents the weight coefficient of the forgotten item; This represents the forgotten term of the backdoor features, used to suppress gradient propagation of residual backdoor features;

[0152] The cloud server calls the AdaLoRA low-rank adaptation parameter branch to restrict the fitting of hidden backdoor features in low-confidence data. It combines the forgetting term to achieve accurate cancellation of backdoor features. Based on the final joint loss, it completes gradient calculation, gradient accumulation and pruning, updates the global large model parameters, eliminates backdoor risks while retaining normal task features, and completes the safe adaptive fine-tuning of the global large model.

[0153] The cloud server uses a standard publicly available test dataset to verify the performance of the fine-tuned model, calculating the clean sample accuracy and backdoor attack success rate. The clean sample accuracy is calculated using the following formula:

[0154] (17)

[0155] in, This represents the accuracy of the model on clean samples. This represents the total number of samples in the standard test set; Indicates the model's predicted label; Indicates the true label of the sample; This indicates an indicator function; a value of 1 indicates a correct prediction, and a value of 0 indicates an incorrect prediction.

[0156] The success rate of backdoor attacks is calculated using the following formula:

[0157] (18)

[0158] in, Indicates the success rate of backdoor attacks; This represents the total number of samples in the backdoor test set; Indicates the predicted label of the backdoor sample; Indicates the target label;

[0159] After the cloud server completes the verification of model performance and defense effectiveness, it saves the optimal model checkpoints and outputs a large, finely tuned security model that can be directly deployed.

[0160] To verify the effectiveness of the proposed method, a series of experiments were designed and executed, and the evaluation was carried out mainly from the following three aspects: (1) privacy protection effect of dataset distillation; (2) semantic calibration and risk screening effect of CLIP cross-modal semantic verification; (3) performance and backdoor defense effect of large model fine-tuning.

[0161] 1. Validation of the privacy protection effect of dataset distillation:

[0162] This experiment aims to verify the effectiveness of our proposed GAN latent space privacy distillation technique, confirming its ability to generate desensitized synthetic data that retains core semantic features while keeping the original private domain data within its domain, thus achieving a balance between privacy protection and data utility. The experiment uses a multi-category private domain image-text dataset as test samples. The client performs lightweight GAN distillation training locally, exporting the synthesized distilled images and labels. From a visual semantic perspective, the distilled images fully preserve the target features, category semantics, and scene information of the original private domain images, fully supporting the feature learning needs of large-scale model fine-tuning. From a pixel data perspective, the distilled images have no direct pixel correlation with the original private domain images, making it impossible to reconstruct the original private domain data from the distilled images, fundamentally avoiding the leakage of original private domain data.

[0163] Experimental results show that the latent space privacy distillation mechanism adopted in this invention achieves the desensitization and representation transformation of the original data while strictly adhering to the constraint that the original private domain data does not leave the domain. This not only preserves the core features required for model training but also eliminates the risk of privacy leakage, achieving the optimal balance between privacy protection and data utility.

[0164] 2. The effectiveness of semantic calibration and risk screening in CLIP cross-modal semantic validation:

[0165] This experiment aims to verify the effectiveness of CLIP cross-modal semantic verification in semantic calibration, poor sample removal, backdoor identification, and credibility labeling, confirming that it can accurately purify distillation data and eliminate image-text semantic fragmentation and backdoor security risks while retaining valid data. The experiment uses mixed distillation data containing poor-quality samples with image-text semantic fragmentation, backdoor-implanted samples carrying white square backdoor triggers, and semantically compliant high-quality samples as input. Sample screening and classification are completed based on CLIP image-text semantic similarity scores. The first category, distillation images with severe image-text semantic fragmentation and poor quality, as well as some backdoor-implanted samples carrying white square backdoors and disrupting semantic consistency, are directly removed because their semantic similarity scores are far below the preset threshold. The second category, distillation samples containing white square features but not affecting image-text semantic alignment, are retained because their semantic similarity scores meet the threshold; these features do not trigger backdoor behavior and do not affect the core semantic expression.

[0166] Experimental results show that the CLIP cross-modal semantic verification mechanism adopted in this invention, based on the consistency of image and text semantics, can accurately remove poor-quality distillation images with poor quality and semantic fragmentation, as well as high-risk backdoor implantation samples, while reasonably retaining effective distillation images with harmless interference and complete semantics. It can efficiently purify data risks while maximizing the retention of usable training data, providing a highly reliable and available safe distillation dataset for subsequent large-scale model safety fine-tuning.

[0167] 3. Performance and overall backdoor defense effectiveness after fine-tuning the large model:

[0168] To comprehensively evaluate the impact of the overall process of this invention on the performance and security of the final large model, this invention is based on the Qwen / Qwen3-VL-2B-Instruct multimodal large model, and uses the Cifar10 dataset for 10 rounds of fine-tuning, selecting the optimal checkpoint to compare its performance indicators.

[0169] The experiment was conducted based on a multimodal large model for fine-tuning tests, and two sets of experiments were set up:

[0170] Experiment 1: Fine-tuning directly using raw private domain data (without privacy protection);

[0171] This invention uses data that has undergone GAN distillation and CLIP screening, combined with AdaLoRA dynamic rank fine-tuning.

[0172] Evaluation indicators:

[0173] CleanAcc (%): A higher percentage indicates better performance of the model on normal tasks;

[0174] Backdoor attack success rate (ASR, %): The lower the ASR, the stronger the backdoor defense capability of the model.

[0175] Table 1

[0176]

[0177] The experimental results in Table 1, based on the CIFAR-10 dataset, clearly demonstrate that the method of this invention achieves an excellent balance between model performance and backdoor defense. The untuned base model achieves a clean sample accuracy of only 72.8% on the CIFAR-10 test set. Using the method of this invention, fine-tuning is performed using only 240 distilled images filtered by GAN distillation and CLIP, increasing the clean sample accuracy to 84.3%, a significant performance gain compared to the base model. Compared to the scheme that directly uses the original private domain data for fine-tuning (clean sample accuracy 96.2%), this invention achieves strict privacy protection by ensuring the original private domain data remains within its domain and significantly reduces the number of image samples used for fine-tuning. In the BadNet backdoor attack scenario, the backdoor attack success rate (ASR) of the method of this invention is only 8.7%, proving that the backdoor triggers in the distilled data have been effectively suppressed, malicious labels are almost ineffective, and the backdoor attack threat is fundamentally contained. Overall, this invention achieves effective improvement in model performance and strong defense against backdoor attacks simultaneously, using only a small amount of distillation data and strictly protecting private domain privacy, thus achieving optimal synergy between model performance, data privacy, and security defense.

[0178] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for fine-tuning large models based on generative distillation and cross-modal semantic verification, characterized in that, Includes the following steps: Step 1: The client performs latent space privacy distillation on the local raw private domain data based on a lightweight generative adversarial network (GAN), exports the synthetic distilled image and synthetic label, and uploads the synthetic distilled image and synthetic label to the edge server. Step 2: The edge server receives the synthetic distillation images and synthetic labels uploaded by each client, performs global aggregation, and completes the image-text semantic consistency detection and data filtering through the CLIP multimodal model. After labeling qualified data with credibility tags, it is uploaded to the cloud server. Step 3: Based on the cleaned qualified data and corresponding credibility tags, the cloud server uses the AdaLoRA fine-tuner to perform dynamic rank allocation and adaptive parameter updates, thereby achieving secure fine-tuning of the large model and backdoor defense. Step three includes the following steps: Step 3-1: The cloud server establishes the AdaLoRA dynamic rank allocation rule based on the data credibility label. Credibility and rank are positively correlated. High-credibility data is assigned high-rank parameters to enhance feature learning, while low-credibility data is assigned low-rank parameters to limit model updates. Step 3-2: The cloud server introduces a forgetting term into the loss function for low-confidence data, uses the low-rank AdaLoRA parameter to limit the fitting of backdoor features, and combines the forgetting term to cancel the residual backdoor features, thus completing the parameter update and safety adaptive fine-tuning of the global large model. Step 3-1 includes the following steps: Step 3-1-1: The cloud server receives the secure distillation dataset uploaded by the edge server, concatenates the images and labeled text to construct multimodal training samples, and generates the input encoding, attention mask and visual feature tensor required for model input; Step 3-1-2: Based on the semantic consistency score of the sample and the preset trust threshold, the cloud server performs AdaLoRA dynamic rank routing allocation, assigning adaptation parameters of corresponding rank size to samples with different trust levels. The dynamic rank routing adopts the following formula: （11）， in, Indicates the rank-based branch of AdaLoRA; This represents the semantic consistency score between the sample text and images; Represents the trust threshold for rank assignment, satisfying ; This represents the high-rank fitting branch corresponding to a high-confidence sample; This represents the low-rank fitting branch corresponding to the low-confidence sample; Step 3-1-3: For multimodal large model fine-tuning scenarios, the cloud server calculates the regularization loss term for trainable parameters, and weights and fuses the multimodal task cross-entropy loss and parameter regularization loss to construct a total fine-tuning loss that balances task performance, model stability, and backdoor defense capabilities. The parameter regularization term uses the following formula: （12）， in, This represents the regularization term for the model parameters; This represents the set of trainable parameters for the model; Indicates trainable parameters; This represents the mean of the squares of the parameters. The total loss function is expressed by the following formula: （13）， in, This represents the total loss from model fine-tuning; This represents the cross-entropy loss for multimodal tasks; This represents the weight coefficient of the regularization term; Step 3-1-4: The cloud server completes gradient accumulation according to the preset gradient accumulation steps, applies clipping constraints to the accumulated gradients, and then updates the global model parameters. The gradient clipping uses the following formula: （14）， in, Represents the gradient of the model parameters; Represents the gradient clipping function; The L2 norm of the gradient; This represents the maximum threshold of the gradient.

2. The method according to claim 1, characterized in that, Step one specifically includes: Step 1-1: Each client independently starts the distillation process locally. Based on the globally preset total number of synthetic samples and the total number of clients participating in the training, the number of synthetic samples to be generated by a single client is determined. Steps 1-2: The client trains a lightweight generative adversarial network (GAN) with semantic pre-alignment on the local machine using private data. The pixel-level optimization of the private data is transformed into latent space optimization with latent space compactness constraints. Based on the joint iteration of adversarial loss, feature matching loss, CLIP semantic pre-constraint loss and latent space compactness constraint loss, the latent vector and generator weights are optimized so that the feature distribution and semantic features of the generated image are close to the distribution of the real private data. Steps 1-3: The client discards the original private data and GAN discriminator, exports the synthetic distillation image and synthetic label, and uploads the synthetic distillation image and synthetic label to the edge server to ensure that the original private domain data does not leave the domain.

3. The method according to claim 2, characterized in that, Step two specifically includes: Step 2-1: The edge server receives the synthetic distillation image and synthetic label from the client, performs global aggregation, and then calls the CLIP multimodal model to calculate the semantic similarity between the synthetic distillation image and the corresponding text. Step 2-2: The edge server compares the semantic similarity with a preset threshold, removes backdoor samples with similarity below the threshold and samples with low semantic consistency, generates credibility labels based on the semantic similarity of qualified samples, and packages the purified distilled image and credibility labels and uploads them to the cloud server.

4. The method according to claim 3, characterized in that: Step 1-1 specifically includes the following steps: Step 1-1-1: Each client independently starts the distillation process locally. The edge server does not participate in the local training of the client. Each client loads its local private data according to its own identity and builds its local private dataset. The local private dataset It includes normal private domain samples and malicious samples that may have backdoors implanted; Step 1-1-2: Based on the globally preset total number of synthetic samples and the total number of clients participating in training, determine the number of synthetic samples required per client according to the equal allocation rule. The allocation of synthetic sample quantity adopts the following formula: （1）， in, Indicates the first The number of synthetic samples required to be generated for each client; This indicates the total number of synthetic samples required globally. This represents the total number of clients participating in the federated distillation training; This indicates the floor function; This indicates an indicator function; it takes the value 1 if the condition within the parentheses is true, and takes the value 0 if the condition is false. This represents the modulo operation between the total number of global synthetic samples and the total number of clients.

5. The method according to claim 4, characterized in that: Step 1-2 specifically includes the following steps: Step 1-2-1: The client initializes a lightweight generative adversarial network distillation model with semantic pre-alignment. The adversarial network distillation model contains a generator. Discriminator With latent vector library Real private data samples With the generator using latent vectors generated synthetic samples Input the data into the discriminator separately, and simultaneously calculate the discriminant loss and semantic pre-constraint loss; Step 1-2-2: Based on the discriminator loss, the client iteratively updates the discriminator model parameters using an adaptive gradient descent algorithm. The discriminator parameter update uses the following formula: （2）， in, This represents the trainable parameters of the discriminator model; This represents the discriminator's training learning rate; This represents the gradient of the discriminator loss function with respect to the discriminator parameters; Steps 1-2-3: The client calculates the generator adversarial loss, generator feature matching loss, CLIP semantic pre-constraint loss, and latent space compactness constraint loss respectively, and then weights and fuses the four types of losses to obtain the total training loss of the generator. The generator adversarial loss is expressed by the following formula: （3）， in, This indicates that the generator is adversarial against loss; The generator feature matching loss is expressed by the following formula: （4）， in, This represents the generator feature matching loss; This indicates the total number of feature layers output by the discriminator; Indicates the discriminator's first... The layer's feature output results; This represents the expected value calculation of the feature data; Represents L1 norm operations; CLIP semantic pre-constraint loss is calculated using the following formula: （5）， in, This represents the semantic pre-constraint loss; This represents the image feature extraction result of the CLIP model. This loss achieves semantic pre-alignment of the generated image by constraining the consistency of image embedding features between the original image and the synthesized image. This represents the cosine similarity operation; The latent space compactness constraint loss is expressed by the following formula: （6）， in, This represents the latent space compactness constraint loss, which is an auxiliary regularization term used only to prevent excessive discretization of the latent space and does not dominate generator training. Indicates the first One latent vector; Represents the mean vector of the latent vector library; Represents L2 norm operations; Indicates the total number of latent vectors; The total generator loss is expressed by the following formula: （7）， in, Represents the total loss function of the generator; This represents the weighting coefficient for adversarial loss; This represents the feature matching loss weight coefficient; Indicates the semantic pre-constraint weight coefficient; This represents the weight coefficient of the latent space compactness constraint; Steps 1-2-4: Based on the generator's total loss, the client synchronously and iteratively updates the generator model parameters and latent vector library parameters. The generator and latent vector library updates are performed using the following formulas: （8）， in, This represents the trainable parameters of the generator model; This represents the generator's training learning rate; This represents the gradient value of the generator's total loss with respect to the generator parameters; Represents the set of parameters in the latent vector library; This represents the training learning rate of the latent vector library; This represents the gradient value of the generator's total loss with respect to the parameters of the latent vector library; Steps 1-2-5: After the client completes the Generative Adversarial Network distillation training, it discards the original private data and discriminator model, exports the synthesized image and synthesized label, and uploads it to the edge server, thus completing the client's privacy distillation process.

6. The method according to claim 5, characterized in that: Step 2-1 specifically includes the following steps: Step 2-1-1: The edge server receives synthetic distillation image samples from each client. With synthetic tags After global aggregation, cross-modal semantic verification is performed; the synthesized distilled image samples are then analyzed. The input is a CLIP image encoder, which, after encoding and mapping, yields the image embedding feature vector corresponding to that sample. ; synthesize distillation image samples The matched text description is input to the CLIP text encoder, and after encoding and mapping processing, the corresponding text embedding feature vector is obtained. ; Step 2-1-2: The edge server calculates the cosine similarity between the image embedding features and the text embedding features to obtain the image-text semantic consistency score for a single sample. The image-text semantic consistency score is calculated using the following formula: （9）， in, Indicates the first The semantic consistency score of the text and image of each distillation sample; This represents the function for calculating cosine similarity. Indicates the first CLIP image embedding feature vectors for each sample; Indicates the first CLIP text embedding feature vectors for each sample; This represents the transpose of the image embedding feature vector; This represents the magnitude of the image embedding feature vector; This represents the magnitude of the text embedding feature vector.

7. The method according to claim 6, characterized in that: Step 2-2 specifically includes the following steps: Step 2-2-1: The edge server uses the CLIP image-text semantic consistency score as a cross-modal semantic trust anchor point. It compares the semantic consistency score of each distillation sample with a preset judgment threshold, directly eliminating semantically fragmented samples caused by backdoor attacks and low-quality distillation samples with excessive semantic deviation. Only compliant samples with high image-text semantic alignment are retained, thus obtaining secure distillation data. The secure data selection uses the following formula: （10）， in, This represents a secure distillation dataset after cross-modal semantic filtering. Indicates the first One distillation image sample; Indicates the first The true label corresponding to each distillation image sample; Indicates the first CLIP semantic consistency score of each distillation sample; This represents the preset semantic consistency threshold, the value of which is adjusted according to the distillation image effect, and is used to determine whether a sample has semantic fragmentation or backdoor risk; Step 2-2-2: Based on the semantic consistency scores of the filtered secure samples, the edge server establishes a positive correlation between the semantic consistency scores and data credibility, and generates a unique data credibility label. This label will serve as the core basis for subsequent AdaLoRA dynamic rank allocation on the cloud server side. The edge server binds the secure distillation images to the corresponding credibility labels one by one, constructs a complete secure distillation dataset containing semantic credibility annotations, and uploads it to the cloud server through an encrypted communication channel.

8. The method according to claim 7, characterized in that: Step 3-2 specifically includes the following steps: Step 3-2-1: Based on the sample credibility labels, the cloud server introduces a forgetting term into the total loss of the model fine-tuning for low-credibility data whose semantic consistency scores are below the trust threshold, constructing a joint loss function with a backdoor feature forgetting mechanism; the backdoor feature forgetting term uses the following formula: （15）， in, This represents a subset of data with low confidence, i.e., satisfying... The sample set; Indicates low confidence samples Trigger sample obtained after injecting a preset backdoor trigger; This represents the forward propagation function of the current global model; Represents the cross-entropy loss function; Indicates sample The true label; Indicates the target label for a backdoor attack; The first term, guiding the model, will trigger the sample to be correctly classified into the true label, while the second term, suppressing the model, will trigger the sample to be misclassified into the backdoor target label. The joint loss is expressed by the following formula: （16）， in, This represents the final fine-tuning loss after introducing the forgetting term; This represents the total loss from fine-tuning the original model; This represents the weight coefficient of the forgotten item; This represents the forgotten term of the backdoor features, used to suppress gradient propagation of residual backdoor features; Step 3-2-2: The cloud server calls the AdaLoRA low-rank adaptation parameter branch to restrict the fitting of hidden backdoor features in low-confidence data, completes gradient calculation, gradient accumulation and pruning based on the final joint loss, updates the global large model parameters, eliminates backdoor risks while retaining normal task features, and completes the safe adaptive fine-tuning of the global large model. Step 3-2-3: The cloud server uses a standard publicly available test dataset to verify the effectiveness of the fine-tuned model, calculating the clean sample accuracy and the backdoor attack success rate. The clean sample accuracy is calculated using the following formula: （17）， in, This represents the accuracy of the model on clean samples. This represents the total number of samples in the standard test set; Indicates the model's predicted label; Indicates the true label of the sample; This indicates an indicator function; a value of 1 indicates a correct prediction, and a value of 0 indicates an incorrect prediction. The success rate of backdoor attacks is calculated using the following formula: （18）， in, Indicates the success rate of backdoor attacks; This represents the total number of samples in the backdoor test set; Indicates the predicted label of the backdoor sample; Indicates the target label; Step 3-2-4: After the cloud server completes the verification of model performance and defense effectiveness, it saves the optimal model checkpoints and outputs a large, finely tuned security model that can be directly deployed.

Images