User data privacy protection-oriented visual transformer structured forgetting method and device

Abstract

The application discloses a visual Transformer structured forgetting method and device for user data privacy protection, and the method comprises the following steps: constructing a visual classification data set, completing image preprocessing, data division and labeling, forming a training set and a test set required by a forgetting task; embedding an MoE-LoRA adapter in a visual Transformer backbone network, performing hierarchical decoupled knowledge positioning training, and constructing a stable class-expert mapping relationship; performing backbone frozen selective knowledge removal, deleting target class knowledge without updating backbone parameters; adopting a triple constraint driven performance recovery mechanism, performing lightweight fine-tuning on the reserved classes, and suppressing incidental forgetting; sending the recovered model into an evaluation process, and outputting a final forgetting model. The device comprises a processor and a memory. The application realizes accurate removal of target class knowledge, complete preservation of non-target class performance, and efficient privacy protection under the premise of freezing the backbone network.

Description

Technical Field

[0001] This invention relates to the field of data privacy protection, and in particular to a visual Transformer structured forgetting method and apparatus for user data privacy protection. Background Technology

[0002] With the widespread application of deep learning technology in fields such as computer vision, natural language processing, and multimodal generation, large-parameter models, represented by the Vision Transformer (ViT), have become the mainstream infrastructure in both industry and academia. However, these models heavily memorize the feature distribution, sample information, and privacy attributes of the training data during training, making it difficult to completely remove the training data once it is incorporated into the model.

[0003] To address data privacy risks and compliance requirements, machine unlearning technology has become a core means of ensuring model privacy and security while meeting regulatory compliance. Machine unlearning aims to completely eliminate the influence of target training data or target categories by modifying model parameters (see [link to relevant documentation]). Figure 1 This allows the forgotten model to behave consistently with a de novo model that has never seen the target data. However, directly applying machine forgetting to visual Transformers presents significant technical bottlenecks:

[0004] First, the visual Transformer employs a multi-head self-attention mechanism, where category information is globally diffused and parameters are highly coupled. Existing general forgetting methods are mostly designed for localized feature structures such as convolutional neural networks, which are difficult to adapt to the above-mentioned structural characteristics of the visual Transformer and cannot accurately locate and completely remove the globally diffused category knowledge.

[0005] Secondly, existing ViT-specific forgetting methods (e.g., NOVO, LoRA-Unlearning) retain the backbone network parameters throughout the process and only achieve pseudo-forgetting by suppression or fine-tuning. The target knowledge remains significantly, making it vulnerable to member inference attacks that steal privacy.

[0006] Furthermore, methods such as LetheViT rely on full parameter fine-tuning and multiple rounds of gradient iteration, resulting in extremely high computational costs and making them difficult to deploy in practical systems.

[0007] Finally, forgetting the target category often destroys features of non-target categories, leading to a significant drop in overall model performance and insufficient practicality.

[0008] Therefore, there is still a lack of a structured solution for machine forgetting for visual Transformers that can simultaneously achieve true forgetting, efficient computation, preservation of model utility, and strong privacy protection. There is an urgent need to propose a dedicated structured forgetting framework for the structural characteristics of ViT, so as to take into account both theoretical and engineering application value. Summary of the Invention

[0009] This invention provides a visual Transformer structured forgetting method and apparatus for user data privacy protection. The invention aims to transform machine forgetting from parameter-level optimization to structure-level subspace deletion. Under the premise of freezing the backbone network, it achieves accurate removal of target category knowledge, complete preservation of non-target category performance, and efficient privacy and security protection. See the description below for details:

[0010] Firstly, a visual Transformer structured forgetting method for protecting user data privacy, the method comprising:

[0011] Construct a visual classification dataset, complete image preprocessing, data partitioning and labeling, and form the training set and test set required for the forgetting task;

[0012] Embedding the MoE-LoRA adapter in the visual Transformer backbone network enables hierarchical decoupling of knowledge localization training, thereby constructing a stable category-expert mapping relationship.

[0013] Perform selective knowledge removal by freezing the backbone, deleting target category knowledge without updating backbone parameters; adopt a performance recovery mechanism driven by triple constraints to perform lightweight fine-tuning of retained categories and suppress collateral forgetting;

[0014] The recovered model is sent to the evaluation process to output the final forgetting model.

[0015] The construction of a stable category-expert mapping relationship is as follows:

[0016] We selected three mainstream visual Transformers, ViT-B, DeiT, and Swin, as the backbones. We injected a MoE-LoRA adapter into each Transformer encoder layer. We adopted a sparse routing mechanism in both shallow and deep layers to orient different category features to independent expert subspaces and constrained them with category routing entropy loss so that each category only activates a small number of dedicated experts.

[0017] The statistical conditional routing distribution is used to select the expert with the highest probability to generate a category-expert mapping table and save it in a fixed manner, thus completing the knowledge structured positioning.

[0018] The step of deleting target category knowledge without updating the backbone parameters is as follows:

[0019] Receive forgetting requests and lock the category of the target to be forgotten; freeze all parameters of the visual Transformer backbone throughout the process, and do not perform backpropagation and gradient updates;

[0020] Based on the category-expert mapping table, query the expert module bound to the target category, and directly reset the corresponding low-rank adapter parameters to 0 or initialize them randomly; the target knowledge is completely deleted through only lightweight parameter operations.

[0021] The lightweight fine-tuning of the retained categories to suppress incidental forgetting is as follows:

[0022] On the preserved category dataset, orthogonal constraints, routing regularization constraints, and knowledge distillation optimization constraints are applied sequentially; lightweight fine-tuning is used to quickly restore model performance and maximize the protection of the discriminative ability of non-target categories.

[0023] Specifically, embedding a MoE-LoRA adapter into the visual Transformer backbone network to perform hierarchically decoupled knowledge localization training and construct a stable category-expert mapping relationship is as follows:

[0024] Let the input token be represented as Each expert corresponds to a set of low-rank parameters, and the output of each expert is represented as:

[0025] ;

[0026] Where x is the input token. Let x represent a d-dimensional real vector, R represent the real number field, and d represent the hidden dimension of the backbone network; This is the feature increment output generated by the i-th expert for the input token. Let be the low-rank upper projection matrix (upgraded dimension matrix) of the i-th expert. Let be the low-rank projection matrix (dimensionality reduction matrix) of the i-th expert, where r is the rank of LoRA, and r is much smaller than d. and Let r×d and d×r be the real matrix spaces respectively, and their subscripts be... Let E represent the index of the i-th expert, and E be the total number of experts.

[0027] Using a linear trainable router:

[0028] ;

[0029] in, For a trainable routing matrix, The router outputs an E-dimensional expert weight distribution vector, where each component represents the activation weight of each expert, and softmax is a normalized exponential function.

[0030] After softmax, the weight assignments for each expert are obtained. The overall MoE-LoRA output is obtained as follows:

[0031]

[0032] in, This represents the overall feature increment output by the MoE-LoRA adapter. The weight assigned to the i-th expert by the router, i.e. The i-th component, the summation index i ranges from 0 to E-1;

[0033] Introduce a class-based routing entropy loss during training:

[0034] ;

[0035] in, For category-based routing entropy loss, Let be the conditional routing probability, which is the conditional probability that the router activates expert e given a sample of class c. c is the class index. The outer summation iterates through all classes in the training set, e is the expert index, the inner summation iterates through all E experts, and log is the natural logarithm.

[0036] Minimizing the routing entropy loss of this category prompts each category to activate only a sparse set of experts, thus achieving structured localization of category knowledge;

[0037] Calculate conditional route distribution And select the k experts with the highest probability for each category:

[0038] ;

[0039] in, This is the set of dedicated experts for category c, i.e., the index set of the k experts with the highest conditional routing probabilities. Top-k represents the probability of selecting a conditional route. The calculation of the maximum k experts, where k is the number of dedicated experts selected for each category, and is a positive integer hyperparameter that is much smaller than the total number of experts E;

[0040] Obtain the category-expert mapping table:

[0041] ;

[0042] in, For the entry in the category-expert mapping table corresponding to category c, store the set of dedicated experts bound to category c. .

[0043] in,

[0044] The orthogonal constraint:

[0045] ;

[0046] in, For orthogonal constraint loss, and Given two distinct expert indices, sum and iterate through all satisfying... ≠ Experts on, and Experts With experts The low-rank projection matrix, where the superscript T denotes the matrix transpose and the subscript F is the notation for the Frobenius norm (F-norm). Represents the square root of the sum of the squares of all elements in the matrix;

[0047] minimize This will cause the parameter matrices of different experts to tend towards orthogonality;

[0048] The routing regularization constraint:

[0049] ;

[0050] in, For routing regularization terms, For a category index in the summation, For different Another type of index, summing and traversing all satisfying ≠ Category pair, and Categories With category The conditional routing probability distribution is given by KL divergence (Kullback-Leibler divergence, relative entropy), which measures the difference between two probability distributions. "∥" is the notation separating the two probability distributions in the KL divergence. Maximizing... This ensures that different categories form differentiated routing paths;

[0051] The knowledge distillation constraint:

[0052]

[0053] in, For knowledge distillation loss, The retained dataset consists of all samples from the retained categories. x represents the input sample in the retained dataset. The summation iterates through all samples in the retained dataset. M represents the pre-forgetting model, and M′ represents the post-forgetting model. and Let M and M′ be the logits (log odds) vectors output by the model before forgetting and the model after forgetting, respectively, for the input sample x. The subscript 2 and superscript 2 of the norm symbol indicate the square of the L2 norm (Euclidean norm) of the vector.

[0054] The final joint optimization objective is:

[0055]

[0056] .

[0057] in, For the joint loss function in the knowledge localization stage, Cross-entropy classification loss is used to ensure the basic classification performance of the model. These are the weighting coefficients for the category-based routing entropy loss. Let be the joint loss function for the knowledge retention stage, and Step 1 and Step 2 be the labels for the knowledge localization stage and the knowledge retention stage, respectively. For the aforementioned category of routing entropy loss, For orthogonal constraint loss The weighting coefficients, For routing regularization terms The weighting coefficients, Loss due to knowledge distillation The weight coefficients are indicated by "↓" for the term being minimized in the joint optimization and "↑" for the term being maximized in the joint optimization (in practice, a negative sign is used to minimize the total loss).

[0058] A second aspect is a visual Transformer structured forgetting device for user data privacy protection, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0059] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0060] The beneficial effects of the technical solution provided by this invention are:

[0061] 1. This invention proposes a structured knowledge localization architecture based on MoE-LoRA (Mixture of Experts-Low-Rank Adaptation), fundamentally solving the problem of highly coupled and difficult-to-precise knowledge removal in visual Transformers: This invention completely breaks the traditional forgetting method's reliance on global parameter fine-tuning, transforming category knowledge from a diffuse state into structured storage in independent subspaces; through hybrid expert sparse activation and lightweight encoding of low-rank adapters, each visual feature is dedicated to a small number of expert modules, combined with category routing entropy loss to achieve strong binding, simplifying knowledge removal from a complex optimization problem to a deterministic structural operation, truly achieving target category knowledge that is localizable, removable, and residue-free;

[0062] 2. This invention constructs a selective knowledge removal mechanism with trunk freezing, achieving efficient and low-cost true forgetting and solving the pain points of existing methods such as high computational cost and inability to completely forget: This invention keeps the core parameters of the visual Transformer completely frozen throughout the forgetting process, and completes knowledge deletion only through table lookup and low-rank parameter reset, without the need for multiple iterations or full updates. The computational cost and time consumption are significantly lower than traditional solutions such as full parameter fine-tuning and adversarial learning; at the same time, it does not retain any traces of target knowledge-related features, making the forgotten model strictly consistent with the initial model that was not trained on the target category.

[0063] 3. This invention designs a triple performance protection mechanism of orthogonal constraints, route regularization, and knowledge distillation to thoroughly alleviate collateral forgetting and achieve the optimal balance between forgetting effect and model utility: To address the problem that removing target knowledge can easily interfere with non-target category features, this invention uses orthogonal constraints to make different expert subspaces independent, route regularization to ensure that different category routing paths do not interfere with each other, and knowledge distillation to keep the output of the retained category stable; the three mechanisms work together to make the accuracy of the retained category close to the ideal upper limit, without accuracy collapse or feature drift.

[0064] 4. This invention achieves highly versatile and robust cross-architecture and cross-dataset forgetting, and is highly adaptable to actual deployment needs: This invention does not require modification of the backbone network structure, is plug-and-play compatible with mainstream visual Transformers such as ViT-B, DeiT, and Swin, and can run stably on multiple datasets such as CIFAR-10, CIFAR-100, and Tiny-ImageNet, supporting single-class forgetting and multi-class batch forgetting; at the same time, it can effectively resist member inference attacks, and the privacy protection index is close to the ideal random level, achieving synergistic optimization in privacy protection, forgetting effect, computational efficiency, and model performance.

[0065] 5. This invention demonstrates significant practical value in specific image application scenarios. In facial recognition systems, it can accurately delete facial feature representations based on user logout requests, preventing the biometric information of departing employees or logged-out users from being remembered and restored by the model, thus meeting compliance requirements for identity privacy protection. In medical image analysis scenarios, it can remove lesion image knowledge of specific patients as needed, avoiding the persistent retention of sensitive medical data in the model and protecting patient data sovereignty. In security monitoring and pedestrian re-identification scenarios, it can implement targeted forgetting of the appearance features of designated individuals, effectively avoiding the risks of data misuse and privacy infringement. In image content review and copyright protection scenarios, it can quickly erase visual knowledge corresponding to illegal categories or protected content, supporting compliant dynamic updates of the model.

[0066] The above scenarios all verify the strong adaptability and engineering deployability of the present invention under real image privacy protection needs, and provide practical and feasible technical support for the compliant operation of visual intelligence systems. Attached Figure Description

[0067] Figure 1 A schematic diagram illustrating the overall principle of machine forgetting is presented;

[0068] In this process, by modifying the model parameters, the influence of the target training data or target category is completely eliminated from the model, so that the behavior of the forgotten model is consistent with the initial model that has never seen the target data.

[0069] Figure 2 A flowchart illustrating the collaborative process of knowledge location, forgetting, and retention in three stages;

[0070] The diagram illustrates how the visual Transformer achieves precise selective machine forgetting by relying on a category-expert mapping mechanism.

[0071] Figure 3 Knowledge location map;

[0072] Among them, the precise binding of the completed category with the exclusive expert parameters locates the key knowledge module corresponding to the target category to be forgotten, and the snowflake represents the freeze operation.

[0073] Figure 4 Forgotten execution graph;

[0074] Among these measures, the expert parameters of the identified target category are reset to precisely erase the knowledge of the specified category.

[0075] Figure 5 Knowledge retention graph;

[0076] Among these, the original general knowledge of the constraint model is not destroyed, thus verifying the actual effect of selective forgetting. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0078] The technical terms involved in the embodiments of the present invention are explained below:

[0079] 1. ViT-B (Vision Transformer-Base, basic version of Vision Transformer).

[0080] The mainstream visual Transformer architecture divides the input image into fixed-size image patches, which are then linearly mapped into a token sequence. The global feature modeling is performed by a 12-layer stacked multi-head self-attention Transformer encoder. It has a hidden dimension of 768, 12 attention heads, and approximately 86 million parameters.

[0081] 2. DeiT (Data-efficient image Transformer);

[0082] Based on the ViT architecture, distillation tokens and knowledge distillation training strategies are introduced, along with strong data augmentation, enabling the model to be efficiently trained and converged on medium-sized datasets without the need for ultra-large-scale pre-training data, thus achieving mainstream visual Transformer architecture.

[0083] 3. Swin (Shifted Window Transformer).

[0084] By restricting self-attention computation to a local window and achieving cross-window information interaction through a shifting window mechanism between adjacent layers, a mainstream visual Transformer architecture with hierarchical multi-scale feature representation is constructed, whose computational complexity is linearly related to the image size.

[0085] 4. MoE-LoRA (Mixture of Experts-Low-Rank Adaptation) adapter;

[0086] The lightweight pluggable structure combines a hybrid expert mechanism with low-rank adaptation technology. It consists of E parallel low-rank adapter experts (E is the total number of experts, a positive integer) and a trainable router. The router assigns activation weights to each expert based on the input token features. Each expert generates incremental corrections to the backbone features through a set of low-rank matrices, thereby encoding category-specific knowledge in a structured form while keeping the backbone network parameters frozen.

[0087] 5. Sparse routing mechanism;

[0088] The router activates only a small number of experts with significant weights among E experts, while the weights of the remaining experts approach zero. In this embodiment of the invention, this is achieved by a combination of softmax routing and category routing entropy loss constraints, so that each input token is processed by only a few experts, thereby reducing computational overhead and promoting the separate storage of category knowledge.

[0089] 6. Independent expert subspace;

[0090] Each expert's low-rank parameter matrix spans a parameter subspace. Under orthogonal constraints, the subspaces of different experts are orthogonal, independent, and do not overlap, ensuring that different categories of knowledge are stored in their respective expert subspaces without interference.

[0091] 7. Category-based routing entropy loss constraint;

[0092] An entropy minimization constraint is imposed on the category-conditional routing probability distribution; by minimizing this loss, the routing distribution of each category tends to be concentrated, and only a small number of dedicated experts are stably activated, thereby establishing a strong binding relationship between categories and experts.

[0093] 8. A limited number of dedicated experts;

[0094] After the knowledge localization training converges, a certain category has k experts that are stably activated with the highest conditional routing probability. The discriminative knowledge of this category is mainly encoded in the low-rank adapter parameters of these experts, where k is a positive integer much smaller than the total number of experts E.

[0095] 9. Category-Expert Mapping Table;

[0096] Using category index c as the key, and the collection of experts specific to that category... A lookup table for values; after knowledge localization training converges, distribution is based on conditional routing. The statistics are generated and stored in a fixed format. During the execution phase, the expert bound to the target category can be directly located by querying this table, without the need to scan global parameters.

[0097] 10. Selective knowledge removal via trunk freezing;

[0098] The forgetting execution method keeps the parameters of the visual Transformer backbone network completely unchanged throughout the entire process (does not perform backpropagation and gradient update), only locates the expert bound to the target category based on the category-expert mapping table, and resets its low-rank adapter parameters to zero or randomly initializes them, thereby deleting the target category knowledge.

[0099] 11. Target category knowledge;

[0100] During training, the model learns discriminative feature representations and decision-making criteria for the target category to be forgotten. In this embodiment of the invention, this knowledge is structured and encoded in the low-rank adapter parameters of the dedicated expert bound to the category after knowledge localization training.

[0101] 12. Categories of targets to be forgotten;

[0102] One or more image categories whose influence needs to be completely removed from the model, as specified by user deletion requests, data authorization revocation, or compliance requirements; all samples corresponding to these categories in the training set constitute the dataset to be forgotten, and samples from the remaining categories constitute the retained dataset.

[0103] 13. Triple constraint;

[0104] The orthogonal constraint, routing regularization constraint and knowledge distillation constraint applied during the knowledge retention stage are collectively referred to as (see formulas (8), (9), and (10) respectively); the three work together to suppress the collateral effects of forgetting operations on non-target categories and protect the discriminative performance of the model on the retained categories after forgetting.

[0105] 14. Assess the completeness of forgetting;

[0106] The Top-1 classification accuracy of the model on test samples of the target category to be forgotten after forgetting (the subscript f represents forget, i.e., the forgotten category); the lower the value, the more thoroughly the target category knowledge is deleted, and ideally it should approach 0 or the level of random guessing.

[0107] 15. Evaluate the model's effectiveness;

[0108] The Top-1 classification accuracy of the model on the retained class test samples after forgetting (the subscript r means retain, i.e., the retained class); the closer this value is to the accuracy of the model on the same test samples before forgetting, the more completely the performance of the non-target class is retained.

[0109] 16. Assess privacy protection capabilities;

[0110] When launching a Membership Inference Attack (MIA) against a forgotten model, the attacker uses the Area Under the Curve (AUC) to distinguish whether a target sample has participated in model training. The closer this value is to 0.5 (random guess level), the less likely the attacker is to determine the membership of the sample to be forgotten, and the stronger the model's privacy protection capability.

[0111] 17. T assesses operational efficiency;

[0112] The time consumed to complete all forgetting operations for a single target category (including category-expert mapping table lookup, low-rank adapter parameter reset, and lightweight fine-tuning); the smaller this value, the higher the forgetting execution efficiency, and ideally it should be much lower than the time consumed by full parameter retraining.

[0113] Example 1

[0114] To address the security flaws of existing visual Transformer machine forgetting techniques: some methods retain backbone parameters throughout the process, achieve pseudo-forgetting only through suppression or fine-tuning, and leave a large amount of target category knowledge in the model parameters. This residual knowledge can be recovered by privacy attacks such as member inference attacks and model inversion attacks, resulting in the continuous exposure of sensitive data such as user faces and medical images to privacy risks, failing to meet data protection compliance requirements such as the "right to be forgotten." Furthermore, coarse parameter-level optimization leads to inaccurate forgetting of target knowledge, large-scale modifications to backbone network parameters cause severe performance degradation in non-target categories, and reliance on complete retraining results in excessively high computational costs. This invention provides a visual Transformer structured forgetting method, SUMA, for user data privacy protection. This method includes five core steps: dataset preparation, knowledge localization training, selective knowledge removal, performance recovery and knowledge retention, and forgetting effect evaluation, as detailed below:

[0115] 101: Construct a visual classification dataset, complete image preprocessing, data partitioning and labeling, and form the training set and test set required for the forgetting task;

[0116] This step includes: selecting three standard visual classification datasets—CIFAR-10, CIFAR-100, and Tiny-ImageNet—covering multi-granularity image classification scenarios of 10, 100, and 200 classes; uniformly scaling the images to the corresponding standard resolution; performing standard data augmentation operations such as pixel normalization, random cropping, and random flipping; dividing the dataset into training, validation, and test sets according to a common ratio; and specifying one or more categories as target categories to be forgotten based on forgetting requirements, with the corresponding samples in the training set constituting the dataset to be forgotten. The remaining categories are designated as retention categories, with their corresponding samples in the training set constituting the retention dataset. In other words, the objects to be forgotten are only the samples of the specified categories in the training set, not the entire training set.

[0117] 102: Embed the MoE-LoRA adapter in the visual Transformer backbone network to perform hierarchically decoupled knowledge localization training and build a stable category-expert mapping relationship;

[0118] This step includes: selecting three mainstream visual Transformers—ViT-B, DeiT, and Swin—as the backbone; injecting a MoE-LoRA adapter into each Transformer encoder layer; employing a sparse routing mechanism in both shallow and deep layers to directionally assign different category features to independent expert subspaces; and using category routing entropy loss constraints to ensure that each category activates only a small number of dedicated experts. After model convergence, the conditional routing distribution is statistically analyzed, and the expert with the highest probability is selected to generate a category-expert mapping table, which is then permanently saved, completing the knowledge-structured localization.

[0119] 103: Perform selective knowledge removal by freezing the main branch, deleting the target category knowledge without updating the main branch parameters;

[0120] This step includes: receiving a forgetting request and locking the target category to be forgotten; freezing all parameters of the visual Transformer backbone throughout the process, without performing backpropagation and gradient updates; querying the expert module bound to the target category according to the category-expert mapping table, and directly resetting the corresponding low-rank adapter parameters to 0 or randomly initializing them; without full parameter fine-tuning, the target knowledge is completely deleted through only lightweight parameter operations.

[0121] 104: After knowledge removal, a triple constraint-driven performance recovery mechanism is adopted to perform lightweight fine-tuning of the retained categories and suppress collateral forgetting;

[0122] This step includes: sequentially applying orthogonal constraints, routing regularization constraints, and knowledge distillation optimization constraints on the retained category dataset; orthogonal constraints make different expert subspaces independent of each other, reducing feature interference; routing regularization constraints maximize the differences in routing distributions of different categories, avoiding path confusion; knowledge distillation optimization constraints use the pre-forgetting model as the teacher to stabilize the retained category output distribution; and lightweight fine-tuning achieves rapid recovery of model performance, maximizing the protection of the discriminative ability of non-target categories.

[0123] 105: Submit the recovered model to the evaluation process, complete the indicator tests from four aspects: forgetting effect, model utility, privacy and security, and computational efficiency, and output the final forgetting model;

[0124] This step includes: calculating on the forgetting category. To assess the thoroughness of forgetting, calculate on the retention category. Evaluate model utility by computation through member inference attack. Assess privacy protection capabilities and calculate the time spent on forgetting a single category. Evaluate operational efficiency; comprehensively verify model performance by combining four indicators, and output a post-forgetting model that meets compliance requirements, is highly accurate, and highly efficient.

[0125] 106: Taking the CIFAR-10 dataset as an example, the "cat" category is set as the target category to be forgotten, and the other 9 categories are retained categories. A mixed test set containing cat images and other categories is input. After step 101, the images are scaled to 32×32 resolution, pixel normalized, and labeled. The results are then fed into the ViT-B backbone network, which has already undergone knowledge localization training in step 102. The expert module bound to the "cat" category is queried according to the category-expert mapping table. In step 103, the corresponding low-rank adapter parameters are directly reset to 0 without updating the backbone parameters. Subsequently, in step 104, a triple constraint lightweight fine-tuning is performed on the retained category images to restore the model's ability to distinguish between the nine retained categories, including "dog," "airplane," and "car." Finally, in step 105, the model is evaluated after forgetting. The results show that the model's prediction output for the input cat images degenerates to a uniform distribution, and the classification accuracy is close to the random level. The classification accuracy for the retained category images remains consistent with that before forgetting. The success rate of member inference attacks drops to close to the random baseline of 50%. The time consumption for single-category forgetting is significantly lower than that of the full-parameter retraining scheme, verifying the effectiveness and efficiency of this method in specific image processing.

[0126] 107: Taking medical image analysis as an example, this paper illustrates the specific application of this method in solving existing technical problems.

[0127] A medical image-assisted diagnostic system has been trained and deployed online based on lesion image datasets provided by multiple medical institutions, including lesion image category samples corresponding to patient B. Patient B or its data provider withdrew its data usage authorization in accordance with data protection regulations, requesting the system to remove the impact of the patient's lesion image knowledge on the model. The system, in step 101, marks the lesion image category corresponding to patient B as the target category to be forgotten; in step 102, it calls the fixed category-expert mapping table to locate the dedicated expert module bound to that category; in step 103, while freezing the backbone network parameters throughout, it only resets the low-rank adapter parameters of the dedicated expert to 0, completing the deletion of the target knowledge. This avoids the risks of residual knowledge being attacked by member inference, model inversion attacks, and ultimately exposing patient privacy, which are common in existing methods that retain backbone parameters throughout. Since the forgetting operation only involves a small number of lightweight parameter resets for dedicated experts, there is no need for extensive modifications to the backbone network. The modification does not require full retraining, significantly reducing computational costs compared to existing schemes and can be completed quickly during clinical system operation intervals. Step 104 performs triple constraint-based minor fine-tuning on the remaining lesion category samples of other patients, avoiding severe damage to the diagnostic performance of non-target lesion categories caused by coarse parameter-level optimization granularity. Step 105 evaluates the model after forgetting, and the results show that the recognition accuracy of lesion images of patient B drops to the random level, the success rate of member inference attacks drops to near the random baseline, and the diagnostic accuracy of lesion categories of other patients is basically consistent with that before forgetting. The time consumed by forgetting single-patient data is much lower than that of the full parameter retraining scheme.

[0128] The above applications demonstrate that this method can achieve the technical effects of complete forgetting of target knowledge, no loss of diagnostic performance for non-target categories, and low computational cost of forgetting, while meeting the requirements of patient data sovereignty and data protection compliance. It effectively solves the problems of knowledge residue, coarse forgetting granularity, incidental performance damage, and excessive computational cost of existing visual Transformer machine forgetting technology.

[0129] Example 2

[0130] The following section combines specific calculation formulas, numerical examples, and... Figures 2-5 The solution in Example 1 will be further described below:

[0131] 201: Data Preprocessing;

[0132] We used three standard datasets: CIFAR-10, CIFAR-100, and Tiny-ImageNet, with image sizes of 32×32, 32×32, and 64×64, respectively. All images underwent standard preprocessing: pixel values ​​were normalized to [0,1], and standard augmentation strategies were used to divide the training, validation, and test sets into a fixed ratio.

[0133] 202: Knowledge Positioning;

[0134] In the SUMA framework, the goal of the training phase is to build a base model with inherent forgetting capabilities, enabling it to maintain high classification performance while providing a well-initialized parameter space for subsequent forgetting operations. To this end, this embodiment of the invention injects a MoE-LoRA adapter into the ViT's Transformer module and trains the model end-to-end, allowing the LoRA adapter to encode category-specific knowledge in a deeply structured form.

[0135] Specifically, let the input token (a word, a serialized image patch feature unit) be represented as: The embodiments of the present invention introduce... There are 10 experts, each corresponding to a set of low-rank parameters. ,in It is a hidden dimension. Let represent the rank of LoRA. The output of each expert is represented as:

[0136] (1)

[0137] in, This is the feature increment output generated by the i-th expert for the input token. Let be the low-rank upper projection matrix (upgraded dimension matrix) of the i-th expert. Let be the low-rank projection matrix (dimensionality reduction matrix) of the i-th expert; subscript Let R represent the index of the i-th expert. In the formula, R represents the real number field. Represents a d-dimensional real vector space. and Let r represent the real matrix spaces of r×d and d×r, respectively.

[0138] To achieve conditional computation, this embodiment of the invention uses a linearly trainable router:

[0139] (2)

[0140] in, It is a trainable routing matrix. This represents the E-dimensional expert weight distribution vector output by the router, where each component represents the activation weight of each expert. After softmax (normalized exponential function) normalization, the weight assignment for each expert is obtained. Thus, the overall MoE-LoRA output is obtained:

[0141] (3)

[0142] in, The overall characteristic increment of the MoE-LoRA adapter output, where E is the total number of experts. The weight assigned by the router to the i-th expert is the i-th component of g(x); the summation index i in formula (3) ranges from 0 to E-1.

[0143] To enhance the correspondence between categories and experts, a category routing entropy loss is introduced during training:

[0144] (4)

[0145] in, Let c represent the conditional routing probability, where c is the class index (the outer summation iterates through all classes in the training set), and e is the expert index (the inner summation iterates through all E experts). That is, given samples of class c, the conditional probability that the router will activate expert e; log is the natural logarithm.

[0146] Minimizing the routing entropy loss for that category will prompt each category to activate only a sparse set of experts, thereby achieving structured localization of category knowledge.

[0147] After convergence, calculate the conditional route distribution. And select the k experts with the highest probability for each category:

[0148] (5)

[0149] in, This is the set of experts specific to category c, i.e., the index set of the k experts with the highest conditional routing probabilities, where Top-k represents the probability of selecting a conditional route. The calculation of the maximum k experts; k is the number of dedicated experts selected for each category, a positive integer hyperparameter that is much smaller than the total number of experts E.

[0150] This results in a category-expert mapping table:

[0151] (6)

[0152] in, For the entry in the category-expert mapping table corresponding to category c, store the set of dedicated experts bound to category c. .

[0153] This mapping transforms category-specific knowledge from a coupled parameter space into a localizable expert subspace, providing a structural basis for subsequent forgetting execution.

[0154] 203: Forgotten execution;

[0155] After knowledge localization, the forgetting process transforms from parameter optimization into a structural operation. Upon receiving the forgetting category... After receiving the request, perform the following steps:

[0156] (1) Freeze the ViT backbone parameters;

[0157] (2) Query Category - Expert Mapping Table ;

[0158] (3) Identify experts relevant to the target category and reinitialize their LoRA parameters:

[0159] (7)

[0160] in, Category of targets to be forgotten The corresponding set of dedicated experts can be found by querying the category-expert mapping table. get, The index of the expert belonging to the set; Formula (7) means to reset the low-rank matrix pair of each expert in the set to zero, where "→" means the parameter reassignment (reset) operation; in actual execution, the low-rank matrix pair can also be randomly reinitialized.

[0161] Since the MoE-LoRA adapter is integrated during initial training, category knowledge is deeply encoded in a specific expert subspace. Therefore, after knowledge localization, true forgetting can be achieved simply by resetting the parameters to zero without interfering with the knowledge representations of other categories. In terms of computational complexity, the mapping query and deletion operations avoid global parameter updates, thus enabling efficient forgetting.

[0162] 204: Knowledge Retention;

[0163] Given that the subspaces of the retained and forgotten categories may overlap, the forgetting operation may degrade the model's performance on the remaining data. Therefore, three core constraints are proposed for knowledge preservation:

[0164] (1) Orthogonal constraint:

[0165] (8)

[0166] in, For different Another expert's index, summing and traversing all satisfying ≠ Experts on, For experts The low-rank projection matrix is ​​denoted by F, which is the subscript of the Frobenius norm (F norm). ‖·‖F represents the square root of the sum of squares of all elements of the matrix; the superscript T represents the matrix transpose.

[0167] minimize This will cause the parameter matrices of different experts to tend to be orthogonal, thereby preventing multiple categories from sharing the same low-rank direction and reducing interference caused by subspace overlap.

[0168] (2) Routing regularization constraints:

[0169] (9)

[0170] Introduce a route diversity regularization term, where maximizing This ensures that different categories form differentiated routing paths, thereby preventing non-target categories from unintentionally relying on experts of the target category. For a category index in the summation, For different Another type of index, summing and traversing all satisfying ≠ Category pair, For category Conditional routing probability distribution For category Conditional routing probability distribution KL divergence (Kullback-Leibler divergence, relative entropy) measures the difference between two probability distributions. "∥" is the notation that separates two probability distributions in KL divergence.

[0171] (3) Knowledge distillation constraint:

[0172] (10)

[0173] in, To preserve the dataset, which consists of all samples of the preserved categories, The output of the model M before forgetting is the logits (log odds vector) of the input sample x. The output of the model M' after forgetting is logits (i.e., the log odds vector); the subscript 2 and superscript 2 of the norm symbol represent the square of the L2 norm (Euclidean norm) of the vector, and formula (10) is the sum of squares of the output differences of all samples in the retained dataset.

[0174] Preserving the dataset Above, the pre-forgetting model and post-forgetting models Perform distillation. Minimize. This helps the model adapt to forgetting operations and prevents the output of non-target categories from drifting.

[0175] Through these constraints, SUMA (Structured Unlearning via MoE-LoRA Adapters) establishes an isolated, localizable, stable, and removable modular framework, thereby maximizing the overall utility and robustness of the model. The final joint optimization objective of SUMA is:

[0176] (11)

[0177] (12)

[0178] in, For the knowledge localization stage (Step 1), the joint loss function is: Cross-entropy classification loss is used to ensure the basic classification performance of the model. The weighting coefficients for the category-based routing entropy loss are... The category routing entropy loss is defined by formula (4). For the joint loss function of the knowledge retention stage (Step 2), where cross-entropy loss... As the primary task loss in the SUMA framework, it is used to guarantee the model's basic classification performance. These are the weighting coefficients for the orthogonal constraint loss. The orthogonal constraint loss is defined by formula (8). These are the weight coefficients for the routing regularization term. The routing regularization term defined in formula (9) The weighting coefficients for knowledge distillation loss are... The knowledge distillation loss is defined by formula (10), "This is the label for the term that is minimized in the joint optimization." " is a marker indicating that the term is maximized in the joint optimization (in the actual implementation, it is included in the total loss for minimization with a negative sign).

[0179] 205: Specific Image Application Examples.

[0180] Taking the CIFAR-10 dataset as an example, the category "cat" is set as the target category to be forgotten, and the other 9 categories (airplane, car, bird, deer, dog, frog, horse, boat, truck) are retained categories.

[0181] Image input and preprocessing: Take a batch of mixed category test images with a resolution of 32×32, and complete pixel normalization and standard augmentation processing according to step 201. Among them, cat images are labeled as forgotten targets, and other category images are labeled as retained objects.

[0182] Knowledge localization stage: The preprocessed image is fed into the ViT-B backbone network embedded with the MoE-LoRA adapter. After end-to-end training in step 202, the cat image activates a fixed sparse subset of experts during forward propagation. Statistical conditional routing distribution is used to select the k experts with the highest probabilities, generating mapping relationships to completely localize the cat visual features in the corresponding expert subspaces.

[0183] Forgetting Execution Phase: Receives a forgetting request for the "cat" category, freezes all ViT-B backbone parameters, queries the mapping table, and directly resets all expert low-rank parameters bound to the cat category to 0, completing the deletion of the target knowledge. This process requires no backpropagation, involves only lightweight parameter zeroing operations, and single-category forgetting takes extremely short time.

[0184] Knowledge Preservation Stage: On the images of the preserved categories, orthogonal constraints, route regularization, and knowledge distillation are applied sequentially. Lightweight fine-tuning is performed according to the joint optimization objective of Step 2 to restore the model's ability to distinguish images of nine preserved categories, such as airplanes, cars, and dogs, and to prevent feature drift and accuracy collapse.

[0185] Forgetting effect verification: The model was input with cat images and retained category images after forgetting. The results showed that the classification output of cat images degenerated to a near uniform distribution, with the accuracy infinitely close to 0. The classification accuracy of retained category images remained consistent with the model before forgetting, and the performance loss was controlled within a very small range. When performing a member inference attack on cat images, the success rate of the attack dropped to a random baseline close to 50%, which verified that the privacy features of the target category images had been completely eliminated, and the model could no longer distinguish whether the image of that category had participated in the training.

[0186] Example 3

[0187] A visual Transformer structured forgetting device for user data privacy protection includes a processor, a memory, input and output interfaces, and a communication module. The memory stores program instructions, and the processor calls the program instructions stored in the memory to make the system execute the steps in Embodiment 1.

[0188] Construct a visual classification dataset, complete image preprocessing, data partitioning and labeling, and form the training set and test set required for the forgetting task;

[0189] Embedding the MoE-LoRA adapter in the visual Transformer backbone network enables hierarchical decoupling of knowledge localization training, thereby constructing a stable category-expert mapping relationship.

[0190] Perform selective knowledge removal by freezing the backbone, deleting target category knowledge without updating backbone parameters; adopt a performance recovery mechanism driven by triple constraints to perform lightweight fine-tuning of retained categories and suppress collateral forgetting;

[0191] The recovered model is sent to the evaluation process to output the final forgetting model.

[0192] The stable category-expert mapping relationship is constructed as follows:

[0193] We selected three mainstream visual Transformers, ViT-B, DeiT, and Swin, as the backbones. We injected a MoE-LoRA adapter into each Transformer encoder layer. We adopted a sparse routing mechanism in both shallow and deep layers to orient different category features to independent expert subspaces and constrained them with category routing entropy loss so that each category only activates a small number of dedicated experts.

[0194] The statistical conditional routing distribution is used to select the expert with the highest probability to generate a category-expert mapping table and save it in a fixed manner, thus completing the knowledge structured positioning.

[0195] Deleting target category knowledge without updating the trunk parameters is as follows:

[0196] Receive forgetting requests and lock the category of the target to be forgotten; freeze all parameters of the visual Transformer backbone throughout the process, and do not perform backpropagation and gradient updates;

[0197] Based on the category-expert mapping table, query the expert module bound to the target category, and directly reset the corresponding low-rank adapter parameters to 0 or initialize them randomly; the target knowledge is completely deleted through only lightweight parameter operations.

[0198] Among these measures, lightweight fine-tuning of the retention categories is performed to suppress collateral forgetting:

[0199] On the preserved category dataset, orthogonal constraints, routing regularization constraints, and knowledge distillation optimization constraints are applied sequentially; lightweight fine-tuning is used to quickly restore model performance and maximize the protection of the discriminative ability of non-target categories.

[0200] Specifically, a MoE-LoRA adapter is embedded in the visual Transformer backbone network to perform hierarchically decoupled knowledge localization training, constructing a stable category-expert mapping relationship as follows:

[0201] Let the input token be represented as Each expert corresponds to a set of low-rank parameters, and the output of each expert is represented as:

[0202] ;

[0203] Where x is the input token. Let x represent a d-dimensional real vector, R represent the real number field, and d represent the hidden dimension of the backbone network; This is the feature increment output generated by the i-th expert for the input token. Let be the low-rank upper projection matrix (upgraded dimension matrix) of the i-th expert. Let be the low-rank projection matrix (dimensionality reduction matrix) of the i-th expert, where r is the rank of LoRA, and r is much smaller than d. and Let r×d and d×r be the real matrix spaces respectively, and their subscripts be... Let E represent the index of the i-th expert, and E be the total number of experts.

[0204] Using a linear trainable router:

[0205] ;

[0206] in, For a trainable routing matrix, The router outputs an E-dimensional expert weight distribution vector, where each component represents the activation weight of each expert, and softmax is a normalized exponential function.

[0207] After softmax, the weight assignments for each expert are obtained. The overall MoE-LoRA output is obtained as follows:

[0208]

[0209] in, This represents the overall feature increment output by the MoE-LoRA adapter. The weight assigned to the i-th expert by the router, i.e. The i-th component, the summation index i ranges from 0 to E-1;

[0210] Introduce a class-based routing entropy loss during training:

[0211] ;

[0212] in, For category-based routing entropy loss, Let be the conditional routing probability, which is the conditional probability that the router activates expert e given a sample of class c. c is the class index. The outer summation iterates through all classes in the training set, e is the expert index, the inner summation iterates through all E experts, and log is the natural logarithm.

[0213] Minimizing the routing entropy loss of this category prompts each category to activate only a sparse set of experts, thus achieving structured localization of category knowledge;

[0214] Calculate conditional route distribution And select the k experts with the highest probability for each category:

[0215] ;

[0216] in, This is the set of dedicated experts for category c, i.e., the index set of the k experts with the highest conditional routing probabilities. Top-k represents the probability of selecting a conditional route. The calculation of the maximum k experts, where k is the number of dedicated experts selected for each category, and is a positive integer hyperparameter that is much smaller than the total number of experts E;

[0217] Obtain the category-expert mapping table:

[0218]

[0219] in, For the entry in the category-expert mapping table corresponding to category c, store the set of dedicated experts bound to category c. .

[0220] in,

[0221] Orthogonal constraints:

[0222] ;

[0223] in, For orthogonal constraint loss, and Given two distinct expert indices, sum and iterate through all satisfying... ≠ Experts on, and Experts With experts The low-rank projection matrix, where the superscript T denotes the matrix transpose and the subscript F is the notation for the Frobenius norm (F-norm). Represents the square root of the sum of the squares of all elements in the matrix;

[0224] minimize This will cause the parameter matrices of different experts to tend towards orthogonality;

[0225] Routing regularization constraints:

[0226]

[0227] in, For routing regularization terms, For a category index in the summation, For different Another type of index, summing and traversing all satisfying ≠ Category pair, and Categories With category The conditional routing probability distribution is given by KL divergence (Kullback-Leibler divergence, relative entropy), which measures the difference between two probability distributions. "∥" is the notation separating the two probability distributions in the KL divergence. Maximizing... This ensures that different categories form differentiated routing paths;

[0228] Knowledge distillation constraints:

[0229]

[0230] in, For knowledge distillation loss, The retained dataset consists of all samples from the retained categories. x represents the input sample in the retained dataset. The summation iterates through all samples in the retained dataset. M represents the pre-forgetting model, and M′ represents the post-forgetting model. and Let M and M′ be the logits (log odds) vectors output by the model before forgetting and the model after forgetting, respectively, for the input sample x. The subscript 2 and superscript 2 of the norm symbol indicate the square of the L2 norm (Euclidean norm) of the vector.

[0231] The final joint optimization objective is:

[0232]

[0233] .

[0234] in, For the joint loss function in the knowledge localization stage, Cross-entropy classification loss is used to ensure the basic classification performance of the model. These are the weighting coefficients for the category-based routing entropy loss.

[0235] Let be the joint loss function for the knowledge retention stage, and Step 1 and Step 2 be the labels for the knowledge localization stage and the knowledge retention stage, respectively. For the aforementioned category of routing entropy loss, For orthogonal constraint loss The weighting coefficients, For routing regularization terms The weighting coefficients, Loss due to knowledge distillation The weight coefficients are indicated by "↓" for the term being minimized in the joint optimization and "↑" for the term being maximized in the joint optimization (in practice, a negative sign is used to minimize the total loss).

[0236] This device can be deployed in the following scenarios:

[0237] (1) Cloud-based large model privacy compliance platform: Provides online forgetting service for deployed visual Transformer models, supports users to exercise the "right to be forgotten", and quickly deletes the impact of specified categories or specified data.

[0238] (2) Model security update system: In scenarios such as model iteration, data elimination, and backdoor removal, the target knowledge is accurately removed without affecting the overall performance and inference speed of the model, and there is no need to retrain the entire model.

[0239] (3) Edge privacy protection equipment: In edge vision equipment such as security, medical, and industrial inspection, local data forgetting and privacy clearing are realized to ensure that sensitive information is not leaked or left behind.

[0240] (4) Multi-user model sharing platform: In the scenario of multi-tenant sharing the model, it supports independent forgetting by user / by category, ensuring data isolation and privacy security for different users.

[0241] The entire system supports functions such as one-click forgetting requests, real-time progress feedback, automatic verification of forgetting effects, automatic reporting of performance indicators, and log retention. It can be seamlessly integrated into existing model management platforms, deep learning training frameworks, and cloud-native inference service systems.

[0242] Table 1

[0243]

[0244] As shown in Table 1 (the Method column represents the method name, Retrained represents the retraining baseline method, Finetune represents the fine-tuning baseline method, DSDA, BlindU, LoRA, LetheViT, and NOVO represent existing comparative forgetting methods, and SUMA (Ours) represents the method proposed in this embodiment; Accf represents the Top-1 accuracy of the forgotten category, with "↓" indicating that a lower value is better, and Accr represents the Top-1 accuracy of the retained category, with "↑" indicating that a higher value is better, and the values ​​in the table are in percentages), a comparison was made with the state-of-the-art forgetting methods on public datasets. It can be found that the SUMA forgetting system proposed in this embodiment has the best performance. Therefore, the SUMA system was deployed in a face recognition scenario.

[0245] Example 4

[0246] System Deployment Background: In facial recognition systems, models need to store a large amount of user facial feature knowledge for extended periods. When a user requests account cancellation, data authorization revocation, or compliant deletion, traditional solutions require retraining the entire dataset, which is time-consuming, costly, and cannot guarantee complete deletion. This embodiment uses the SUMA method to achieve accurate and structured forgetting of facial knowledge for a specified user. Specific execution flow:

[0247] 401: Forgot Request Reception: The system receives user cancellation requests through input / output interfaces, parses the target user's identity identifier, maps the corresponding face category to the internal category index of the model, forms a structured forget instruction, and writes it into the task queue.

[0248] 402: Face Knowledge Localization: The processor calls the trained category-expert mapping table in memory, queries the MoE-LoRA expert subspace bound to the target user's face category, and accurately locates the storage location of the user's face features in each layer of the ViT encoder without scanning global parameters.

[0249] 403: Structured Forgetting Execution: The system freezes the core parameters of the visual Transformer throughout the process, only resetting the low-rank adapter parameters (Aei, Bei) corresponding to the target user's face category to 0, thus completely deleting the face feature knowledge. The entire forgetting operation is completed within milliseconds to seconds, without interrupting the system's normal recognition service for other online users.

[0250] 404: Performance recovery for non-target users: After the forgetting process, the system applies a triple mechanism of orthogonal constraints, routing regularization, and knowledge distillation to retain user face data, and performs lightweight fine-tuning to ensure that the face recognition accuracy of other registered users is not affected, and to avoid an increase in the false recognition rate caused by incidental forgetting.

[0251] 405: Forgotten Effect Verification and Log Retention: The system automatically performs member inference attack tests on the target user's face image to verify whether the attack success rate has dropped to the random baseline level (≈50%); at the same time, it examines the model output distribution of the target user's face input to confirm that the prediction confidence has degraded to a uniform distribution; all verification results, along with the forgetting timestamp and operation log, are automatically archived to support regulatory audits and compliance evidence retention.

[0252] 406: Expanded System Application Scenarios:

[0253] (1) Deregistration of facial recognition data of departing employees: After an employee leaves the company, the system deletes all knowledge residues of their facial features in the recognition model with one click, preventing departing employees from restoring their biometric information through model inversion.

[0254] (2) Privacy clearing of public security system: When a person no longer needs to be monitored or their authorization is revoked, their facial category knowledge is accurately removed to ensure that the privacy rights of citizens' portraits are not permanently remembered by the model.

[0255] (3) User logout in multi-tenant face recognition platform: In a platform where multiple users share face recognition services, it is supported to independently forget users at the individual user level, ensuring that logout user data is completely isolated from online user data.

[0256] (4) Face data compliance audit response: When regulatory agencies require data deletion verification, the system can automatically output a forgetting effect verification report, provide quantifiable privacy deletion proof, and meet the compliance requirements of data protection regulations such as GDPR.

[0257] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0258] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0259] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0260] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0261] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0262] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0263] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0264] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual Transformer structured forgetting method for user data privacy protection, characterized in that, The method includes: Construct a visual classification dataset, complete image preprocessing, data partitioning and labeling, and form the training set and test set required for the forgetting task; Embedding the MoE-LoRA adapter in the visual Transformer backbone network enables hierarchical decoupling of knowledge localization training, thereby constructing a stable category-expert mapping relationship. Perform selective knowledge removal by freezing the backbone, deleting target category knowledge without updating backbone parameters; adopt a performance recovery mechanism driven by triple constraints to perform lightweight fine-tuning of retained categories and suppress collateral forgetting; The recovered model is sent to the evaluation process to output the final forgetting model.

2. The visual Transformer structured forgetting method for user data privacy protection according to claim 1, characterized in that, The stable category-expert mapping relationship is constructed as follows: We selected three mainstream visual Transformers, ViT-B, DeiT, and Swin, as the backbones. We injected a MoE-LoRA adapter into each Transformer encoder layer. We adopted a sparse routing mechanism in both shallow and deep layers to orient different category features to independent expert subspaces and constrained them with category routing entropy loss so that each category only activates a small number of dedicated experts. The statistical conditional routing distribution is used to select the expert with the highest probability to generate a category-expert mapping table and save it in a fixed manner, thus completing the knowledge structured positioning.

3. The visual Transformer structured forgetting method for user data privacy protection according to claim 1, characterized in that, The method of deleting target category knowledge without updating the backbone parameters is as follows: Receive forgetting requests and lock the category of the target to be forgotten; freeze all parameters of the visual Transformer backbone throughout the process, and do not perform backpropagation and gradient updates; Based on the category-expert mapping table, query the expert module bound to the target category, and directly reset the corresponding low-rank adapter parameters to 0 or initialize them randomly; the target knowledge is completely deleted through only lightweight parameter operations.

4. The visual Transformer structured forgetting method for user data privacy protection according to claim 1, characterized in that, The lightweight fine-tuning of the retention categories to suppress incidental forgetting is as follows: On the preserved category dataset, orthogonal constraints, routing regularization constraints, and knowledge distillation optimization constraints are applied sequentially; lightweight fine-tuning is used to quickly restore model performance and maximize the protection of the discriminative ability of non-target categories.

5. The visual Transformer structured forgetting method for user data privacy protection according to claim 1, characterized in that, The method involves embedding a MoE-LoRA adapter into the visual Transformer backbone network to perform hierarchically decoupled knowledge localization training and construct a stable category-expert mapping relationship as follows: Let the input token be represented as Each expert corresponds to a set of low-rank parameters, and the output of each expert is represented as: ； Where x is the input token. Let x represent a d-dimensional real vector, R represent the real number field, and d represent the hidden dimension of the backbone network; This is the feature increment output generated by the i-th expert for the input token. Let be the low-rank upper projection matrix of the i-th expert. Let r be the low-rank projection matrix of the i-th expert, and r be the rank of LoRA. and Let r×d and d×r be the real matrix spaces respectively, and their subscripts be... Let E represent the index of the i-th expert, and E be the total number of experts. Using a linear trainable router: ； in, For a trainable routing matrix, The router outputs an E-dimensional expert weight distribution vector, where each component represents the activation weight of each expert, and softmax is a normalized exponential function. After softmax, the weight assignments for each expert are obtained. The overall MoE-LoRA output is obtained as follows: ； in, This represents the overall feature increment output by the MoE-LoRA adapter. The weight assigned to the i-th expert by the router, i.e. The i-th component, the summation index i ranges from 0 to E-1; Introduce a class-based routing entropy loss during training: ； in, For category-based routing entropy loss, Let be the conditional routing probability, which is the conditional probability that the router activates expert e given a sample of class c. c is the class index. The outer summation iterates through all classes in the training set, e is the expert index, the inner summation iterates through all E experts, and log is the natural logarithm. Minimizing the routing entropy loss of this category prompts each category to activate only a sparse set of experts, thus achieving structured localization of category knowledge; Calculate conditional route distribution And select the k experts with the highest probabilities for each category: ； in, This is the set of dedicated experts for category c, i.e., the index set of the k experts with the highest conditional routing probabilities. Top-k represents the probability of selecting a conditional route. The calculation of the maximum k experts, where k is the number of dedicated experts selected for each category, and is a positive integer hyperparameter that is much smaller than the total number of experts E; Obtain the category-expert mapping table: ； in, For the entry in the category-expert mapping table corresponding to category c, store the set of dedicated experts bound to category c. .

6. The visual Transformer structured forgetting method for user data privacy protection according to claim 4, characterized in that, The orthogonal constraint: ； in, For orthogonal constraint loss, and Given two distinct expert indices, sum and iterate through all satisfying... ≠ Experts on, and Experts With experts The low-rank projection matrix, where the superscript T denotes the matrix transpose and the subscript F is the notation for the Frobenius norm. Represents the square root of the sum of the squares of all elements in the matrix; minimize This will cause the parameter matrices of different experts to tend towards orthogonality; The routing regularization constraint: ； in, For routing regularization terms, For a category index in the summation, For different Another type of index, summing and traversing all satisfying ≠ Category pair, and Categories With category The conditional routing probability distribution is given by KL divergence, which measures the difference between two probability distributions. "∥" is the notation separating the two probability distributions in the KL divergence. Maximizing... Ensure that different categories form differentiated routing paths; The knowledge distillation constraint: ； in, For knowledge distillation loss, The retained dataset consists of all samples from the retained categories. x represents the input sample in the retained dataset. The summation iterates through all samples in the retained dataset. M represents the pre-forgetting model, and M′ represents the post-forgetting model. and These are the logits vectors output by the model M before forgetting and the model M′ after forgetting for the input sample x, respectively. The subscript 2 and superscript 2 of the norm symbol indicate the square of the L2 norm of the vector. The final joint optimization objective is: 。 in, For the joint loss function in the knowledge localization stage, Cross-entropy classification loss is used to ensure the basic classification performance of the model. The weighting coefficients for the category-based routing entropy loss are... Let be the joint loss function for the knowledge retention stage, and Step 1 and Step 2 be the labels for the knowledge localization stage and the knowledge retention stage, respectively. For the aforementioned category of routing entropy loss, For orthogonal constraint loss The weighting coefficients, For routing regularization terms The weighting coefficients, Loss due to knowledge distillation The weight coefficients are indicated by "↓" for terms that are minimized in the joint optimization and "↑" for terms that are maximized in the joint optimization.

7. A visual Transformer structured forgetting device for user data privacy protection, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.

Images