Forgetting learning method based on causal knowledge decoupling

Abstract

The invention discloses a forgetting learning method based on causal knowledge decoupling, and aims to solve the balance problem of excessive forgetting (low fidelity) or incomplete forgetting (low effectiveness) easily occurring when shared knowledge is processed by the existing forgetting learning technology. The core of the method is that a confounding factor of shared knowledge is identified through causal analysis, and the confounding factor is decoupled into attribute-level semantics for refined removal. The method mainly comprises the following steps: firstly, decomposing sample characteristics into cross-category shared attribute semantics and category-specific unique attribute semantics by using a knowledge decoupling module based on a variational auto-encoder (beta-VAE); secondly, an anti-fact reasoning generation module is used for synthesizing an anti-fact sample located near a decision boundary through the unique attribute of the reserved class and the shared attribute of the class to be forgotten by logically combining the unique attribute of the reserved class and the shared attribute of the class to be forgotten; thirdly, through a knowledge refining module, using the generated anti-fact sample as an anchor point, and constructing a decision boundary under a comparative learning framework; and finally, constraint optimization is carried out through a reconstructed forgetting learning loss function, the loss function is composed of multiple retention loss (MRL) for ensuring non-target class performance of the model and contrast forgetting loss (CFL) for enhancing forgetting boundary rigidity, and forgetting balance between fidelity and effectiveness is realized. According to the method, through decoupling and reconstruction of attribute-level semantics, the robustness, generalization ability and calculation efficiency of the model after the data forgetting request is executed are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data privacy protection technology, and more specifically, to a forgetting learning method for privacy erasure of deep neural networks, particularly a forgetting learning method based on causal knowledge decoupling and counterfactual reasoning. Background Technology

[0002] With increasing awareness of data privacy protection and the introduction of related regulations (such as the General Data Protection Regulation (GDPR), users have been granted the "right to be forgotten." Machine Unlearning, a technique that removes specific private data from a trained model without retraining from scratch, has become a research hotspot. Its core objective is to maintain the model's fidelity (performance on the remaining data) while ensuring effectiveness (complete removal of the target data).

[0003] However, existing forgetting learning methods still face significant challenges in practical applications. This invention, through in-depth causal analysis, reveals that the limitations of existing technologies primarily stem from their improper handling of "shared knowledge":

[0004] 1. Excessive Forgetting Problem: Since there is often semantic overlap between the target class (data to be forgotten) and the retention class (e.g., visual attributes such as "hairy" and "four legs" shared by "cat" and "tiger"), existing methods are prone to accidentally deleting this cross-class shared knowledge when removing the target class, which leads to a significant drop in model performance when processing retention class data, i.e., low fidelity.

[0005] 2. Incomplete Forgetting Problem: To maintain fidelity, some methods tend to retain shared features. However, these residual feature maps create confounding effects in the feature space, allowing the model to implicitly identify category attributes that should be forgotten through shared features, resulting in incomplete forgetting and thus low effectiveness.

[0006] 3. Confounding Factors: Structural causal model (SCM) analysis reveals that the observed sample features include both "shared attributes" and "unique attributes." In existing forgetting learning processes, shared knowledge acts as a confounding factor, establishing a non-causal path between the forgotten category and the prediction result. Most existing techniques only perform heuristic fine-tuning at the model parameter level, lacking a mechanism to effectively decouple shared and unique knowledge at the semantic level.

[0007] 4. Ambiguity of the forgetting boundary: Due to the lack of fine-grained control over attribute-level semantics, existing methods struggle to define precise forgetting decision boundaries, especially when dealing with categories with high semantic similarity. They are unable to establish rigid boundaries that can both thoroughly eliminate target privacy and accurately protect shared features.

[0008] Therefore, how to identify and decouple shared knowledge from a causal perspective, and how to reconstruct a more rigid decision boundary using the decoupled semantic information to achieve balance, is a key technical challenge that urgently needs to be solved in the field of forgetting learning. Summary of the Invention

[0009] The technical problem this invention aims to solve is the balance between over-forgetting (low fidelity) and incomplete forgetting (low effectiveness) that existing forgetting learning techniques often encounter when processing shared knowledge with semantic overlap. This invention provides a forgetting learning method based on causal knowledge decoupling. This method identifies and decouples shared knowledge as a confounding factor through causal analysis, thereby ensuring high fidelity of the model for retained data while achieving efficient forgetting of the target data.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] This invention proposes a forgetting learning method based on causal knowledge decoupling. Its core idea is to use structural causal models (SCM) to identify and decouple shared knowledge as a "confounding factor," simulate the distribution of decision boundary regions through counterfactual reasoning, and refine the model using a reconstructed loss function within a contrastive learning framework. This method effectively resolves the conflict between excessive and incomplete forgetting inherent in traditional methods, ensuring high fidelity of the model for retained data while achieving efficient erasure of target data, thus achieving an optimal balance performance.

[0012] This method mainly consists of the following core modules and steps:

[0013] 1. Decoupling modules by incorporating knowledge from β-VAE:

[0014] This module is responsible for performing feature extraction and decoupling operations based on causal logic, decomposing complex sample features into mutually independent attribute-level semantic groups. Feature decoupling architecture: A dual-encoder structure is constructed using a variational autoencoder (VAE). The first encoder extracts cross-class shared attribute semantics, while the second encoder extracts unique attribute semantics with class-discriminative characteristics. Multi-dimensional semantic supervision: Shared attributes are supervised by an observation classifier to capture the observation context, ensuring it is class-independent; simultaneously, unique attributes are supervised by a class classifier to capture the core class label. Decoupling optimization constraints: Optimization is achieved using the weighted evidence lower bound (ELBO) loss function, which combines reconstruction loss and a hyperparameter-adjusted KL divergence term, thereby achieving attribute-level decoupling in the semantic space and eliminating the interference of confounding factors.

[0015] 2. Counterfactual sample generation:

[0016] This module utilizes decoupled attribute semantics for "imaginary" reasoning, simulating sample distribution near the decision boundary by constructing counterfactual data: Semantic abduction: Extracting shared attribute feature vectors from samples to be forgotten. Intervention: Randomly sampling unique attribute feature vectors from retained samples (e.g., extracting the "wing" attribute from birds), and performing an intervention to combine the two. Predictive synthesis: Decoding the combined vector into a counterfactual sample using a decoder. This sample accurately simulates the scenario of combining the "core semantics of the retained category" and the "shared background of the forgotten category" in the feature space, used to precisely locate the forgetting boundary.

[0017] 3. Knowledge Refinement Module:

[0018] This module is responsible for performing the final model update operation, achieving fine-tuning of the forgetting boundary through a reconstructed loss function: Multiple retention constraints: Calculate the distributional differences between the original model and the model to be updated when handling counterfactual samples. This module ensures the model maintains semantic invariance of shared attributes during the forgetting process, guaranteeing the model's fidelity to the retained data and preventing excessive forgetting. Contrastive forgetting mechanism: Under the contrastive learning framework, counterfactual samples and samples to be forgotten are defined as positive sample pairs to bring them closer, while samples to be forgotten are defined as negative sample pairs to push them apart. Boundary rigidity reinforcement: Through the above contrastive operations, the model is forced to shift the feature representation of the forgotten data towards random anchor points, thereby enhancing the rigidity of the forgetting boundary and solving the problem of incomplete forgetting caused by semantic overlap.

[0019] 4. Loss Balance and Parameter Update: Comprehensive Loss Construction

[0020] The final update objective is a weighted average of multiple retention loss and contrastive forgetting loss. Adaptive adjustment: balancing exploration (forgetting intensity) and convergence (performance preservation) is achieved by adjusting hyperparameters and temperature parameters.

[0021] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0022] 1. This invention identifies and decouples shared knowledge as a confounding factor through causal analysis, achieving an effective balance between fidelity and effectiveness. It achieves a balance between accuracy on retained data and clearance rate on forgotten data, approaching the performance of a de novo training model.

[0023] 2. This invention generates counterfactual samples near the decision boundary through a "causal-intervention-prediction" logic, and establishes a new decision boundary with high rigidity by combining a contrastive learning mechanism. Compared with traditional methods, this invention significantly reduces the success rate of Membership-Based Argumentation (MLA) attacks and effectively suppresses knowledge residue caused by semantic overlap, thereby ensuring the thoroughness of data erasure and privacy security.

[0024] 3. This method eliminates the need for retraining on the complete dataset. While maintaining high accuracy, its computational cost is significantly lower than training a model from scratch, and it requires only a minimal amount of additional data to achieve efficient forgetting. The modular design of this framework allows it to flexibly adapt to different neural network architectures and can be widely applied to proprietary privacy-preserving tasks across various vertical industries. Attached Figure Description

[0025] Figure 1 This is a schematic diagram comparing the prediction performance of the present invention with existing forgetting learning methods;

[0026] Figure 2 This invention relates to a structural causal model (SCM) analysis diagram;

[0027] Figure 3 This is a diagram of the forgetting learning system architecture proposed in this invention;

[0028] Figure 4 This is a schematic diagram illustrating the process of establishing rigid decision boundaries in this invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention will be described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0030] This invention provides an implementation example of a forgetting learning method based on causal knowledge decoupling. The method is implemented in typical image classification task scenarios, such as on the CIFAR-10 or CIFAR-100 datasets, using ResNet-18 or ResNet-34 as the basic network architecture. The goal is to quickly remove relevant privacy information and maintain the model's performance on the remaining data after receiving a forgetting request for a specific category.

[0031] Step 1, as follows Figure 1 As shown, this invention first conducts an in-depth analysis of the two core pain points in current forgetting learning:

[0032] Excessive forgetting: When performing forgetting operations, the model may accidentally delete shared knowledge in the retained categories that is similar to the deleted categories, resulting in a significant drop in the model's performance on retained data (i.e., low fidelity).

[0033] Incomplete forgetting: Due to the existence of shared knowledge, the residual feature-to-category mapping may be accidentally activated, causing knowledge that should be deleted to be recalled by the model, making it impossible to completely remove privacy information (i.e., low effectiveness).

[0034] Step two, as Figure 2 As shown, this invention models and analyzes the above problem by constructing a structural causal model (SCM):

[0035] Causal variable definition: The model includes shared knowledge K and forgotten data D. f , retain data D r The features are represented by E and the predicted output is Y. The expected goal of forgetting learning is to forget the data D. f In order to obtain a risk-free prediction result Y r The prediction result satisfies Y r =YY f .

[0036] Meaning of each causal path: K→D f and K→D r This indicates that there is shared knowledge between retained and forgotten data, meaning that K acts as a confounding factor, affecting both simultaneously; D f →E and D r →E indicates that the model's feature representations are extracted from both retained and forgotten data; D f →Y and D r →Y represents the direct causal effect from image data to label prediction. E→Y represents the final prediction based on feature representation.

[0037] Problem path analysis: Over-forgetting refers to the tendency of existing forgetting mechanisms to sever all connections related to forgotten data, which often leads to the erroneous removal of the causal path K→D. r This operation inadvertently forgets the shared knowledge contained in the retained data, thus triggering a performance collapse of the model on the retained dataset. Incomplete forgetting refers to the fact that, in order to protect model performance, some mechanisms maintain the causal path K→D. fThis residual connection means that shared knowledge can still activate features of the forgotten category, thus hindering accurate forgetting of the target knowledge. Over-forgetting: When performing forgetting operations, the model's performance on retained data degrades significantly (i.e., low fidelity) because it mistakenly deletes shared knowledge in the retained category that is similar to the deleted category.

[0038] To clarify the main problems faced by balancing forgetting learning, this invention models it as a two-level classification problem: 1) multi-class prediction for each retained class; 2) binary prediction between the forgotten class and all retained classes. Based on this, the goal of forgetting learning can be specifically described by two adjustment loss functions: multiple retention loss and contrastive forgetting loss. These two loss functions focus on constraining knowledge related to fidelity and effectiveness, aiming to remove shared knowledge K as a confounding factor. Specifically, this approach employs a two-step optimization strategy to effectively solve the problem of constructing a rigid forgetting boundary (i.e., the decision boundary between the forgotten class and the retained classes) in the semantic space: First, the decision boundary is roughly outlined using knowledge based on the significantly retained classes to prioritize ensuring high model fidelity; then, feature knowledge is obtained by significantly contrasting the forgotten class and the retained classes to explicitly reshape and strengthen the rigidity of the boundary, ensuring high model effectiveness. In contrast, traditional forgetting learning loss functions are highly coupled when dealing with fidelity and effectiveness knowledge, making it difficult to finely distinguish between the two, thus failing to effectively suppress over-forgetting and incomplete forgetting. Specifically, to address the above problems, this invention constructs a new causal path K→D c →E′ and Y, through counterfactual data D c Replace the original path without cutting K→D r Under the premise of achieving D f The complete stripping away of it.

[0039] Step 3, as Figure 3 As shown, a knowledge decoupling module combining β-VAE is constructed to eliminate the confounding effects of shared knowledge. Based on the causal analysis in step two, in order to block the interference of the confounding factor K, this invention utilizes a variational autoencoder β-VAE to construct a decoupling network, decomposing image features into mutually independent latent variables in the semantic space. The specific implementation process is as follows:

[0040] This invention designs an encoder-decoder structure to infer unobservable variables S and U under two observable constraints: observation context O and class label Y. Specifically, we first decompose each sample into 1) a unique attribute U corresponding to inherent information and class-related information to predict the label Y, and 2) shared attributes S and attributes shared between different classes. This decomposition is achieved by extracting S and U from the samples using encoders φ and ψ respectively, where φ encodes the shared attributes as Q. φ(S|X), ψ encodes the class's unique attribute as Q. ψ (U|X). Then, based on the Markov condition between factors, we solve the joint distribution P(S, U|X) = Q of the entangled properties. φ (S|X)Q ψ (U|X):

[0041]

[0042] Where P(X, S, U, O, Y) represents the joint distribution of all factors, and P(X|S, U) represents the decoder P θ The reconstructed output, P(O|S), is determined by the classifier θ. O Predict that P(Y|U) is determined by the classifier θ S Prediction. To ensure that the above decoupling process is constrained by the joint distribution P(X, S, U, O, Y) defined by the dataset, this invention maximizes the weighted evidence lower bound (ELBO) loss function L. U The network parameters are optimized. The loss function is defined as follows:

[0043]

[0044] The first two items supervise classification accuracy, while the third item improves reconstruction quality by generating original samples. KL Let φ represent the Kullback-Leibler divergence to maintain consistency between the unwrapped properties and their prior distributions P(S) and P(O), where β is a weighting parameter. Essentially, ELBO optimizes the unwrapping in a self-supervised manner, where φ is determined by θ. O Supervision focuses on the observation context of sample X, regardless of its class. Meanwhile, ψ is determined by θ. S Supervision focuses on category-unique semantics, excluding confounding factors from shared knowledge, in order to further generate counterfactual samples.

[0045] Step 4: Based on the decoupling distribution Q obtained in Step 3 φ (S|X) and Q ψ (U|X), this invention utilizes counterfactual reasoning to generate counterfactual D. c And through three typical steps of "causal attribution-intervention-prediction", counterfactual samples x located near the decision boundary are generated. c The specific implementation process is as follows:

[0046] Causation: For the dataset D to be forgotten f Each forgotten sample x f Using encoder Q φ (S|x=x fSampling was performed on the "tiger" sample to extract its shared attributes (e.g., extracting the shared attribute "hairy" from the "tiger" sample) as the background for counterfactual reasoning. Intervention: From the retained dataset D r A sample x is randomly selected from the sample. r and using encoder Q ψ (U|X=x r Extract its unique attribute (e.g., extract the unique attribute "has wings" from the "bird" sample). Then perform an intervention to combine the shared attribute with the unique attribute of the preserved sample. Prediction: Utilize decoder P. e (X|S=s f U = u r Decode the combined attributes to generate a counterfactual sample x. c The sample (e.g., "a tiger with wings") is located precisely near the decision boundary between the forgotten category (e.g., tiger) and the retained category (e.g., bird) in the feature space, and is used for subsequent precise forgetting.

[0047] Step 5: Based on the dual-objective knowledge refinement module, utilize the generated counterfactual dataset D c This invention refines model knowledge by establishing new causal paths, specifically including two optimization objectives: memory and forgetting. Multiple Retention Loss (MRL) optimization is used to establish the causal path K→D. c →Y is used to resolve the issue of retaining data D. r To address the confounding issue of counterfactual samples x, this invention designs a Multiple Retention Loss (MRL). c With forgotten sample x c The shared attribute S is used, and this loss aims to keep S constant during the forgetting process. The calculation formula is as follows:

[0048]

[0049] Among them, h w0 (*) represents the original model, h wu (*) denotes the forgetting model. This is achieved by minimizing L... M This makes the predicted distribution of forgotten samples by the forgetting model close to the predicted distribution of counterfactual samples by the original model, thus allowing U f Approaching multiple U r At the same time, maintain the shared attribute S f Unchanged. This mechanism implicitly makes D c Replaced D f This successfully eliminated confounding factors.

[0050] Contrastive Forgetting Loss (CFL) Optimization: To establish the causal path K→D c→To enhance the feature representation E′, this invention introduces a contrastive learning mechanism. Specifically, counterfactual samples and forgotten samples are considered positive sample pairs, and retained samples and forgotten samples are considered negative sample pairs. Contrastive forgetting loss (CFL) is used to bring positive sample pairs closer together while pushing away negative sample pairs. The calculation formula is as follows:

[0051]

[0052] Where τ is a scalar temperature parameter, z = h wu (x) represents the extracted feature representation, z f , z p , z n These correspond to the features of forgotten samples, positive samples (counterfactual samples), and negative samples (retained samples), respectively.

[0053] Step 6, based on the final total loss function L = L M +λL C Where λ is a hyperparameter used to balance memory and forgetting, K→D is achieved through joint optimization. c →Causal reconstruction of E′ and Y.

Claims

1. A forgetting learning method based on causal knowledge decoupling, characterized in that, Includes the following steps:

1. Knowledge Decoupling Module This module is responsible for performing feature extraction and decoupling operations based on the Structural Causal Model (SCM), aiming to identify and separate shared knowledge as a confounding factor. This module decouples unobservable attributes into independent semantic groups by constructing an encoder-decoder structure. Semantic feature encoding: using the first encoder Q φ Extract the shared attribute semantics S across categories in the samples, and use the second encoder Q. ψ Extracting unique attribute semantics U with category-discriminating characteristics; Multi-dimensional semantics: Constructing an observation classifier θ O By minimizing the observation loss, the shared attribute semantics S is ensured to capture the category-independent observation context O; a category classification θ is constructed. Y By minimizing the category loss, the unique attribute semantics U is ensured to capture the core category label Y. Self-supervised optimization constraints: The weighted evidence lower bound (ELBO) loss function is used for optimization. This function combines the reconstruction loss (used to restore the original sample features X) and the KL divergence term adjusted based on the hyperparameter β (used to constrain the independence of the latent variable distribution), thereby achieving attribute decoupling in the semantic space and eliminating the interference of confounding factors.

2. Knowledge Refinement Module Combining Counterfactual Reasoning This module is responsible for using the decoupled attribute semantics to perform "imaginary" reasoning, and refining the model decision boundary by constructing counterfactual samples to achieve a balance between forgetting and retention performance. Counterfactual sample generation: Semantic attribution: Extracting shared attribute feature vectors from samples to be forgotten; Intervention operation: Randomly sampling unique attribute feature vectors from retained samples and performing intervention actions to combine the two; Prediction generation: Decoding the above combined vectors into counterfactual samples using generation, the samples accurately simulate the decision blind spot in the feature space that combines the "core semantics of the retained category" and the "shared background of the forgotten category"; Forgetting optimization based on reconstruction loss: Multiple retention constraints: By calculating the KL divergence of the predicted probability distribution of counterfactual samples by the original model and the model to be updated, the model is constrained to retain shared semantics unrelated to the forgetting target, preventing excessive forgetting; Contrastive forgetting mechanism: Constructing a contrastive learning loss, defining counterfactual samples and samples to be forgotten as positive sample pairs to narrow the feature distance, while defining them and retained samples as negative sample pairs to widen the distance, thereby enhancing the rigidity of the forgetting boundary; Balanced performance optimization index: Updating the gradient of the target model through weighted total loss, and adjusting the coefficients to achieve the optimal balance between fidelity and effectiveness after forgetting request.

Images