federated forgetting method based on category prototype alignment and semantic space calibration

Abstract

This invention proposes a federated forgetting method based on category prototype alignment and semantic space calibration. The method involves local forgetting of the target category on each client side and constructing category prototypes in the feature space to represent category semantic information. Then, on the server side, the model parameters and category prototypes uploaded by each client are aggregated, and a category prototype alignment mechanism is introduced to stably preserve the semantic structure of the categories. Simultaneously, a semantic exclusion strategy is used to weaken the semantic influence of the category to be forgotten. A two-stage semantic calibration mechanism is designed to adaptively adjust the forgetting intensity based on semantic uncertainty and prototype similarity, achieving efficient forgetting without accessing the original data. This method effectively reduces semantic interference during the forgetting process while ensuring data privacy, and improves the model's prediction stability and generalization performance on the remaining categories. Experimental results show that this invention has significant advantages in forgetting effectiveness, model accuracy, and resistance to privacy attacks.

Description

Technical Field

[0001] This invention relates to federated learning, passive domain adaptation, and category forgetting techniques, specifically a federated forgetting method based on category prototype alignment and semantic space calibration. Background Technology

[0002] In distributed learning environments, balancing data privacy and model performance is a key challenge. Federated learning (FL) allows multiple clients to collaboratively train a model without sharing data, thus protecting data privacy. However, this distributed training approach in federated learning also introduces problems such as model performance instability, data inconsistency, and privacy leaks. As model training progresses, some clients may need to delete data from specific categories; this process is called Federated Unlearning (FUL). Federated Unlearning requires deleting specific knowledge without complete retraining, while maintaining the stability of the global model and avoiding unnecessary shifts in the decision boundary due to the forgetting process. Existing federated forgetting methods typically rely on parameter-level tuning, which struggles to avoid shifts in the decision boundary, leading to semantic interference during the forgetting process and impacting model performance. Furthermore, existing federated forgetting methods often neglect the category prototypes and semantic structures of the data, failing to precisely control and manage category information in the global semantic space.

[0003] Balancing data privacy and model performance remains a long-standing challenge. While traditional privacy protection techniques (such as encryption and anonymization) can effectively protect sensitive data, they often excessively destroy valuable information, impacting subsequent model training. For example, in protecting medical data, excessive anonymization can lead to the loss of key medical features, preventing the trained model from extracting effective diagnostic information and ultimately affecting its practical application. Therefore, in federated learning environments, effectively deleting data of specified categories, ensuring data privacy, avoiding data perturbation, and maintaining model performance and semantic consistency are core issues that urgently need to be addressed. Summary of the Invention

[0004] To address issues such as decision boundary shift and semantic interference during model training in distributed learning environments, this invention discloses a federated forgetting method based on category prototype alignment and semantic space calibration. This method precisely controls the forgetting process of category information through innovative category prototype alignment and semantic space calibration techniques, avoiding model performance degradation. Simultaneously, combined with a federated learning architecture, it enables collaborative learning among different clients while protecting data privacy, ultimately improving model stability and accuracy. This invention employs the following technical solution, with steps including:

[0005] Step 1: Each client independently trains a local model using local data and calculates a category prototype for each class. The server aggregates the local models from each client to generate a global model and aggregates the category prototypes from each client to form a global category prototype. This process is repeated multiple times to obtain the optimal global model and global category prototype. Step 2: During model training, each client performs a category-level forgetting operation based on the global model and the global category prototype. Specifically, each client adjusts the category prototype according to the target category to be forgotten, suppressing the semantic influence of the category to be forgotten and maintaining the stability of the decision boundary of the retained category. Step 3: During category prototype alignment, the server introduces a semantic space calibration mechanism. By calculating the similarity and distance between category prototypes, the relative positions of different category prototypes are adjusted to ensure that the prototypes of the retained categories remain stable in the semantic space, avoiding semantic interference from forgotten categories. Simultaneously, exclusion constraints are applied to prototypes of forgotten categories to ensure that they do not affect the representation of the retained categories. Step 4: Based on the results of category prototype alignment and semantic space calibration, adjust the parameters of the global model, retrain the model, and record performance metrics such as loss and accuracy in each training round. Continuously optimize the model performance through multiple training rounds until it meets the preset performance standards. Step 5: If the model performance does not meet expectations or there is room for further optimization during model training, adjust the class prototype alignment and semantic calibration strategies based on the evaluation results, perform the class-level forgetting operation again, and continue to optimize the model until the model performance reaches a satisfactory level. Step 6: The client uses the optimal global model as the initial target model and generates initial pseudo-labels on the local target domain data. A teacher model is generated using a self-supervised learning method, and its pseudo-labels guide the learning of the student models. The server aggregates the student models from each client and broadcasts the updated global model back to the clients, completing one iteration. This process is repeated until training is complete.

[0006] Furthermore, step 1 specifically includes: Assume the total number of clients participating in federated learning is K, and the client ID is... Each client The local source domain dataset is ,in, For the first One input sample, This is the corresponding tag. For the client The total number of samples. Each client uses a pre-trained model (Vision Transformer) as the feature extraction backbone to initialize the local source domain model. The cross-entropy loss function is used for local model training. The loss function is defined as follows: (1) in, For local models of samples The probability distribution of category prediction Used to measure the difference between the model's classification predictions and the true labels. Optimized using the gradient descent algorithm. Each client updates local model parameters This yields the converged local model.

[0007] After the local model training is completed, the client Based on the sample features extracted by the model, the local category prototype (i.e., the cluster center of the sample features of that category) is calculated using the following formula: (2) This formula obtains the category semantic representation by minimizing the L2 distance between sample features and the prototype. The server receives local model parameters from all K clients. and local category prototype Perform global aggregation operations separately: (3) The weights are determined by the proportion of client sample sizes, ensuring that clients with larger datasets contribute more significantly to the global model. The global category prototype is obtained by weighted aggregation of the local category prototypes uploaded by each client, calculated using the following formula: (4) The local training and global aggregation processes are repeated for T iterations until the global model converges, yielding the optimal global model parameters. and the optimal global category prototype .

[0008] Furthermore, step 2 specifically includes: The server receives a federated forgetting request and determines the set of categories to be forgotten. And obtain the set of retained categories. Then, the set of categories to be forgotten Optimal global model and the optimal global category prototype Broadcast to all clients. Each client Based on global information, the local dataset is divided into reserved category datasets. and the dataset to be forgotten .

[0009] Client Based on global model Initialize the local forgetting model The goal of performing category-level forgetting training is to maximize the prediction uncertainty of the category to be forgotten while preserving the classification performance of the original categories. Its loss function is defined as: (5) Optimize using gradient descent algorithm Client updates forgotten model parameters And recalculate the local category prototype based on the updated model: for the retained categories Recalculated local prototype It will be anchored to the global prototype. Nearby, to stably preserve the semantic structure of the category. For categories to be forgotten. Recalculated local prototype It will be pushed away from the preserved category prototype, achieving initial semantic separation.

[0010] After training is complete, the client uploads the updated forgetting model parameters and the adjusted local category prototype to the server.

[0011] Furthermore, step 3 specifically includes: The server receives all updated model parameters and local category prototypes uploaded by clients. It first constructs two core interaction potential functions to characterize the semantic relationships between category prototypes: The repulsive interaction potential is used to measure the semantic similarity between the prototypes of the category to be forgotten and the category to be retained, achieving smooth semantic repulsion that decays with distance. The formula is as follows: (6) Where a and b are any two global category prototypes. As a scale hyperparameter (controlling the repulsive force decay rate), the closer a and b are, the more... The larger the value, the stronger the repulsive force.

[0012] Anchoring interaction potential is used to constrain and preserve the category prototype, stabilizing it near the initial global prototype and avoiding semantic drift. The formula is: (7) This function directly measures the L2 distance between two prototypes. The greater the distance, the greater the anchoring loss, which can effectively and stably preserve the semantic structure of the categories.

[0013] Based on this interaction potential function, the server constructs a Category Prototype Alignment (CPA) loss function, aiming to minimize the anchoring loss of the retained category prototypes while maximizing the repulsion loss between the category to be forgotten and the retained category prototypes. The formula is as follows: (8) The server minimizes Iteratively update the global category prototype and perform semantic space calibration: initialize the optimizer and set the initial global prototype to... Iteratively perform gradient updates until... When convergence or the preset number of iterations is reached, the update formula is: (9) After calibration, the global prototype of class r∈R is preserved. It will stabilize near the initial position, awaiting the forgetting of the global prototype of category u∈U. It maximizes the distance to all retained category prototypes, completely cuts off the semantic interference of the forgotten category to the retained category, and completes the structured calibration of the semantic space.

[0014] Furthermore, step 4 specifically includes: The server is based on a calibrated global category prototype. Forgotten model parameters uploaded by the client Perform a global model update, and use the federated averaging algorithm to aggregate the forgotten model parameters to obtain the updated global model parameters: (10) The server records performance metrics such as the loss value and class retention accuracy for each training round. If the model performance meets the preset standard, the iteration stops; otherwise, it returns to step 2 to continue performing class-level forgetting and semantic calibration operations.

[0015] Furthermore, step 5 specifically includes: If the model performance does not meet expectations or there is room for further optimization, the server adaptively adjusts the category prototype alignment and semantic calibration strategy based on the evaluation results: if the accuracy of the retained category decreases significantly, the weight of the anchoring interaction potential is increased to strengthen the stability of the retained category prototype; if the semantic residue of the category to be forgotten is severe, the scale parameter is increased. To enhance the repulsive force between the prototypes of the categories to be forgotten and those to be retained; if the model converges too slowly, adjust the balancing hyperparameters. and Optimize the weight allocation for forgetting training and the total loss. After adjustment, repeat steps 2-4 of the category-level forgetting and semantic calibration process until the model performance reaches satisfactory results.

[0016] Furthermore, step 6 specifically includes: The client uses the optimal global model As the initial target model, self-supervised iterative optimization is performed on the local target domain data: The client utilizes the optimal global model Unlabeled samples for the local target domain Perform reasoning to generate an initial set of pseudo-labels. The teacher model is obtained by pre-training on the target domain data using a self-supervised learning method. Used to generate stable pseudo-labels; initializes the optimal global model as the student model. It receives pseudo-label guidance from the teacher model.

[0017] Client The training loss for the student model is defined as the joint loss of knowledge distillation loss and feature similarity loss: (11) The server aggregates the student model parameters from all clients, updates the global model, and broadcasts it to the clients. This self-supervised iterative process is repeated until the model converges, completing the final domain adaptive optimization.

[0018] Through the above steps, this invention discloses a federated forgetting method based on category prototype alignment and semantic space calibration. This method, without relying on the original source domain data, introduces category-level semantic regulation and a federated forgetting mechanism, effectively enhancing the model's generalization ability and stability to target domain data in heterogeneous distributed scenarios. At each client, category prototype extraction and aggregation are first performed based on the global model and local source domain data. By extracting the feature cluster centers of each sample as category prototypes, high-dimensional semantic information is compressed into compact prototype vectors; the server-side weighted aggregation generates global category prototypes as semantic anchors; during local training, classification loss is optimized for retained categories while maximizing the prediction entropy of the categories to be forgotten, thus initially achieving category-level forgetting. The server-side introduces a category prototype alignment and semantic space calibration mechanism, using anchored interaction potentials to stably retain category prototypes and repulsive interaction potentials to separate the category prototypes to be forgotten, avoiding semantic interference and providing a reliable semantic reference for target domain adaptation. Then, pseudo-labels are generated using unlabeled data in the target domain. Specifically, the teacher model, initialized on a semantically calibrated global model, predicts target domain data. The prediction results are then used as training labels for the student models, which are trained using knowledge distillation. Simultaneously, a federated learning framework aggregates model parameter updates from each client. This approach protects the local data privacy of each client while collaboratively optimizing the global model through aggregating these updates, further improving the model's generalization performance and forgetting ability in heterogeneous target domain scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the federated forgetting method based on category prototype alignment and semantic space calibration of the present invention. Figure 2 This is a block diagram of the module structure of the federated forgetting method based on category prototype alignment and semantic space calibration of the present invention. Detailed Implementation To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention: This invention discloses a federated forgetting method based on category prototype alignment and semantic space calibration. It does not rely on original source domain data. First, category prototypes are extracted from local source domain data on each client. The server then weights and aggregates these prototypes to generate a global category prototype as a semantic anchor. During local training, classification loss is optimized for retained categories while maximizing the prediction entropy of the category to be forgotten, achieving initial category-level forgetting. Then, through the server-side category prototype alignment and semantic space calibration mechanism, category prototypes are stably retained and the category prototypes to be forgotten are separated, providing semantic reference for target domain adaptation. Subsequently, the teacher model is initialized based on the calibrated global model, and pseudo-labels are generated for unlabeled data in the target domain. Knowledge distillation guides student model training to adapt to the target domain. Finally, a federated learning framework is used to aggregate the model parameter updates from each client, protecting data privacy while collaboratively optimizing the global model. This method, based on category prototype regulation of the semantic space and combining pseudo-labels and knowledge distillation techniques, achieves stable federated forgetting and domain adaptation without requiring original source domain data, significantly enhancing the model's stability and generalization ability in heterogeneous target domains.

[0021] Figure 1 shows the flowchart of the method of this invention. This embodiment conducts experimental verification based on four public benchmark datasets: Office-Home, Office-31, PACS, and BreakHis. The Office-Home dataset is used as the core verification scenario. This dataset contains 15,500 images and 65 categories, divided into four visually significantly different domains: Art, Clipart, Product, and RealWorld. The experiment adopts a cross-domain setting of "three domains as the source and one domain as the target," which fits the passive domain and adapts to the actual application scenario. In terms of data allocation, client and server data are divided according to the federated heterogeneous distributed characteristics. 10% of the local data of each client is extracted as global verification data for the server. The source domain data is divided into training and test sets in a 9:1 ratio for training and performance evaluation of the local source domain models of each client. The target domain data is all unlabeled samples, which satisfies the passive domain adaptation constraint. This embodiment, based on the core mechanism of category prototype alignment and semantic space calibration, combines a federated learning framework and a self-supervised pseudo-label adaptation strategy to complete category-level federated forgetting and cross-domain adaptive training on unlabeled target domain data, achieving a dual improvement in model generalization performance and structural stability. The specific training steps are as follows: Step 1: Using the Office-Home dataset as the experimental basis, each source domain is distributed to different clients to build a federated learning distributed training architecture. Each client extracts 10% of its local source domain data as global validation data and uploads it to the server. The remaining 90% of the source domain data is divided into local training and test sets in a 9:1 ratio. The target domain data remains unlabeled throughout the process, which fits the requirements of the source domain-free adaptation scenario.

[0022] Step 2: Each client trains a local source domain model based on its local source domain training set, using Vision Transformer as the backbone network. After completion, the basic classification performance is verified on the local test set. The server collects the parameters of all client local models, aggregates them to generate a global source domain model, evaluates its performance on the global verification data, and selects and saves the global source domain model with the best classification accuracy.

[0023] Step 3: The server distributes the optimal global source domain model to all clients. Each client extracts the feature representation of each type of sample in its local source domain training set through the model and calculates the category prototype to form a local category prototype set. Each client uploads its local category prototype to the server. The server performs weighted aggregation on the same type of prototype to generate a global category prototype set, which serves as the core anchor point for semantic space regulation. At the same time, it determines the set of categories to be forgotten and the set of categories to be retained, and initiates the category-level federated forgetting process.

[0024] Step 4: Each client performs local federated forgetting training using the global category prototype as a semantic reference: For retained categories, the model's classification performance is maintained through classification loss optimization, anchoring the semantic consistency between the local retained category prototype and the global prototype; for forgotten categories, their semantic feature expression is weakened by maximizing prediction entropy, achieving category semantic suppression at the local level. After training, each client recalculates and updates its local category prototype, synchronizing the updated local model parameters and the local category prototype to the server.

[0025] Step 5: Based on the updated prototypes and model parameters uploaded by each client, the server performs category prototype alignment and semantic space calibration: It retains the global prototype of each category through anchored interaction potential constraints, stabilizing its semantic structure and decision boundaries to avoid semantic drift; it uses repulsive interaction potentials to separate the global prototype of the category to be forgotten from the global prototype of the retained category in the semantic space, eliminating semantic residues and interference from the category to be forgotten. After calibration, a semantically stable calibrated global model is generated and distributed to all clients as the initial model for target domain adaptation.

[0026] Step 6: Each client extracts features from the unlabeled target domain data based on the calibrated global model. Initial cluster centers are determined through feature normalization and cosine similarity measurement. The clustering results are iteratively updated, and high-quality pseudo-labels are generated. A stable teacher model is built based on these pseudo-labels. The student model is initialized using the calibrated global model. A knowledge distillation strategy is employed to transfer soft-label knowledge from the teacher model to the student model. The student model parameters are dynamically updated using an exponential moving average strategy to ensure robustness in feature learning. A total loss function, including prediction entropy loss and cross-entropy loss, is constructed by combining local target domain data and optimized pseudo-labels. The total loss is minimized using an optimizer, and the student model parameters are updated and uploaded to the server. The server performs federated averaging aggregation on the student model parameters uploaded by all clients, generating an updated global target model, which is then rebroadcast to all clients. The process of target domain pseudo-label optimization, teacher-student model distillation training, and parameter federated aggregation is repeated for ten rounds of federated adaptive training. The model's classification performance in the target domain is monitored throughout the process until the model converges and achieves optimal generalization.

[0027] In this experiment, the training parameters for federated learning were set as follows: A total of 10 federated training epochs were conducted, with each client completing 20 local training epochs within each epoch. During the source domain model training phase, the batch size was set to 32 to balance training computational efficiency and model convergence stability; the learning rate was set to 0.005, and the weight decay coefficient was 0.0001, effectively avoiding gradient explosion and vanishing problems while ensuring training efficiency. During the category prototype extraction and federated forgetting phase, the batch size was adjusted to 4, and the learning rate was reduced to 0.0005 to meet the training requirements for fine-tuning of the semantic space. During the target domain adaptive training phase, the batch size was restored to 32, the learning rate remained at 0.0005, and stochastic gradient descent (SGD) was selected as the core optimization algorithm. The momentum parameter was set to 0.9 to accelerate model parameter update convergence and reduce training oscillations. Simultaneously, a momentum decay of 0.001 was configured to gradually reduce the influence of momentum as training progresses, improving the stability of the model in the later stages of training. In the loss function, the weights of classification loss and semantic calibration loss are both set to 1, achieving a balanced optimization between preserving class performance and semantic space structured regulation, and ensuring the collaborative training effect of class prototype alignment and federated forgetting.

[0028] This invention uses accuracy as the core evaluation metric for model performance, and the formula for calculating accuracy is expressed as in equation (12): (12) in, Indicates the total number of categories. Indicates the first The number of samples in the class that are actually positive samples and correctly predicted by the model; Indicates the first The number of samples in the class that are actually negative samples but were correctly predicted by the model; Indicates the first The number of samples that are actually negative but are incorrectly predicted as positive by the model; Indicates the first The number of samples in a class that are actually positive but are incorrectly predicted as negative by the model.

[0029] Based on the above steps and parameter settings, a passive domain adaptive federated forgetting architecture based on category prototype alignment is constructed, such as... Figure 2 As shown in Table 1, the ablation results demonstrate the ablation study results of this invention on four public datasets: Office-31, Office-Home, PACS, and BreakHis. The data in the table clearly shows that the method of this invention significantly outperforms basic federated passive domain adaptation methods on various heterogeneous datasets, effectively achieving a dual improvement in both category-level knowledge removal and target domain generalization performance.

[0030] Table 1 Ablation experiment results on four major public datasets

[0031] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated forgetting method based on category prototype alignment and semantic space calibration, characterized in that include: S1: Each client trains the initial model on its local dataset to obtain a local model, extracts the feature representations of each category, and constructs the corresponding category prototypes; the server aggregates the model parameters uploaded by each client to obtain a global model, and synchronously aggregates the category prototypes of each client to form a global category prototype set. S2: The server receives a forgetting request for a specified category and sends the current global model and the category to be forgotten to each client; each client initializes its local model based on the global model and performs the category forgetting operation locally. By reducing the prediction confidence of the model on the category to be forgotten, the semantic information of the target category is suppressed. S3: During the local model update process, each client constructs a joint optimization objective function based on the retained category data and the category data to be forgotten. The retained category performs classification learning, while the category to be forgotten achieves semantic weakening by maximizing the prediction entropy, thereby achieving category-level forgetting without accessing the original data. S4: Each client recalculates the feature representation of each category based on the updated model, updates the corresponding category prototype, and uploads the local model parameters and category prototype to the server; the server performs weighted aggregation on the category prototypes uploaded by each client to obtain the global category prototype; S5: The server performs category prototype alignment and semantic space calibration operations based on global category prototypes. Constraints are imposed on retained category prototypes to maintain their semantic stability, and exclusion constraints are imposed on forgotten category prototypes to keep them away from the retained category semantic space. The update intensity is adaptively adjusted according to the category semantic sensitivity, thereby achieving stable semantic reconstruction. S6: The server distributes the calibrated global model and category prototypes to each client, and enters the next round of federated iterative training until the preset number of training rounds is reached or the model performance converges, finally obtaining a global model that has completed category forgetting and has stable performance.

2. The federated forgetting method based on category prototype alignment and semantic space calibration according to claim 1, characterized in that: When performing the category forgetting operation on the client side, during local training, the classification loss is used to optimize the retained category samples, while the semantic information of the unforgotten category samples is weakened by maximizing the prediction entropy. The optimization objective function is expressed as: （1） in, Represents the classification loss function. To predict the entropy function, For sensitivity, This indicates that the category dataset is preserved. For the dataset of categories to be forgotten.

3. The federated forgetting method based on category prototype alignment and semantic space calibration according to claim 1, characterized in that: Clustering or mean calculation is performed on the feature representations of each category to obtain the prototype representation of that category in the feature space. The calculation method is as follows: （2） in, Indicates the client Medium category The prototype, This represents the feature representation of the sample.

4. The federated forgetting method based on category prototype alignment and semantic space calibration according to claim 1, characterized in that: When the server aggregates category prototypes, it uses a weighted least squares approach to calculate the global category prototype, which is expressed as follows: （3） in, Indicates the client In category Weight on, Indicates the total number of clients. Represents the variable to be optimized. Used to indicate category global prototype candidate values, This represents the L2 norm, used to measure the Euclidean distance between vectors.

5. The federated forgetting method based on category prototype alignment and semantic space calibration according to claim 1, characterized in that: In the process of category prototype alignment and semantic space calibration, a semantic action function is introduced to describe the relationship between different category prototypes, including attraction constraints and repulsion constraints, which are defined as follows: （4） （5） in, Used to describe semantic exclusion relationships between categories To maintain the stability of category prototypes, during the semantic space calibration process, stability constraints are applied to the retained category prototypes, and distance constraints are applied to the forgotten category prototypes to move them away from the retained category semantic space, thereby achieving category semantic decoupling.

6. The federated forgetting method based on category prototype alignment and semantic space calibration according to claim 1, characterized in that: Semantic space calibration also includes a semantically sensitive adaptive adjustment mechanism, which determines the update intensity of different categories by calculating the curvature information at the category prototype, expressed as: （6） in, Indicates category Semantic sensitivity is used to measure how sensitive the category is to changes in the current model. The trace operation of a matrix is ​​used to sum the diagonal elements of the matrix to characterize the overall curvature, and the updated intensity is normalized based on the curvature: （7） in, Indicates category The corresponding update weights, This indicates the preset maximum update weight. Indicates category The curvature value, This represents the largest curvature value among all categories.

Images