Class-level federated forgetting method and system based on double-layer expert mixing and tf-idf discriminative pruning

Abstract

The application provides a class-level federated forgetting method and system based on double-layer expert mixing and TF-IDF discriminant pruning, which comprises the following steps: distributing an initial or updated global feature extractor to each client participating in federated learning training; receiving local feature extractors uploaded by each client; using an outer expert mixing model to aggregate all received local feature extractors, specifically comprising: when a forgetting request containing a target class set is received from a client, calculating the channel-level discriminant score of each local feature extractor in each target class, pruning the channels of each local feature extractor whose channel-level discriminant score exceeds a set threshold, and weighting and aggregating all pruned local feature extractors to obtain an updated global feature extractor; when no forgetting request is received, weighting and aggregating all received local feature extractors according to a fixed gating strategy.

Description

Technical Field

[0001] This invention relates to the field of federated forgetting technology, and in particular to a class-level federated forgetting method and system based on two-layer expert hybridization and TF-IDF discriminative pruning. Background Technology

[0002] With the widespread application of artificial intelligence, Federated Learning (FL) has attracted widespread attention due to its ability to achieve joint modeling across multiple terminals without sharing users' original data. However, with the increasing demand for privacy protection and distributed intelligent processing, and the promulgation of privacy protection laws and regulations such as the General Data Protection Regulation (GDPR), Federated Learning is widely used in scenarios such as personalized video recommendation, image retrieval, and voice interaction. How to realize the "right to be forgotten" has become an urgent problem to be solved. Traditional Federated Learning builds a global model without sharing original data by training locally on the client and aggregating parameters on the server. However, under privacy laws, how to make the model "forgettable" for specific user data has become a current technical challenge. Therefore, Federated Unlearning (FU) has emerged as a key technology for effectively removing the influence of sensitive user information from a joint model, and its core objective is to selectively remove the influence of target data on the model without retraining the entire model from scratch.

[0003] In a typical federated learning framework, clients train local models and upload updates, which are then aggregated by the server to form a global model. In the federated learning (FU) task, after a client initiates a forget request, the influence of specified data on model behavior needs to be removed from the global model. Existing methods mainly include full retraining, differential privacy interference, model pruning and masking, and local tuning to achieve forgetting of specific data or categories. However, in real-world federated learning scenarios, clients may hold different types of data (such as images, text, and speech), and their neural network structures vary, making it difficult for the aforementioned methods to adapt to heterogeneous model architectures. Furthermore, the inconsistent sensitivity of different modalities to forgetting operations further exacerbates problems such as model performance degradation and increased training overhead.

[0004] To address the federated forgetting problem under heterogeneous conditions, federated forgetting techniques initially emerged with the "full retraining" approach. This involved retraining the model directly on the remaining data after a client initiated a deletion request, ensuring the contribution of the deleted sample was completely erased. However, this method had extremely high computational and communication costs, making it unsuitable for large-scale scenarios. To reduce these costs, researchers introduced "differential privacy noise injection," adding noise during model updates to mitigate the impact of individual data points. However, this method amplifies accuracy loss in strongly non-IID scenarios. Subsequent "local backtracking" or "partial retraining" strategies only revised the historical gradients and local weights related to the deletion request, significantly reducing global computation while maintaining the deletion effect.

[0005] Existing federated forgetting techniques propose a "full retraining" approach: after removing the target data, the model is completely retrained from scratch to achieve the most thorough forgetting effect. This approach is theoretically the most complete, but its extremely high training cost makes it difficult to directly apply to large-scale, real-time scenarios. Another approach uses differential privacy or noise perturbation to eliminate the contribution of specific data to the model, approximating forgetting by injecting random noise into updates; however, under non-IID data distributions, such noise often amplifies performance loss, leading to a significant decline in model accuracy. To balance efficiency and accuracy, researchers have proposed local retraining strategies, adjusting only affected historical gradients or local parameters rather than retraining the entire model. Some subsequent works have further broadened the applicability of these methods by combining longitudinal data partitioning, asynchronous training, and client cluster undoing mechanisms. Additionally, pruning techniques assess the importance of channels or neurons and then mask or delete unnecessary parts to achieve fine-grained forgetting. Some solutions also combine this technique with dynamic client selection or knowledge distillation to improve efficiency and environmental adaptability. While the aforementioned methods each have their advantages, they still face challenges such as high resource consumption, uncontrollable performance, and a lack of verification mechanisms in environments with non-independent identically distributed (Non-IID) data and heterogeneous model structures. Furthermore, existing heterogeneous federated learning techniques face additional challenges when transferred to federated forgetting tasks. For example, maintaining consistency among clients after dynamic pruning is difficult, collaboration conflicts occur frequently during data deletion, and communication load increases significantly.

[0006] Therefore, there is an urgent need to design new federated forgetting methods and frameworks to better address the problems existing in the current federated forgetting technologies and improve overall efficiency and usability. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a class-level federated forgetting method and system based on two-layer expert hybridization and TF-IDF discriminative pruning, in order to eliminate or improve one or more defects existing in the prior art.

[0008] One aspect of the present invention provides a class-level federated forgetting method based on two-layer expert mixture and TF-IDF discriminative pruning. In each communication round, the method includes: distributing an initial or updated global feature extractor to each client participating in federated learning training; wherein each client aggregates the global feature extractor and its locally stored local feature extractors using an inner-layer expert mixture model; receiving local feature extractors uploaded by each client; and aggregating all received local feature extractors using an outer-layer expert mixture model, specifically including: upon receiving a forgetting request from a client containing a target class set, calculating the value of each local feature extractor in the target class set. The system combines the channel-level TF-IDF distributions of each target category, calculates the channel-level discriminant score of each local feature extractor for each target category based on the channel-level TF-IDF distributions, prunes channels in each local feature extractor whose channel-level discriminant scores exceed a set threshold, determines the weighting coefficients of each local feature extractor based on the channel-level discriminant scores, and aggregates all pruned local feature extractors according to the weighting coefficients. The weighted aggregation results are then aligned to obtain an updated global feature extractor. If no forget request is received, all received local feature extractors are weighted and aggregated according to a fixed gating strategy.

[0009] In some embodiments of the present invention, the inner-layer expert hybrid model includes a global feature extractor, a local feature extractor, a first gating network, and a prediction head module; each client uses the inner-layer expert hybrid model to aggregate the global feature extractor and its locally stored local feature extractor, including: in the first round of communication, the first gating network fuses the global feature extractor and the local feature extractor according to a preset ratio; starting from the second round, the first gating network determines the channel-level discriminant score based on the channel-level TF-IDF distribution, dynamically adjusts the weighting coefficients of the global feature extractor and the local feature extractor using the channel-level discriminant score, and aggregates the global feature extractor and the local feature extractor according to the weighting coefficients.

[0010] In some embodiments of the present invention, the method further includes a federated forgetting operation verification step, specifically including: after completing each round of federated forgetting, the server records the set of removed channels in the global feature extractor and saves the TF-IDF distribution snapshot for each round; by comparing the output probability of the global feature extractor in the target category before and after pruning, the degree of prediction failure is evaluated to directly verify the effect of the federated forgetting process.

[0011] In some embodiments of the present invention, the method further includes: performing stability testing on federated learning training without accessing the original data, indirectly confirming whether the influence of deleted categories has been removed from the model structure.

[0012] In some embodiments of the present invention, when the target category set contains multiple target categories, the pruning process includes: calculating the channel-level TF-IDF distribution and channel-level discriminant score for each target category separately according to the multi-category sequential processing mechanism, and then forming a column-independent mask for each channel separately, generating a channel deletion set through intersection operation, and pruning each local feature extractor according to the channel deletion set.

[0013] Corresponding to the above method, the present invention also provides a class-level federated forgetting method based on two-layer expert hybridization and TF-IDF discriminative pruning. In each communication round, the method includes: receiving an initial or updated global feature extractor from the server; aggregating the global feature extractor and its locally stored local feature extractors using an inner-layer expert hybridization model, specifically including: performing the same structural pruning on the local feature extractors according to the pruning process of the global feature extractor to maintain consistent federated forgetting depth between the client and server; calculating the channel-level TF-IDF distribution of each local feature extractor in each target category, calculating the channel-level discriminative score of each local feature extractor in each target category based on the channel-level TF-IDF distribution, determining the weighting coefficients of the global feature extractor and the pruned local feature extractors based on the channel-level discriminative scores, and locally weighting and aggregating the global feature extractor and the pruned local feature extractors according to the weighting coefficients; and uploading the locally stored local feature extractors to the server after the federated learning round ends.

[0014] In some embodiments of the present invention, the method further includes: fine-tuning the global feature extractor based on the local feature extractor after structural pruning, according to an annealed learning rate strategy.

[0015] In some embodiments of the present invention, the method further includes performing local precise forgetting and / or local forgetting recovery on the client side, specifically including: local precise forgetting, using the calculated channel-level TF-IDF distribution and channel-level discriminant score to mask channels in the local feature extractor whose channel-level discriminant scores exceed a set threshold; local forgetting recovery, retaining each channel deleted by the local feature extractor, and when a local forgetting recovery request is received, recovering the target channel included in the local forgetting recovery request.

[0016] Corresponding to the above methods, the present invention also provides a class-level federated forgetting system based on two-layer expert hybrid and TF-IDF discriminative pruning, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0017] In accordance with the above methods, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0018] The class-level federated forgetting method proposed in this invention, based on a two-layer expert hybrid approach and TF-IDF discriminative pruning, enables clients with different modalities and model structures to perform personalized and controllable class-level forgetting operations in federated learning. This avoids full model retraining, significantly reduces the cost of forgetting, and maximizes the retention of reasoning ability for unforgotten categories.

[0019] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0020] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart of a class-level federated forgetting method in some embodiments of the present invention.

[0022] Figure 2 This is a flowchart of a class-level federated forgetting method in some other embodiments of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0024] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0027] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0028] While existing federated forgetting techniques have proposed various improvements in terms of computational overhead, communication costs, and deletion granularity to address complex real-world scenarios involving heterogeneous clients and strongly non-IID data, compatibility challenges remain due to heterogeneous model structures. For example, most forgetting methods assume homogeneous client models. When terminals use differentiated networks, traditional local backtracking or pruning strategies struggle to ensure that deletion depth is synchronized with global performance, potentially leading to a significant drop in accuracy for undeleted categories. Furthermore, when the same data segment contributes differently to different clients, deletion requests on one end may disrupt the performance of another. Existing work lacks efficient mechanisms to coordinate such cross-end conflicts and maintain aggregation fairness. Moreover, although pruning and local retraining perform well in experimental environments, real-world deployments still require frequent synchronization of numerous gradients or sub-model updates, causing communication volume to increase dramatically with the number of nodes and model size, which edge devices struggle to handle. Additionally, existing evaluations often focus on accuracy or inference error, lacking quantitative metrics for "whether deletion is thorough," making it difficult to meet compliance requirements such as GDPR for traceable and verifiable forgetting.

[0029] To overcome the problems of existing federated forgetting techniques, this invention proposes a class-level federated forgetting method and system based on two-layer expert hybridization and TF-IDF discriminative pruning. This method is compatible with heterogeneous federated environments and can achieve accurate, efficient and verifiable data deletion while maintaining the performance of the model's main task.

[0030] To further reduce the overhead of federated forgetting, pruning methods can be employed. Linear filtering and neuron masking can deactivate parameters strongly correlated with the target data without altering the overall network structure, thus enabling rapid forgetting. Building upon this, researchers have proposed "class-based pruning," which calculates the contribution of each channel to a specific class and prunes high-contribution channels to achieve more refined class-level forgetting under non-independent and identically distributed (Non-IID) conditions, while reducing the resource consumption caused by repeated retraining.

[0031] To extend application scenarios from homogeneous models to heterogeneous client environments, personalized and heterogeneous federated learning methods can be introduced into the field of federated forgetting (FU). For example, mutual learning uses low-rank adapters to alternately train local and shared models to adapt to different data distributions on each end; prototype alignment uses class prototypes to calibrate the global model, mitigating aggregation drift caused by non-independent and identically distributed models; distillation and ensemble fusion significantly reduce the communication cost of synchronizing complete weights across multiple ends through knowledge transfer or sub-model fusion. In terms of parameter compression, pruning and sub-model training have been proven to significantly reduce model size and are easily integrated with forgetting processes. In addition, mechanisms such as asynchronous forgetting of vertically segmented models, reversible training of cross-domain random forests, and isolated coding sharding also provide feasible ideas for deploying FU in vertical data and large-scale node scenarios.

[0032] This invention proposes a class-level federated forgetting method applicable to heterogeneous model structures and non-independent and identically distributed (Non-IID) data environments. The method employs a two-layer Mixture-of-Experts (MoE) structure. This application divides federated training and federated forgetting into four key stages based on communication rounds: (1) client selection and model distribution; (2) client-side local training and feature uploading; (3) server aggregation and global forgetting operation; and (4) client-side local fine-tuning and forgetting recovery after receiving updates. This four-stage process is executed cyclically within each communication round until all training and forgetting tasks are completed.

[0033] In terms of system architecture, this invention introduces a two-layer Mixture of Experts (MoE) structure, deployed on both the client and server sides. On the client side, an inner MoE model is constructed, consisting of a global feature extractor, a local feature extractor, a first gating network, and a prediction head module. The global feature extractor is provided by the server to maintain semantic consistency with other clients; the local feature extractor is trained by the client using its own private data, taking into account personalized tasks; the first gating network coordinates the fusion weights between the two; and the prediction head module performs the final classification prediction. On the server side, an outer MoE model is constructed, using the local feature extractors uploaded by all clients as "client experts" and the global feature extractor maintained by the server itself as "global experts." A centralized gating mechanism is used to achieve expert selection and aggregation updates. On the client side, a local feature expert and personalized gating network are constructed. The TF-IDF mechanism is used to evaluate and select feature channels, achieving accurate identification and forgetting of local sensitive categories. On the server side, feature extractors uploaded from various clients are aggregated to form a global expert set. A global gating mechanism is used to perform aggregate analysis and channel pruning on the impact of specific categories, thus achieving unified forgetting processing across clients and modalities. Furthermore, to reduce communication burden and protect user privacy, this invention can transmit only lightweight feature extractors during federated training and forgetting, rather than the complete model parameters, thus balancing efficiency and privacy. Through the above design, this invention achieves efficient, accurate, and personalized forgetting of specific category information while ensuring model performance stability. It is particularly suitable for complex federated learning environments under non-IID conditions, and has good practical application value and prospects for widespread application.

[0034] Figure 1 The flowchart below illustrates a class-level federated forgetting method in some embodiments of the present invention. The execution entities are the server and / or client, with global forgetting performed on the server side. Within each communication round, the method includes: Step S110: Distribute the initial or updated global feature extractor to each client participating in federated learning training; wherein, each client uses an inner-layer expert hybrid model to aggregate the global feature extractor and its locally stored local feature extractor.

[0035] Step S120: Receive the local feature extractors uploaded by each client.

[0036] Step S130: Aggregate all received local feature extractors using an outer-layer expert hybrid model. Specifically, this includes: Step S131: When a forget request containing a set of target categories is received from the client, calculate the channel-level TF-IDF distribution of each local feature extractor on each target category included in the target category set. Based on the channel-level TF-IDF distribution, calculate the channel-level discriminant score of each local feature extractor on each target category. Prune the channels of each local feature extractor whose channel-level discriminant scores exceed a set threshold. Determine the weighting coefficient of each local feature extractor based on the channel-level discriminant scores. Aggregate all pruned local feature extractors according to the weighting coefficients. Align the weighted aggregation results to obtain the updated global feature extractor.

[0037] Step S132: When no forget request is received, perform weighted aggregation on all received local feature extractors according to a fixed gating strategy.

[0038] This application specifically designs a two-layer expert hybrid structure, constructing an inner expert hybrid model (InnerMoE) on the client side and an outer expert hybrid model on the server side. The inner expert hybrid model includes a global feature extractor, a local feature extractor, a first gating network, and a prediction head module. The outer expert hybrid model uses the local feature extractor uploaded by the client as the client expert and the global feature extractor maintained by the server as the global expert, selecting and aggregating updates between the client expert and the global expert. The global feature extractor is provided and maintained by the server, the local feature extractor is trained by the client using its own private data, the first gating network coordinates the fusion weights between the global and local feature extractors, and the prediction head module performs the final classification prediction.

[0039] The class-level federated forgetting method proposed in this invention, based on a two-layer expert hybrid approach and TF-IDF discriminative pruning, enables clients with different modalities and model structures to perform personalized and controllable class-level forgetting operations in federated learning. This avoids full model retraining, significantly reduces the cost of forgetting, and maximizes the retention of reasoning ability for unforgotten categories.

[0040] In some embodiments of the present invention, the inner-layer expert hybrid model includes a global feature extractor, a local feature extractor, a first gating network, and a prediction head module. Accordingly, in this embodiment, each client utilizes the inner-layer expert hybrid model to aggregate the global feature extractor and its locally stored local feature extractor, including: in the first round of communication, the first gating network fuses the global feature extractor and the local feature extractor according to a preset ratio; from the second round onwards, the first gating network determines the channel-level discriminant score based on the channel-level TF-IDF distribution, dynamically adjusts the weighting coefficients of the global feature extractor and the local feature extractor using the channel-level discriminant score, and aggregates the global feature extractor and the local feature extractor according to the weighting coefficients.

[0041] By adopting this embodiment of the invention, the problems of existing federated forgetting methods in heterogeneous environments, such as coarse forgetting granularity, large model performance loss, high computational and communication overhead, and lack of verifiable mechanisms, can be solved.

[0042] In some embodiments of the present invention, the method further includes a federated forgetting operation verification step, specifically including: after completing each round of federated forgetting, the server records the set of removed channels in the global feature extractor and saves the TF-IDF distribution snapshot for each round; by comparing the output probability of the global feature extractor in the target category before and after pruning, the degree of prediction failure is evaluated to directly verify the effect of the federated forgetting process.

[0043] The embodiments of this invention are compatible with heterogeneous federated environments and can achieve accurate, efficient and verifiable data deletion while maintaining the performance of the model's main task.

[0044] Furthermore, in some embodiments of the present invention, the method further includes: performing stability testing on federated learning training without accessing the original data, indirectly confirming whether the influence of deleted categories has been removed from the model structure.

[0045] In some embodiments of the present invention, when the target category set contains multiple target categories, the pruning process includes: calculating the channel-level TF-IDF distribution and channel-level discriminant score for each target category separately according to the multi-category sequential processing mechanism, forming a column-independent mask for each channel separately, generating a channel deletion set through intersection operation, and pruning each local feature extractor according to the channel deletion set.

[0046] By employing this embodiment of the invention, excessive destruction of the feature space due to overlapping target categories can be avoided.

[0047] On the other hand, this invention takes each client as the execution subject and proposes a class-level federated forgetting method based on two-layer expert hybrid and TF-IDF discriminative pruning. Figure 2 The flowchart of the class-level federated forgetting method in some other embodiments of the present invention is shown. Within each communication round, the method includes: Step S210: Receive the initial or updated global feature extractor sent by the server.

[0048] Step S220: Aggregate the global feature extractor and its locally stored local feature extractors using an inner-layer expert hybrid model. Specifically, this includes: performing the same structural pruning on the local feature extractors based on the pruning process of the global feature extractor to maintain consistency in the federated forgetting depth between the client and server; calculating the channel-level TF-IDF distribution of each local feature extractor in each target category; calculating the channel-level discriminant score of each local feature extractor in each target category based on the channel-level TF-IDF distribution; determining the weighting coefficients of the global feature extractor and the pruned local feature extractors based on the channel-level discriminant scores; and locally weighting and aggregating the global feature extractor and the pruned local feature extractors according to the weighting coefficients.

[0049] Step S230: After the federated learning round is completed, upload the locally stored local feature extractor to the server.

[0050] The class-level federated forgetting method proposed in this invention, based on a two-layer expert hybrid approach and TF-IDF discriminative pruning, enables clients with different modalities and model structures to perform personalized and controllable class-level forgetting operations in federated learning. This avoids full model retraining, significantly reduces the cost of forgetting, and maximizes the retention of reasoning ability for unforgotten categories.

[0051] In some embodiments of the present invention, the method further includes: fine-tuning the global feature extractor based on the local feature extractor after structural pruning, according to an annealed learning rate strategy.

[0052] By adopting this embodiment of the invention, structural abrupt changes caused by pruning can be avoided, thus preventing the model performance from collapsing. At the same time, it can promote the model to gradually adapt to the residual feature space after structural changes.

[0053] In some embodiments of the present invention, the method further includes performing local precise forgetting and / or local forgetting recovery on the client side, specifically including: local precise forgetting, using the calculated channel-level TF-IDF distribution and channel-level discriminant score to mask channels in the local feature extractor whose channel-level discriminant scores exceed a set threshold; local forgetting recovery, retaining each channel deleted by the local feature extractor, and when a local forgetting recovery request is received, recovering the target channel included in the local forgetting recovery request.

[0054] In a specific embodiment of the present invention, the specific process of federated training and federated forgetting in each communication round is as follows: (1) Client selection and model distribution: At the beginning of each round of communication, the server randomly selects K clients from N clients to participate in training.

[0055] (2) Local training and feature upload on the client: The selected client will receive the global feature extractor for the current round. The feature extractor is loaded into the local Inner MoE structure by the client and trained in conjunction with its local model to learn the fused feature representation. After the local training phase is complete, the client uploads the aligned local feature extractor. Instead of the complete model or prediction header parameters, this reduces the communication burden and the risk of privacy leaks.

[0056] (3) Server aggregation and global forgetting operation: After receiving all the feature extractors uploaded by the clients, the server uses them as an "expert list" and inputs them into Outer MoE to compare with the existing global experts. All participants participate in the aggregation. If a category-level forget request U is triggered in this round, the server will perform class-based pruning and update its global expert model accordingly; if no forget is triggered, the gating mechanism and expert selection strategy from the previous round will be used. After the update, the server will resend the aggregation results to the client.

[0057] (4) After receiving the update, the client completes local fine-tuning and forgetting recovery: The client receives the updated aggregation result from the server. In this application, the aggregation result is updated by updating the global feature extractor contained in the client's inner expert hybrid model. The client then uses the updated global feature extractor to perform gating updates and completes the local fine-tuning process.

[0058] The specific processes described above for federated training and federated forgetting cover both conventional federated training and class-level forgetting scenarios. In normal training rounds, the system focuses on rapid aggregation to improve generalization ability; in forgetting-triggered rounds, the system focuses on eliminating the influence of specified classes, combining a first gating network and pruning mechanisms to "erasure" the target class while maintaining performance stability for non-target classes. By iteratively executing the above four-stage process, this invention can achieve collaborative optimization of training and forgetting across multiple communication rounds and supports flexible configuration of training cycles, forgetting-triggered rounds, and class selection strategies.

[0059] In other embodiments of the present invention, in order to achieve efficient personalized feature learning and class-level forgetting, an Inner Mixture-of-Experts (Inner MoE) model is introduced on the client side. This model consists of two feature extractors, a first gating network, and a prediction head module, which are used to extract shared semantic information, learn local feature biases, dynamically allocate expert weights, and complete the final classification prediction, respectively.

[0060] Specifically, Inner MoE comprises the following components: (1) The Global Expert is initialized uniformly by the server and broadcast to the selected clients in each round of communication. Its structure and parameters are consistent, and it is used to capture feature information that is common to all clients. During the forward propagation of the model, the Global Expert can be used to compensate for the missing semantic expressions in the local model, thereby improving the prediction stability of small sample categories or rare modalities.

[0061] (2) The local expert is trained autonomously by the client based on a private dataset, reflecting the personalized features of the current terminal in terms of data distribution, input modality, and category distribution. This expert is relatively independent of the global expert structure, allowing the client to flexibly configure it according to different network architectures (such as CNN, ResNet, BiLSTM, etc.) to adapt to heterogeneous computing needs.

[0062] (3) The first gating network (Personalized Gating Network) is responsible for dynamically allocating weights between global experts and local experts. This first gating network calculates the fusion ratio based on the response of the input data to the two groups of experts (i.e., activation features) and their historical performance (such as category accuracy, feature entropy, etc.). In the first round of communication, the gating mechanism uses a preset ratio (such as 0.5 / 0.5) for fusion; from the second round onwards, the gating weights will be dynamically adjusted according to the TF-IDF distribution to emphasize the experts who perform better on the target category. The gating output is the weighted sum of the outputs of the two feature extractors, which constitutes the final intermediate representation.

[0063] (4) The prediction head module receives the feature representation after gating output and performs the final classification prediction. The structure of this module can be configured according to the specific task (such as multilayer perceptron, softmax classifier, etc.) and is trained end-to-end locally with cross-entropy loss as the target. In order to improve the robustness of the model, a regularization term is added to the local training process to maintain the consistency between the local expert output and the global features, thereby reducing the distribution drift during the aggregation process.

[0064] In some embodiments of this invention, a channel weight evaluation method based on the TF-IDF index is designed to assist the gating mechanism in determining the importance of each channel and its relevance to the target category in class-level forgetting scenarios. This index combines the channel's activation intensity (Term Frequency, TF) in a specific category with its inverse document frequency (IDF) across all categories, and its basic form is shown below: in, This represents the activation value of the j-th channel of client k in category u, where |U| is the total number of categories. A higher TF-IDF score indicates that the channel has a higher discriminative ability in the target category.

[0065] The first gating network described above uses channel-level scores to assign weights to different experts, and also provides a basis for subsequent channel pruning and forgetting mechanisms. During training, the client calculates the corresponding channel-level TF-IDF distribution for all samples, and uses this result to adjust the model structure or mask highly relevant channels to achieve accurate forgetting at the class level.

[0066] This architecture is highly scalable, supporting seamless integration between different modalities (images, text, speech, etc.) and heterogeneous model architectures. Through dynamic gating mechanisms and discriminative channel scoring, it can effectively improve personalized performance and target category recognition capabilities.

[0067] In some embodiments of this invention, to achieve unified and coordinated class-level forgetting operations under a multi-client heterogeneous model structure, an Outer Mixture-of-Experts (Outer MoE) model is introduced on the server side. This model uses local feature extractors uploaded by clients as its core and dynamically selects and weights each "client expert" through a centralized gating network to form a global expert representation with stable performance. This mechanism not only supports cross-client information fusion but also provides a unified execution interface for target channel removal operations in class-level forgetting.

[0068] In each round of communication, the server receives local feature extractors uploaded by each client. These feature extractors are considered "client-side experts" and are uniformly input into the Outer MoE model, along with the global feature extractors currently held by the server. Together, they form an expert pool. When no forget request is triggered, Outer MoE uses a fixed gating strategy to perform weighted aggregation of client experts to update the new global expert parameters, which are the updated global feature extractor. Once a category forget request U is received, Outer MoE will activate a series of pruning and re-aggregation mechanisms to remove the influence of that category in the global feature extractor.

[0069] Specifically, when a forget request is triggered, the server-side aggregation and global forgetting process is as follows: First, Outer MoE calculates the channel-level TF-IDF distribution of all client-uploaded feature extractors for the target category. This process reuses the client-side calculation method to obtain the discriminant score for each channel in the target category. Subsequently, the server removes channels from each client expert with scores higher than a set threshold R, thus forming the pruned client-side feature extractor \hat{\theta}_{e}^{k}.

[0070] Next, the server uses the TF-IDF scores as weighting coefficients to perform weighted aggregation (fusion) on all client experts, generating a new global feature extractor (in expert mixture models, the global feature extractor is also called the global expert representation). The weighted aggregation process is expressed as follows: in, Indicates client The average score of all channels under the target category is used as an indicator of expert importance. This aggregation method effectively suppresses the residual influence of highly relevant feature channels on the target category, while preserving the generalization ability to non-target categories.

[0071] After obtaining the aggregation results, the server will also generate a new global feature extractor. The input batch normalization module performs alignment processing to mitigate the distribution shift problem caused by differences in model structure. The mathematical expression for this step is: Ensure that the output distribution remains consistent across all clients. The server then broadcasts the updated global feature extractor and gating policy configuration to all online clients, enabling them to perform the next round of local training or forgetting.

[0072] Of particular note is that, in cases of multi-category forgetting, this invention employs a "category-by-category pruning and sequential aggregation" mechanism. This involves processing each category to be deleted sequentially, performing channel filtering and expert aggregation at each stage, and ensuring that each category of information is independently removed without interference during the final fusion. This design effectively solves the channel overlap problem under multi-target interference, significantly improving the coverage and accuracy of the deletion operation.

[0073] Through the above mechanisms, Outer MoE not only has the ability to handle heterogeneous client structures and information inconsistencies, but also can perform class-level impact removal in a lightweight manner, providing a unified and stable forgetting entry point for the entire federated system.

[0074] To achieve effective deletion of target categories without sacrificing overall model performance, this invention designs a complete class-level federated forgetting process, including a triggering mechanism, local pruning strategy, local fine-tuning process, and subsequent verification and compensation steps. This process is nested within a standard federated communication wheel and starts immediately when conditions are met, enabling the server to coordinate with other clients to complete rapid forgetting synchronization simultaneously with client requests.

[0075] In some specific embodiments of the present invention, the process of server-side and client-side jointly performing federated learning and federated forgetting is as follows: When a client is in a communication round Initiate a forget request, specifying the set of target categories you wish to remove from the global model. Afterward, the server enters the "Unlearning Phase." At this time, the server freezes the current global expert parameters, pauses the regular aggregation process, and instead initiates a specific channel filtering and global update procedure, which includes the following steps: First, the server retrieves the latest version of the feature extractor from historical uploads and calculates the performance of all experts in the target category. The channel-level TF-IDF distribution is used to identify the set of channels in the model that have the strongest discriminative power against the category to be forgotten. This is used to establish a global channel mask for subsequent pruning. The channel importance metric here still uses the following formula: For each client expert, when a certain channel If a channel's TF-IDF score exceeds a globally set threshold R in any target category, that channel is considered to need to be removed, and a structural culling operation is performed. The server processes each client's uploaded model independently and inputs the pruned model into a new global expert aggregation: in, This indicates a copy of the client-side feature extractor that has completed channel removal.

[0076] Subsequently, the server will generate the new global expert The update is distributed to all online clients. Upon receiving the update, each client synchronously performs the following operations: First, it performs the same structural pruning on the local expert again based on the global channel mask to ensure the forgetting depth remains consistent with the server's; second, it retains the unpruned local channels and performs a few rounds of fine-tuning using the remaining local class data. This fine-tuning follows an annealing learning rate strategy: , in, To fine-tune the number of rounds, The initial learning rate is used. This adjustment strategy avoids structural abrupt changes caused by pruning, preventing model performance collapse, and promotes the model's gradual adaptation to the residual feature space after structural changes.

[0077] Furthermore, in some embodiments, to ensure the verifiability of the forgetting operation, the present invention retains the following verification path: First, after completing each round of federated forgetting, the server records the global channel removal index set and simultaneously saves a snapshot of the TF-IDF distribution for each round; second, by comparing the output probabilities of the global feature extractor in the target category before and after pruning, the degree of prediction failure is evaluated to directly confirm the effect of the federated forgetting execution; third, without accessing the original data, training stability detection (e.g., changes in prediction variance, class mean drift, etc.) is performed based on the (retrained) retained class data to indirectly confirm whether the influence of the deleted categories has been removed from the model structure.

[0078] In some embodiments of the present invention, in order to alleviate the channel redundancy deletion problem that may be caused by multi-class overlapping pruning, the present invention supports a multi-class sequential processing mechanism, that is, TF-IDF calculation and channel pruning are performed separately for each category to be deleted to form a category-independent mask, and then the final channel deletion set is generated through intersection operation to avoid excessive destruction of feature space due to overlapping target categories.

[0079] The class-level federated forgetting method and system proposed in this invention, based on a two-layer expert hybrid approach and TF-IDF discriminative pruning, enables clients with different modalities and model structures to perform personalized and controllable class-level forgetting operations in federated learning. This avoids full model retraining, significantly reduces the cost of forgetting, and maximizes the retention of reasoning ability for unforgotten categories.

[0080] Furthermore, the entire forgetting process can be flexibly inserted into the communication cycle of model training, supporting both unified deletion after training and intermittent execution during training, providing technical support for different application scenarios (such as edge data compliance and user-initiated revocation). The average latency of executing one forgetting process is much less than the time required for full model retraining, and it remains within the accuracy degradation control range (typically <3%), meeting the performance and efficiency requirements of industrial-grade data deletion.

[0081] Experiments have confirmed that after a single forgetting trigger, the accuracy of target category recognition decreases significantly, while the average accuracy of non-target categories does not decrease significantly. While maintaining the same forgetting depth, the communication overhead is significantly reduced compared to full retraining. Compared to mainstream differential privacy noise injection or full retraining strategies, this invention achieves synergistic optimization in model fidelity, communication and computation costs, and forgetting interpretability, demonstrating significant technological advancements.

[0082] The key technical points of the class-level federated forgetting method and system based on two-layer expert hybridization and TF-IDF discriminative pruning designed in this invention include, but are not limited to: (1) For non-IID federated learning scenarios, a "two-layer expert hybrid" forgetting framework is constructed. On the client side, the dynamic fusion of local experts and global experts is achieved through the inner layer MoE, and on the server side, the unified coordination across clients and modalities is achieved through the outer layer MoE. From the perspective of architecture, global consistency and local differences are taken into account, which significantly improves the portability and applicability of class-level forgetting.

[0083] (2) A channel-level TF-IDF discriminant index is introduced, which measures the discriminative power of each channel for a specified category by the intensity of convolutional activation. This provides an interpretable quantitative standard for the forgetting range and serves as a pruning threshold to ensure consistent forgetting depth. Through this quantification mechanism, accurate category-level influence removal can be achieved without relying on the original data.

[0084] (3) A distributed discriminative pruning-aggregation process was designed. The server performs discriminative pruning on the feature extractor uploaded by the client based on TF-IDF weights and generates a new global expert using a weighted aggregation strategy. The client synchronously performs isomorphic pruning and performs lightweight fine-tuning. While maintaining the stable convergence of the global model, the accuracy of the target class is reduced to zero while only slightly impairing the performance of the non-target class.

[0085] (4) A lightweight communication and batch normalization alignment mechanism is proposed. Only the aligned feature extractor is transmitted instead of the complete model weights. Combined with batch normalization realignment, the model distribution drift is suppressed. While ensuring privacy and security, the communication overhead is reduced to one-fifteenth of the traditional retraining scheme, the server computation is reduced to one-seventh, and the client computation is reduced to one-sixtieth.

[0086] (5) Construct verifiable forgetting criteria. Utilize TF-IDF scores and pruned records to provide a traceable and auditable objective chain of evidence for forgetting effects, meet the rigid requirements for post-event verification in compliance scenarios, and lay the technical foundation for industrial-grade privacy governance and accountability.

[0087] The class-level federated forgetting method and system based on bi-layer expert hybridization and TF-IDF discriminative pruning proposed in this invention can solve the technical bottleneck of class-level data forgetting in heterogeneous federated learning environments, which makes it difficult to balance efficiency, accuracy and verifiability. It also reduces communication and computing costs while meeting the "right to be forgotten" requirements of regulations such as the GDPR.

[0088] Corresponding to the above method, the present invention also provides a class-level federated forgetting system based on two-layer expert hybrid and TF-IDF discriminative pruning. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0089] Corresponding to the methods described above, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0090] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0091] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0092] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0093] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A class-level federated forgetting method based on two-layer expert hybridization and TF-IDF discriminative pruning, characterized in that, Within each communication round, the method includes: The server distributes the initial or updated global feature extractor to each client participating in federated learning training; each client then uses an inner-layer expert hybrid model to aggregate the global feature extractor and its locally stored local feature extractor. Receive local feature extractors uploaded by various clients, which are aggregated using an inner-layer expert hybrid model; The aggregation of all received local feature extractors using an outer layer expert hybrid model specifically includes: when a forget request containing a target category set is received from the client, calculating the channel-level TF-IDF distribution of each local feature extractor across each target category in the target category set; calculating the channel-level discriminant score of each local feature extractor across each target category based on the channel-level TF-IDF distribution; pruning channels in each local feature extractor whose channel-level discriminant scores exceed a set threshold; determining the weighting coefficients of each local feature extractor based on the channel-level discriminant scores; weighting and aggregating all pruned local feature extractors according to the weighting coefficients; and aligning the weighted aggregation results to obtain an updated global feature extractor; when no forget request is received, weighted aggregation of all received local feature extractors is performed according to a fixed gating strategy.

2. The method according to claim 1, characterized in that, The inner-layer expert hybrid model includes a global feature extractor, a local feature extractor, a first gating network, and a prediction head module; Each client utilizes an inner-layer expert hybrid model to aggregate the global feature extractor and its locally stored local feature extractor, including: In the first round of communication, the first gating network fuses the global feature extractor and the local feature extractor according to a preset ratio; Starting from the second round, the first gating network determines the channel-level discriminative score based on the channel-level TF-IDF distribution, dynamically adjusts the weighting coefficients of the global feature extractor and the local feature extractor using the channel-level discriminative score, and aggregates the global feature extractor and the local feature extractor according to the weighting coefficients.

3. The method according to claim 1, characterized in that, The method also includes a federated forgetting operation verification step, which specifically includes: after completing each round of federated forgetting, the server records the set of removed channels in the global feature extractor and saves the TF-IDF distribution snapshot for each round. By comparing the output probability of the global feature extractor in the target category before and after pruning, the degree of prediction failure is evaluated to directly verify the effect of federated forgetting.

4. The method according to claim 3, characterized in that, The method further includes: Without accessing the original data, stability testing of federated learning training is performed to indirectly confirm whether the effects of removed categories have been removed from the model structure.

5. The method according to claim 1, characterized in that, When the target category set contains multiple target categories, the pruning process includes: Following a multi-category sequential processing mechanism, channel-level TF-IDF distribution and channel-level discriminant score are calculated separately for each target category. Then, a column-independent mask is formed for each channel. Channel deletion set is generated through intersection operation, and each local feature extractor is pruned according to the channel deletion set.

6. A class-level federated forgetting method based on two-layer expert hybridization and TF-IDF discriminative pruning, characterized in that, Within each communication round, the method includes: The client receives the initial or updated global feature extractor from the server. The aggregation of the global feature extractor and its locally stored local feature extractors using an inner-layer expert hybrid model specifically includes: performing the same structural pruning on the local feature extractors based on the pruning process of the global feature extractor to maintain consistency in the federated forgetting depth between the client and server sides; calculating the channel-level TF-IDF distribution of each local feature extractor in each target category; calculating the channel-level discriminant score of each local feature extractor in each target category based on the channel-level TF-IDF distribution; determining the weighting coefficients of the global feature extractor and the pruned local feature extractors based on the channel-level discriminant scores; and locally weighting and aggregating the global feature extractor and the pruned local feature extractors according to the weighting coefficients. After the current round of federated learning is completed, the locally stored local feature extractors are uploaded to the server. The server then uses an external expert hybrid model to aggregate all received local feature extractors. Specifically, this includes: when a forget request containing a target category set is received from the client, calculating the channel-level TF-IDF distribution of each local feature extractor across each target category in the target category set; calculating the channel-level discriminant score of each local feature extractor across each target category based on the channel-level TF-IDF distribution; pruning channels in each local feature extractor whose channel-level discriminant scores exceed a set threshold; determining the weighting coefficients of each local feature extractor based on the channel-level discriminant scores; weighting and aggregating all pruned local feature extractors according to the weighting coefficients; and aligning the weighted aggregation results to obtain an updated global feature extractor. If no forget request is received, weighted aggregation of all received local feature extractors is performed according to a fixed gating strategy.

7. The method according to claim 6, characterized in that, The method further includes: fine-tuning the global feature extractor based on the local feature extractor after structural pruning, according to an annealed learning rate strategy.

8. The method according to claim 6, characterized in that, The method also includes performing local precise forgetting and / or local forgetting recovery on the client side, specifically including: Local precise forgetting utilizes the calculated channel-level TF-IDF distribution and channel-level discriminant score to mask channels in the local feature extractor whose channel-level discriminant scores exceed a set threshold. Local forgetting recovery retains all channels deleted by the local feature extractor. When a local forgetting recovery request is received, the target channel included in the local forgetting recovery request is recovered.

9. A class-level federated forgetting system based on two-layer expert hybrid and TF-IDF discriminative pruning, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images