A personalized federated learning model training method for alleviating feature and knowledge misalignment

Abstract

The application discloses a personalized federated learning model training method for relieving feature and knowledge dislocation, and belongs to the technical field of federated learning model training. The method solves the problems of feature dislocation, decision maker knowledge dislocation, rare class knowledge dilution and unstable decision boundary in personalized federated learning. A server initializes global feature extractor and global decision maker parameters and issues the parameters, a client updates a backbone network and a local personalized decision maker through two-step decoupling optimization based on local image data, generates a class prototype, and fine-tunes the global decision maker after freezing the backbone network; the client uploads the backbone network, the fine-tuned decision maker, the class prototype and the number of samples of each class, and the local personalized decision maker is reserved; the server aggregates the backbone network to obtain the global feature extractor, generates a client customized global decision maker according to the double-factor similarity of the prototype direction and the sample quantity, issues an update of the local model, and completes image processing by fusing the outputs of the two types of decision makers during reasoning.

Description

Technical Field

[0001] This invention belongs to the field of federated learning model training technology, specifically involving a personalized federated learning model training method to alleviate feature and knowledge misalignment. Background Technology

[0002] Federated Learning (FL) is a distributed artificial intelligence machine learning paradigm that allows multiple participants to collaboratively train neural network models without sharing the original data, thereby leveraging distributed data resources while protecting data privacy. A typical FL framework achieves collaborative training through the following process: a server distributes a global model, clients train on local data and upload model updates, and the server aggregates the updates and continues iterating. Because it only exchanges model parameters without exposing the original data, this approach has a natural advantage in privacy protection.

[0003] However, schemes such as FedAvg assume that all clients' data follows the same distribution. In real-world cross-device and cross-organizational scenarios, data often comes from different users, devices, and organizations, and is rarely independently identically distributed (non-IID). This statistical heterogeneity makes it difficult for a single global model to simultaneously adapt to the data distributions of all clients, leading to performance degradation and convergence difficulties, and even performing worse than models trained locally by each client in isolation.

[0004] To mitigate the problem of statistical heterogeneity, personalized federated learning (pFL) has been proposed, aiming to build client-specific models for each client that can adapt to the characteristics of their local data while retaining the advantages of cross-client collaboration. Among various personalization techniques, a mainstream approach is parameter decoupling, which divides the model into a feature extractor (backbone network) and a decision network: the backbone network is shared and aggregated among clients to synchronize feature extraction capabilities, while the decision network remains personalized to focus on the task-related classification for each client.

[0005] However, existing decoupling-based personalized federated learning methods still suffer from two types of misalignment problems in the face of statistical heterogeneity: First, there is a feature mismatch between the backbone network and the decision network. At the beginning of each round of communication, the client's local backbone network is replaced by the aggregated global backbone network. However, the global backbone network reflects the averaged feature distribution across all clients, which deviates from the local distribution of any individual client. When the client's locally fine-tuned decision network is directly applied to this global backbone network, the received feature representations deviate from its optimized distribution area, resulting in a backbone network-classifier mismatch. This reduces decision accuracy and makes training unstable.

[0006] Second, there is a knowledge misalignment within the decision-making network. On the one hand, a single decision-maker cannot simultaneously carry out cross-client collaborative knowledge and specialize it in a local distribution, thus their goals conflict. On the other hand, aggregating each decision-maker with uniform weights for each client will treat each client indiscriminately, without considering how much data that client actually holds for a particular decision category. Therefore, rare or missing categories on a particular client are easily diluted and dominated by uninformative or noisy contributions.

[0007] Existing technologies struggle to simultaneously address both types of misalignment. For instance, schemes that merely share the backbone network while retaining purely local decision-makers (such as FedPer and FedRep) fail to provide decision-makers with any category-level collaborative knowledge. Schemes employing a unified global shared head (such as FedRoD) have the same, rather than customized, head for each client, and the feature extractor is driven solely by the global objective, preventing personalized heads from shaping the representations they rely on. Schemes that directly reuse local decision-makers on the aggregated backbone network (such as FedAS) fail to achieve mutual adaptation, leaving behind a feature misalignment between the backbone network and the decision-maker. Prototype-based schemes (such as FedProto and FedGH), while using lightweight class prototypes as a communication medium to reduce communication costs and enhance privacy, aggregate prototypes with category-independent uniform weights, leading to unreliable clients dominating prototypes in rare or missing categories, resulting in biased global prototypes. Furthermore, prototypes are merely used as averaging targets, failing to leverage their potential to guide "which clients should collaborate on which decision category." Furthermore, if weighting is based solely on the prototype orientation (such as cosine similarity), the cosine similarity between non-zero prototypes tends to be concentrated in a narrow range and has low distinguishability, making it difficult to effectively distinguish the reliability of different collaborators.

[0008] In summary, the core problem currently faced by personalized federated learning in decision-making tasks is that there is feature mismatch between the aggregate backbone network and the local decision-maker, and knowledge mismatch within the decision-maker caused by class-independent uniform aggregation. Both of these factors lead to feature representation mismatch, unstable decision boundaries, and dilution of rare class knowledge, ultimately limiting the improvement of decision accuracy and robustness. Summary of the Invention

[0009] To address the issues of feature misalignment caused by the mismatch between the aggregated backbone network and the local decision-maker, knowledge misalignment and dilution of rare class knowledge caused by class-independent uniform aggregation, and unstable decision boundaries in personalized federated learning, this invention provides a training method for personalized federated learning models that mitigates feature and knowledge misalignment.

[0010] The method is specifically as follows: Step 1: The server initializes global model parameters, which include global feature extractor parameters and global decision-maker parameters; the server then distributes the initialized global model parameters to the selected clients. Step 2: The client receives the global feature extractor parameters and the global decision-maker parameters customized for the client, initializes the local backbone network with the global feature extractor parameters, and updates the local backbone network and the local personalized decision-maker through two-step decoupling optimization using local image training data; at the same time, the client generates a category prototype based on the updated local backbone network. Step 3: The client freezes the updated local backbone network, fine-tunes the customized global decision-maker to adapt to the feature space of the local backbone network, obtains the fine-tuned global decision-maker, and freezes it. Step 4: The client uploads the updated local backbone network parameters, the fine-tuned global decision-maker parameters, the category prototype and the number of samples per category to the server, while the local personalized decision-maker is kept locally and not uploaded. Step 5: The server aggregates the local backbone network parameters uploaded by all clients to obtain updated global feature extractor parameters; and calculates the two-factor similarity that takes into account both the prototype direction and the number of samples for each category of the uploaded client. Based on this, the global decision-maker of each client is aggregated in a weighted manner to generate a global decision-maker customized for that client. Step 6: The server distributes the updated global feature extractor parameters and the global decision-maker parameters customized for each client to the corresponding client. The client updates its local model accordingly to obtain the updated local model. Step 7: The client uses the updated local model to process the image to be processed, and performs weighted fusion of the outputs of the global decision-maker and the local personalized decision-maker to obtain the image processing result.

[0011] Furthermore, in step 1, the process of initializing the global model parameters is as follows: the server generates initial parameters based on the global model and stores the initial parameters on the server as the global model baseline; the backbone network of the global model adopts the same architecture on each client. The process of distributing the global model parameters to the selected clients via the server is as follows: First, the server randomly selects a set of participating clients from the entire client set according to a set participation ratio. The number of selected clients is the product of the participation ratio and the total number of clients in the entire set, rounded down. Then, the server distributes the parameters to the selected clients. The updated global feature extractor parameters and the global decision-maker parameters customized for each client are distributed to each client in the client set.

[0012] Furthermore, the two-step decoupling optimization includes cross-entropy update and prototype anchoring alignment, in Within each local training cycle, the following two steps are performed alternately within each mini-batch of image data: a cross-entropy update step, which updates both the local backbone network and the local personalized decision-maker; and a prototype anchoring alignment step, which updates only the local backbone network using the locally cached category prototypes as distribution anchors.

[0013] Furthermore, generating category prototypes includes the following steps: Local feature collection: During the last local training cycle, the client iterates through the categories in the local training set. Image samples are input into the updated local backbone network to obtain corresponding local feature representations. The local feature collection is performed in an embedding manner, without generating additional data traversal. Category prototype generation: The client generates prototypes for categories. The sample calculates the category prototype, which is the mean of the feature representation of all samples in that category. The generated category prototype is cached locally and used as the distribution anchor point for prototype anchoring alignment in the next round of prototype anchoring alignment steps.

[0014] Furthermore, the fine-tuning process for the customized global decision-maker is as follows: under the condition of freezing the updated local backbone network, using the received customized global decision-maker parameters as initial values, fine-tuning is performed on the local image training data with cross-entropy loss for one local training cycle to obtain the fine-tuned global decision-maker parameters, so that the customized global decision-maker is adapted to the feature space obtained by training the local backbone network; the fine-tuned global decision-maker parameters are frozen immediately after the fine-tuning is completed.

[0015] Furthermore, when the client uploads the updated local backbone network parameters, the fine-tuned global decision-maker parameters, the category prototype, and the number of samples per category to the server, it also uploads the client's identifier and training round information to the server so that the server can record the set of clients participating in this round of training.

[0016] Furthermore, the aggregation of local backbone network parameters is specifically as follows: the server accumulates the local backbone network parameters uploaded by each client participating in this round layer by layer, and averages them according to the number of clients: for each layer of parameters of the backbone network, the corresponding layer parameters of each client are accumulated, and then divided by the number of participating clients to obtain the updated global feature extractor parameters.

[0017] Furthermore, generating a global decision-maker customized for the client includes the following steps: Step 511, Prototype Normalization: The server performs L2 normalization on the category prototypes uploaded by each client. Step 512, two-factor similarity calculation, for each category and any two clients and Calculate the category-level two-factor similarity that takes into account both prototype orientation and sample size; Step 513, mask normalization, define the client. In category The effective set of collaborators on the client, i.e. and All holding categories The samples; apply a temperature-scaled mask to the effective collaborators to obtain class-by-class aggregation weights; Step 514: Weighted aggregation by category. The server performs a weighted summation of the global decision-makers uploaded by each client according to the aforementioned weighted aggregation categories, generating the client's... A customized global decision-maker.

[0018] Furthermore, the client updates the local model as follows: the server saves the updated global feature extractor parameters and the global decision-maker parameters customized for each client to the parameter repository according to the communication round number; the server distributes the updated global feature extractor parameters, the global decision-maker parameters customized for each client, and the training round information to the corresponding clients according to the client selection results in the next round; the client updates its local backbone network with the updated global feature extractor parameters, updates its local customized global decision-maker with the global decision-maker parameters customized for each client, and combines it with the locally retained personalized decision-maker to obtain the updated local model.

[0019] Furthermore, image processing specifically includes: Step 71: The client performs preprocessing on the image to be processed, including normalization, resizing, and format conversion. Step 72: Input the preprocessed image into the updated local backbone network to obtain the image feature representation; Step 73: Input the image feature representation into the global decision-maker and the local personalized decision-maker respectively to obtain the global prediction and personalized prediction, and then calculate the result according to the fusion coefficient. Weighted fusion.

[0020] The beneficial effects of the method are as follows: This invention uses a prototype-anchored feature alignment mechanism, taking the client's own category prototype as the distribution anchor point, to pull the aggregated global backbone network back to the client's local data distribution. This solves the feature misalignment problem between the aggregated backbone network and the local decision-maker, effectively coordinates global semantic consistency and local distribution differences, suppresses feature drift, stabilizes training dynamics, and thus improves image decision accuracy and training stability in heterogeneous data scenarios.

[0021] This invention employs a prototype-guided dual-head customization mechanism. On the server side, it uses a dual-factor similarity approach, balancing prototype orientation and sample quantity, as weights and applies mask normalization to perform category-level weighted aggregation of the global decision-makers for each client. This creates a customized global decision-maker for each client, resolving the knowledge misalignment problem within the decision-maker. For each category, knowledge is extracted only from reliable collaborators with similar prototype orientations and sufficient samples. Clients lacking that category are assigned zero weight, thus preventing rare or missing categories from being diluted and dominated by unreliable clients, unifying and stabilizing the decision boundary. This mechanism requires no public datasets or complex generative models, meeting the stringent requirements of modern federated learning environments for communication efficiency and privacy protection.

[0022] This invention employs a dual-decision-maker collaborative design, which weights and fuses the outputs of the global decision-maker and the local personalized decision-maker during inference. This allows the global decision-maker to provide cross-client collaborative knowledge, while the local personalized decision-maker retains client-specific knowledge. The two complement each other, further enhancing robustness to statistical heterogeneity and overall decision performance.

[0023] This invention uses lightweight category prototypes as a communication medium, with the same set of category prototypes acting as distribution anchors on the client side and semantic guides on the server side, enabling feature alignment and decision-maker customization to be driven by structural consistency among the same clients. Since only backbone network parameters, global decision-maker parameters, and category prototypes are exchanged without exchanging original data, local personalized decision-makers, or complete models, communication and computational overhead, privacy leakage risks, and model structure leakage risks are significantly reduced, while improving the efficiency of knowledge sharing among heterogeneous clients.

[0024] The method described in this invention is applicable to fields with high data privacy requirements and significant differences in data distribution among participating parties, such as medical image analysis, industrial visual inspection, and edge intelligent devices. Attached Figure Description

[0025] Figure 1This is a flowchart of the application of the personalized federated learning model training method for mitigating feature and knowledge misalignment to image classification in Implementation 1. The execution subjects and execution order of steps 1 to 7 are shown in two swimlanes, one on the server side and one on the client side, as well as the interaction relationship between the server and the client in parameter distribution and uploading in each communication round. Figure 2 This is a diagram illustrating the overall framework and data flow of a personalized federated learning model training method for image classification that alleviates the misalignment between features and knowledge. It shows the dual roles of the same set of category prototypes: on the client side, they serve as distribution anchors for prototype feature alignment (role 1); on the server side, they serve as semantic guides for prototype dual-head customization (role 2). The diagram also shows the dual-head structure on the client side consisting of a feature extractor, a global prediction head, and a personalized prediction head, as well as the weighted fusion based on the fusion coefficient α and the frozen / trainable states of each module. Figure 3 This is a schematic diagram of the training process of the client in one communication round as described in Implementation Method 1, which shows the following steps in sequence: Step 1: Download and replace the local global prediction head and backbone network with a customized global prediction head and a global feature extractor; Step 2: Perform joint optimization of the backbone network and personalized prediction head for E local cycles with the local category prototype as the anchor point; Step 3: Freeze the backbone network and fine-tune the global prediction head; Step 4: Collect the category prototype and upload the backbone network, global prediction head and category prototype. Figure 4 This is a two-step decoupling optimization diagram of client-side prototype anchoring feature alignment as described in Implementation Method 3, showing step A cross-entropy update (updating both the local backbone network and the personalized prediction head simultaneously) and step B prototype anchoring alignment (updating only the local backbone network using the category prototype cached in the previous round as the anchor point). Figure 5 This is a schematic diagram of the principle of server-side prototype-guided dual-head customization described in Implementation Method 7. It shows the process of calculating class-level dual-factor similarity from category prototypes and class-by-class sample quantity, obtaining class-by-class aggregation weights through mask Softmax normalization, and then weighting and aggregating each client's global prediction head into a customized global prediction head. Figure 6 This is a schematic diagram comparing the accuracy of the present invention with that of various comparison methods under the practical heterogeneity setting of the CIFAR-100 dataset. Detailed Implementation

[0026] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] Implementation Method 1, see Figure 1 and Figure 3 ( Figure 3 In This represents the global prediction header parameters before local fine-tuning. This implementation method describes the backbone network parameters before local fine-tuning. This implementation method provides a personalized federated learning model training method to alleviate feature and knowledge misalignment, and applies the method to image classification, where the decision-maker is a prediction head / classifier for image classification. The method includes the following steps: Step 1: The server initializes global model parameters, which include global feature extractor parameters and global prediction head parameters; the server then distributes the initialized global model parameters to the selected clients. Step 2: The client receives the global feature extractor parameters and the global prediction head parameters customized for the client. It initializes the local backbone network with the global feature extractor parameters and updates the local backbone network and the local personalized prediction head using local image training data through a two-step decoupling optimization: the cross-entropy step updates both the local backbone network and the local personalized prediction head simultaneously, while the prototype anchoring alignment step updates only the local backbone network using the locally cached category prototypes as distribution anchors. At the same time, the client generates category prototypes based on the updated local backbone network. Step 3: The client freezes the updated local backbone network, fine-tunes the customized global prediction head to fit the feature space of the local backbone network, and obtains the fine-tuned global prediction head and freezes it. Step 4: The client uploads the updated local backbone network parameters, the fine-tuned global prediction head parameters, the category prototype and the number of samples per category to the server, while the local personalized prediction head is kept locally and not uploaded. Step 5: The server aggregates the local backbone network parameters uploaded by all clients to obtain updated global feature extractor parameters; and calculates the two-factor similarity that takes into account both the prototype direction and the number of samples for each category uploaded by each client. Based on this, the global prediction head of each client is aggregated in a weighted manner to generate a global prediction head customized for that client. Step 6: The server distributes the updated global feature extractor parameters and the global prediction head parameters customized for each client to the corresponding client. The client updates its local model accordingly and obtains the updated local model. Step 7: The client uses the updated local model to classify the image to be classified, and performs weighted fusion of the outputs of the global prediction head and the local personalized prediction head to obtain the image classification result.

[0028] Before the first round of communication, each client first performs local warm-up training for several cycles using cross-entropy loss, and extracts and caches the initial category prototypes and records the number of samples for each category, so as to provide the initial basis for subsequent prototype anchoring alignment and prototype guided aggregation.

[0029] Implementation Method 2: This implementation method is a further limitation of Implementation Method 1, and provides an example of step 1, which involves initializing the global model parameters.

[0030] The process of initializing global model parameters is as follows: the server generates initial parameters based on the global model and stores the initial parameters on the server as the baseline of the global model; the backbone network of the global model adopts the same architecture on each client, such as a four-layer convolutional neural network or residual network for image classification tasks; The process of distributing the global model parameters to the selected clients via the server is as follows: First, the server randomly selects a set of participating clients from the entire client set according to the set participation ratio:

[0031] in Indicates the current communication round. This indicates the complete client set. The selected quantity is the product of the participation ratio and the total number of elements in the entire set, rounded down. In a fully participatory scenario ; This indicates a random selection operation.

[0032] Then, the server will... Global feature extractor parameters after each round of updates With global prediction header parameters customized for each client Distribute to the set Each client in .

[0033] Implementation Method 3, see below Figure 4 ( Figure 4 middle This represents the predicted output of the model's classification head. This embodiment describes the sample feature vector extracted by the model. This embodiment further defines embodiment one and provides an example of step 2, the two-step decoupling optimization.

[0034] The two-step decoupling optimization is in Within each local training cycle, the following two steps are performed alternately within each mini-batch of image data: Step 211, Cross-entropy Update, given a mini-batch of image data , This refers to a mini-batch dataset, and the client... Utilizing local personalized prediction heads Calculate the cross-entropy loss and update the local backbone network simultaneously. With local personalized prediction head parameters :

[0035]

[0036] in, express and The update function, Indicates that it is powered by the local backbone network. For images Extracted feature representation, For cross-entropy loss, Indicates the learning rate. The loss function represents the loss function for the backbone network. gradient, This indicates the loss function applied to the personalized prediction head parameters. The gradient.

[0037] The gradients of the local backbone network are passed only through the local personalized prediction head. The data is fed back without going through the frozen global prediction head, thereby promoting collaboration between the local backbone network and local personalized prediction heads in capturing local proprietary knowledge. Since the gradients of the local backbone network are only fed back through the local personalized prediction head and not through the frozen global prediction head, this update enables the local backbone network and the local personalized prediction head to work together to capture local knowledge while maintaining the discriminative ability of the local backbone network. Step 212, prototype anchoring and alignment, using the local backbone network updated in step 211. Re-extract features from the same mini-batch of data, calculate the prototype anchoring alignment loss, and update only the local backbone network:

[0038]

[0039] in, Represents the L2 norm, For the previous round of caching, category The corresponding local category prototype, To control the hyperparameters of alignment strength; when It degenerates into pure cross-entropy training. The larger the value, the stronger the alignment with the local prototype; this step only updates the local backbone network, while the local personalized prediction head remains unchanged; By alternating between steps 211 and 212 within each mini-batch, a balance is achieved between local discriminative learning and prototype anchoring distribution regularization. This ensures that the local backbone network maintains its classification and discrimination capabilities while its feature representations continuously align with the local data distributions of each client, thereby alleviating feature misalignment between the aggregated backbone network and the local classification head.

[0040] Decoupling the two steps instead of merging them into a single composite loss avoids convergence instability and alignment failure caused by the mutual cancellation of the gradient directions of the classification target and the alignment target when they are inconsistent. By first discriminating and then aligning the distribution in each mini-batch, the distribution regularization anchored by the prototype is applied to the backbone network after classification optimization, resulting in feature representations that are both discriminative and aligned with the local distribution, thereby alleviating the feature misalignment between the aggregated backbone network and the local classifier.

[0041] Implementation Method Four: This implementation method is a further limitation of Implementation Method One, and provides an example of step 2, which involves generating a category prototype.

[0042] The process of generating category prototypes includes the following steps: Step 221, Local Feature Collection: During the last local training cycle, the client iterates through the categories in the local training set. Image samples are input into the updated local backbone network to obtain corresponding local feature representations. The local feature collection is performed in an embedding manner, without generating additional data traversal. Step 222, category prototype generation, the client identifies categories. The sample calculates the category prototype, which is the mean of the feature representations of all samples in that category:

[0043] in Indicates the client The local data is labeled as A subset of samples; client The set of category prototypes is denoted as The generated category prototypes are cached locally and used as distribution anchor points for prototype anchoring alignment in the next step 212. This indicates the total number of categories.

[0044] Implementation Method 5: This implementation method further defines Implementation Method 1, and provides examples of steps 3 and 4, namely, the global prediction head fine-tuning and uploading.

[0045] The process of fine-tuning the customized global prediction head is as follows: freeze the updated local backbone network. Under the condition of receiving the customized global prediction header parameters Using the initial values, fine-tuning is performed on the local image training data with cross-entropy loss for one local training cycle to obtain the fine-tuned global prediction head parameters. This allows the customized global prediction head to be adapted to the feature space obtained from training the local backbone network; the settings are then frozen immediately after fine-tuning. This is to prevent cross-client collaborative knowledge from being overwritten by subsequent training. The upload process is as follows: the client uploads the updated local backbone network parameters. Fine-tuned global prediction head parameters Category Prototype Set and the number of samples per class The client's identifier and training round information are uploaded to the server so that the server can record the set of clients participating in this training round; while the local personalized prediction head parameters It keeps data locally and never uploads it. Because the client only uploads backbone network parameters, global prediction head parameters, and category prototypes, and does not upload original images, local personalized prediction heads, or complete local models, the risk of privacy and model structure leakage is significantly reduced.

[0046] Implementation Method Six: This implementation method is a further limitation of Implementation Method One, and provides an example of step 5, which involves aggregating the local backbone network parameters.

[0047] The process of aggregating local backbone network parameters is as follows: The server accumulates the local backbone network parameters uploaded by each client participating in this round layer by layer, and averages them according to the number of clients: For each layer of backbone network parameters, the corresponding layer parameters of each client are accumulated, and then divided by the number of participating clients. The updated global feature extractor parameters are obtained: .

[0048] Implementation Method Seven, see below Figure 5 ( Figure 5 In this embodiment, M represents the total number of clients participating in the training. This embodiment further defines the first embodiment and provides an example of step 5, which involves generating a customized global prediction head for each client.

[0049] The process of generating a customized global prediction head for each client includes the following steps: Step 511, Prototype Normalization: The server performs L2 normalization on the category prototypes uploaded by each client.

[0050] Step 512, two-factor similarity calculation, for each category and any two clients and Calculate the category-level two-factor similarity that takes into account both prototype orientation and sample size:

[0051] Among them This represents the inner product, which, since the prototypes have been normalized, is equivalent to the cosine similarity between the two prototypes, reflecting the similarity between the two clients in terms of category. Semantic direction consistency; the latter factor The weights are based on the number of samples, reflecting the client's... In category The statistical reliability of the prototype estimate characterizes the statistical reliability of the prototype estimate. For the client The number of samples per class, Represents categories in all clients Maximum number of samples .

[0052] This two-factor design significantly enhances the discriminability of similarity, overcomes the problem that relying solely on orientation metrics makes it difficult to distinguish reliable collaborators due to the narrow range of cosine similarity, and enables subsequent temperature scaling to play an effective role. Step 513, mask normalization, define the client. In category Effective collaborator set client and All holding categories The samples; apply a temperature-scaled mask softmax to the effective collaborators to obtain class-wise aggregation weights:

[0053] in, Indicates the client and client In category Class-wise aggregation weights , Indicates the client and In category Class-level similarity, where Temperature hyperparameters used to control the concentration of weight distribution; for clients The category that is completely missing (Right now , The row weights for this category are reset to zero; the masking operation removes invalid pairs from both the numerator and denominator to prevent them from retaining non-negligible weights after softmax. ; Step 514: Weighted aggregation by category. The server performs a weighted summation of the global prediction headers uploaded by each client according to the aforementioned weighted aggregation methods, generating the client's prediction headers. Custom global prediction head:

[0054] in, Indicates the first In the round of iteration, for the client Customized global prediction head in category Weight on, For the client The first of the global prediction head classification weight matrix Line, i.e., category Associated parameters; because aggregation is performed independently by category, the client... The custom global prediction head actually comes from different combinations of collaborators for different categories: for categories that the client already has, aggregation focuses on collaborators with similar prototypes to reinforce existing knowledge; for categories with very few samples, the sample number weight will direct aggregation towards collaborators with sufficient reliable data to supplement the knowledge of underrepresented categories; for categories that are completely missing, the corresponding prediction head line does not receive collaborative updates, as the client never predicts this category on the local distribution, so it has no adverse effect.

[0055] The neural feature transformation or projection module can be used as an optional component to further map each client category prototype to a shared latent space before aggregation to enhance intra-class consistency and inter-class separability. However, in this invention, the client backbone network adopts a homogeneous architecture, and its category prototypes are in a consistent feature space, so weighted aggregation can be performed directly using the above-mentioned two-factor similarity.

[0056] Implementation Method Eight: This implementation method is a further limitation of Implementation Method One, and provides an example of step 6, where the client updates the local model.

[0057] The server will update the global feature extractor parameters. With global prediction header parameters customized for each client Save the data to the parameter repository according to the communication round number; the server will then, based on the client selection results in the next round, , And the training round information is distributed to the corresponding clients; the clients use Update the local backbone network, Update the local custom global prediction head and combine it with the locally retained personalized prediction head. This yields an updated local model. Intuitively, a customized global prediction head can accelerate local convergence by preserving the personalized knowledge in the local backbone network and the personalized prediction head, while also integrating global decision-making capabilities from reliable collaborators, thus improving robustness to statistical heterogeneity.

[0058] Implementation Method Nine: This implementation method is a further limitation of Implementation Method One, and provides an example of step 7, the image classification.

[0059] The image classification includes the following steps: Step 71: The client performs preprocessing on the images to be classified, including normalization, resizing, and format conversion. Step 72: Input the preprocessed image into the updated local backbone network to obtain the image feature representation; Step 73: Input the image feature representation into the global prediction head and the local personalized prediction head respectively to obtain the global prediction and personalized prediction, and then calculate the result according to the fusion coefficient. Weighted fusion: ; in, This represents the prediction result of the global prediction head. This indicates the prediction results from the local personalized prediction head; Let be the dual-head fusion coefficient, taken as... The category corresponding to the largest component is used as the classification result and output. This process can be completed locally without uploading the original image or intermediate features to the server, thus protecting user privacy. Clients can choose to use the classification results locally only or upload the aggregated statistical information (such as classification accuracy) as needed, to achieve a privacy-friendly image classification service.

[0060] Implementation Method 10: This implementation method further defines Implementation Method 1 and provides an example of hyperparameter configuration.

[0061] Under practicality settings, the alignment strength As the dominant factor affecting performance, the best results are achieved when taking the median value: too small a value results in weak alignment, degenerating into ordinary cross-entropy training; too large a value causes the alignment term to dominate, collapsing features to their prototypes and sacrificing discriminability. Under the condition of taking the median value, the performance is affected by temperature. With fusion coefficient They are not sensitive to fluctuations over a wide range, making them easy to optimize; as an example, the local learning rate can be used. Local training cycle Alignment strength ,temperature Fusion coefficient This approach balances the selectivity of category-based aggregation with the collaboration of global prediction heads and local personalized prediction heads.

[0062] Implementation Method 11: This implementation method evaluates the effectiveness of the invention on several widely used image classification benchmark datasets.

[0063] The image classification benchmark data includes Fashion-MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet; among which Fashion-MNIST contains 70,000 grayscale images of clothing in 10 categories, CIFAR-10 and CIFAR-100 contain 60,000 natural images in 10 and 100 categories respectively, and Tiny-ImageNet contains 100,000 images in 200 categories. The method proposed in this invention is compared with representative methods widely used in personalized federated learning, including: FedAvg Federated Averaging Algorithm; FedProto Federated Prototype Algorithm; FedGH FedGlobal-Heterogeneous Algorithm; FedPer Federated Personalization Algorithm; LG-FedAvg Local-Generative FedAvg Algorithm; FedRep Federated Representation Algorithm; FedRoD Federated Representation-Offset Decoupling Algorithm; FedCP Federated Client-Prototype Algorithm; FedAS Federated Adaptive Selection Algorithm; FedAMP Federated Adaptive Message Passing Algorithm; FedFomo Federated Follow-Only-Me Algorithm; pFedCS Personalized Federated Client-Selection Algorithm; FedALA Federated Adaptive Local Alignment Algorithm.

[0064] This implementation evaluates the method under two widely adopted statistical heterogeneity settings: a pathological setting and a utility setting. In the pathological setting, each client is assigned data from a fixed number of categories (e.g., sampling 2 / 2 / 10 categories from Fashion-MNIST / CIFAR-10 / CIFAR-100 respectively), resulting in extreme label skew. In the utility setting, a Dirichlet distribution (denoted as ) is used. Data is divided among clients, concentration parameters (Default 0.1) The smaller the value, the more pronounced the non-independent and identically distributed nature of the distribution. The method described in this invention is named pFedHCA. As shown in Table 1, pFedHCA outperforms all comparison methods on most datasets in both settings. In the pathological setting, pFedHCA achieves 99.87%, 91.80%, and 73.50% on Fashion-MNIST, CIFAR-10, and CIFAR-100, respectively, with a 3.05% improvement over the strongest comparison method, FedAS, on CIFAR-100. In the practical setting, pFedHCA achieves 61.98% on CIFAR-100, an improvement of approximately 4.46% over the strongest comparison method, and also leads on Tiny-ImageNet.

[0065] Table 1. Test accuracy (%) under pathological and practical settings.

[0066] This advantage stems from pFedHCA's use of prototype-anchored feature alignment on the client side to pull the aggregated backbone network back to its local distribution and alleviate feature misalignment, and its use of prototype-guided dual-head customization on the server side to perform category-level, dual-factor weighted customized aggregation of the classification heads and alleviate knowledge misalignment. This allows for the application of consistent and stable decision boundaries while maintaining low communication and computational overhead. Even in scenarios with extreme label skew and missing most categories on the client side, masking and sample quantity-aware aggregation can still assign zero weight to missing categories and borrow knowledge from reliable collaborators, maintaining the discriminative ability of the learning prototype.

[0067] To further verify the effectiveness of each component, ablation experiments were conducted under CIFAR-100 usability settings: removing the prototype-guided dual-head customization (changing to a uniform average aggregation classification head) resulted in an accuracy decrease of approximately 1.75%; removing the prototype anchored feature alignment (making...) The accuracy dropped by approximately 8.16%, the largest decrease, indicating that aligning the aggregated backbone network to the local distribution is crucial to mitigating feature misalignment; inference using only the global prediction head ( The rate decreased by approximately 1.89% when using personalized prediction head inference ( The performance decreased by about 1.92% when the two prediction heads were complementary and fused, indicating that the performance was optimal.

[0068] Figure 6 The test accuracy of this invention compared with various comparative methods is shown under the CIFAR-100 utility setting: pFedHCA rises rapidly after several rounds and continues to lead all comparative methods, and has small fluctuations and good stability after convergence.

[0069] In summary, this invention solves the technical problems of feature misalignment caused by the mismatch between the aggregated backbone network and the local classifier in personalized federated learning scenarios, as well as the misalignment of classifier knowledge and instability of decision boundaries caused by class-independent uniform aggregation, thereby improving the accuracy and robustness of image classification tasks in heterogeneous data scenarios.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for training a personalized federated learning model to alleviate the mismatch between features and knowledge, characterized in that, The method includes the following steps: Step 1: The server initializes global model parameters, which include global feature extractor parameters and global decision-maker parameters; the server then distributes the initialized global model parameters to the selected clients. Step 2: The client receives the global feature extractor parameters and the global decision-maker parameters customized for the client, initializes the local backbone network with the global feature extractor parameters, and updates the local backbone network and the local personalized decision-maker through two-step decoupling optimization using local image training data; at the same time, the client generates a category prototype based on the updated local backbone network. Step 3: The client freezes the updated local backbone network, fine-tunes the customized global decision-maker to adapt to the feature space of the local backbone network, obtains the fine-tuned global decision-maker, and freezes it. Step 4: The client uploads the updated local backbone network parameters, the fine-tuned global decision-maker parameters, the category prototype and the number of samples per category to the server, while the local personalized decision-maker is kept locally and not uploaded. Step 5: The server aggregates the local backbone network parameters uploaded by all clients to obtain updated global feature extractor parameters; and calculates the two-factor similarity that takes into account both the prototype direction and the number of samples for each category of the uploaded client. Based on this, the global decision-maker of each client is aggregated in a weighted manner to generate a global decision-maker customized for that client. Step 6: The server distributes the updated global feature extractor parameters and the global decision-maker parameters customized for each client to the corresponding client. The client updates its local model accordingly to obtain the updated local model. Step 7: The client uses the updated local model to process the image to be processed, and performs weighted fusion of the outputs of the global decision-maker and the local personalized decision-maker to obtain the image processing result.

2. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 1, characterized in that, In step 1, the process of initializing the global model parameters is as follows: the server generates initial parameters based on the global model and stores the initial parameters on the server as the global model baseline; the backbone network of the global model adopts the same architecture on each client. The process of distributing the global model parameters to the selected clients via the server is as follows: First, the server randomly selects a set of participating clients from the entire client set according to a set participation ratio. The number of selected clients is the product of the participation ratio and the total number of clients in the entire set, rounded down. Then, the server distributes the parameters to the selected clients. The updated global feature extractor parameters and the global decision-maker parameters customized for each client are distributed to each client in the client set.

3. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 2, characterized in that, The two-step decoupling optimization includes cross-entropy update and prototype anchoring alignment. Within each local training cycle, the following two steps are performed alternately within each mini-batch of image data: a cross-entropy update step, which updates both the local backbone network and the local personalized decision-maker; and a prototype anchoring alignment step, which updates only the local backbone network using the locally cached category prototype as the distribution anchor point.

4. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 3, characterized in that, Generate category prototypes Includes the following steps: Local feature collection: During the last local training cycle, the client iterates through the categories in the local training set. Image samples are input into the updated local backbone network to obtain corresponding local feature representations. The local feature collection is performed in an embedding manner, without generating additional data traversal. Category prototype generation: The client generates prototypes for categories. The sample calculates the category prototype, which is the mean of the feature representation of all samples in that category. The generated category prototype is cached locally and used as the distribution anchor point for prototype anchoring alignment in the next prototype anchoring alignment step.

5. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 4, characterized in that, The process of fine-tuning the customized global decision-maker is as follows: under the condition of freezing the updated local backbone network, the received customized global decision-maker parameters are used as the initial values, and the cross-entropy loss is used to fine-tune one local training cycle on the local image training data to obtain the fine-tuned global decision-maker parameters, so that the customized global decision-maker is adapted to the feature space obtained by training the local backbone network. Immediately after fine-tuning, freeze the fine-tuned global decision-maker parameters.

6. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 5, characterized in that, When the client uploads the updated local backbone network parameters, the fine-tuned global decision-maker parameters, the category prototype, and the number of samples per category to the server, it also uploads the client's identifier and training round information to the server so that the server can record the set of clients participating in this round of training.

7. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 6, characterized in that, The aggregation of local backbone network parameters is specifically performed as follows: the server accumulates the local backbone network parameters uploaded by each client participating in this round layer by layer, and averages them according to the number of clients: for each layer of parameters in the backbone network, the corresponding layer parameters of each client are accumulated, and then divided by the number of participating clients to obtain the updated global feature extractor parameters.

8. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 7, characterized in that, Generating a global decision maker customized for the client involves the following steps: Step 511, Prototype Normalization: The server performs L2 normalization on the category prototypes uploaded by each client. Step 512, two-factor similarity calculation, for each category and any two clients and Calculate the category-level two-factor similarity that takes into account both prototype orientation and sample size; Step 513, mask normalization, define the client. In category The effective set of collaborators on the client, i.e. and All holding categories The samples; apply a temperature-scaled mask to the effective collaborators to obtain class-by-class aggregation weights; Step 514: Weighted aggregation by category. The server performs a weighted summation of the global decision-makers uploaded by each client according to the aforementioned weighted aggregation categories, generating the client's... A customized global decision-maker.

9. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 8, characterized in that, The client updates the local model as follows: The server saves the updated global feature extractor parameters and the global decision-maker parameters customized for each client to the parameter repository according to the communication round number; Based on the client selection results in the next round, the server distributes the updated global feature extractor parameters, the global decision-maker parameters customized for each client, and the training round information to the corresponding clients; The client updates its local backbone network with the updated global feature extractor parameters, updates its local customized global decision-maker with the global decision-maker parameters customized for each client, and combines it with the locally retained personalized decision-maker to obtain the updated local model.

10. The personalized federated learning model training method for mitigating feature and knowledge misalignment according to claim 9, characterized in that, Image processing specifically includes: Step 71: The client performs preprocessing on the image to be processed, including normalization, resizing, and format conversion. Step 72: Input the preprocessed image into the updated local backbone network to obtain the image feature representation; Step 73: Input the image feature representation into the global decision-maker and the local personalized decision-maker respectively to obtain the global prediction and personalized prediction, and then calculate the result according to the fusion coefficient. Weighted fusion.

Images