Model updating method and image classification method based on representation fusion and head training

Abstract

Based on the model updating method and image classification method characterized by fusion and head training, it relates to the field of federated learning and privacy protection technology, specifically relates to a local heterogeneous model updating method used in a federated learning framework. It solves the problems of unreliable prototype aggregation, non-uniform decision boundary and excessive dependence on public data set for knowledge transmission caused by geometric inconsistency of feature space in heterogeneous federated learning, while significantly reducing communication cost and model structure leakage risk. The method comprises the following steps: the server distributes global model parameters to the client; the client generates category prototypes based on the local feature extractor, and after feature fusion with the global feature extractor, uploads the updated parameters and prototypes; the server aggregates the parameters and prototypes, trains the global prediction head using the aggregated prototypes, and then issues the global prediction head; the client updates the local model according to the global prediction head and performs image classification. The application is suitable for medical image analysis, industrial visual inspection and edge intelligent devices.

Description

Technical Field

[0001] This invention relates to the fields of federated learning and privacy protection, specifically to a method for updating local heterogeneous models in a federated learning framework. Background Technology

[0002] Federated learning (FL) is a distributed machine learning paradigm that allows multiple participants to collaboratively train a model without sharing the original data, thereby leveraging distributed data resources while protecting data privacy. A typical FL framework achieves collaborative training through the following process: a server distributes the global model, clients train locally and upload updates, and the server aggregates the updates and continues iterating. This approach offers privacy advantages because it only exchanges model parameters without exposing the original data. However, this approach assumes all clients are identical, while in real-world cross-device scenarios, participating devices vary significantly in terms of data distribution, model composition, computing power, and bandwidth. In particular, statistical heterogeneity (non-independent and identically distributed) causes the global model to favor clients with abundant data, leading to performance degradation and convergence difficulties.

[0003] To mitigate the problem of statistical heterogeneity, personalized federated learning (pFL) has been proposed, aiming to adapt to local data characteristics while preserving collaborative advantages. However, most personalized methods still assume that clients use the same model architecture, and transmitting complete model parameters or gradients is not only costly in terms of communication but may also leak model structure information. To directly address the challenges of model heterogeneity and communication privacy, heterogeneous federated learning (HtFL) has emerged as a new paradigm. HtFL allows clients to use different model architectures and process heterogeneous data, and reduces communication costs and improves performance by sharing knowledge (rather than model parameters) among clients. However, existing methods still face some limitations: for example, knowledge distillation methods send predicted logits as global knowledge on a shared dataset, but these methods heavily rely on the quality and availability of the shared dataset; while data-free knowledge distillation methods introduce shared global isometric small models to interact with heterogeneous local models, or share lightweight class representations (i.e., prototypes) as global knowledge, they usually require clients to upload samples for weighted averaging, which may lead to data distribution leakage. More importantly, the category prototypes extracted by different model architectures are in heterogeneous feature spaces, with distribution offsets and geometric inconsistencies, making direct aggregation unreliable.

[0004] In summary, the core problem currently faced by heterogeneous federated learning model training is that knowledge distillation-based techniques rely too heavily on public datasets, and prototype sharing schemes without data knowledge distillation suffer from unreliable prototype aggregation due to geometric inconsistencies in heterogeneous feature spaces, ultimately leading to unstable global decision boundaries and limiting the improvement of accuracy in image classification applications. Summary of the Invention

[0005] This invention solves the problems of unreliable prototype aggregation, inconsistent decision boundaries, and over-reliance on public datasets for knowledge transfer caused by inconsistent feature space geometry in heterogeneous federated learning. At the same time, it significantly reduces communication costs and the risk of model structure leakage.

[0006] The model update method based on representation fusion and head training includes the following steps:

[0007] Step A1: The server initializes the global isomorphic model parameters, which include global feature extractor parameters and global prediction head parameters; the server then distributes the initialized global isomorphic model parameters to the selected clients.

[0008] Step A2: The client uses local training data to generate a category prototype based on the local feature extractor in the local heterogeneous model, and performs feature extraction through the local feature extractor and the initialized global feature extractor parameters respectively, and adaptively fuses the two types of features to obtain the fused global feature extractor parameters.

[0009] Step A3: The client uploads the parameters of the fused global feature extractor and the category prototype to the server;

[0010] Step A4: The server aggregates the parameters of the global feature extractor uploaded by all clients to obtain updated global feature extractor parameters, and aggregates the category prototypes to obtain global prototypes.

[0011] Step A5: The server uses the global prototype to train the global prediction head parameters to obtain the trained global prediction head parameters.

[0012] Step A6: The client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server, which are used to update the local heterogeneous model to obtain the updated local feature extractor and local prediction head.

[0013] In a further optimized approach, in step A1, the server initializes the global isomorphic model parameters as follows: the server generates initial parameters based on the global isomorphic model. and the initial parameters Stored on a server as a baseline for a globally isomorphic model;

[0014] The server distributes the initialized global isomorphic model parameters to the selected clients as follows:

[0015] First, the server randomly selects a set of participating clients from the entire client set according to the set participation ratio. :

[0016] ,

[0017] in Indicates the current communication round. The set participation ratio, Indicates from the complete collection The number of clients selected randomly is the participation ratio. Total number of episodes The value of the product rounded down;

[0018] Then, the server updates the global isomorphic model parameters in round (t-1). Distribute to the set Each client in the process.

[0019] In further optimization, step A2, the adaptive fusion includes the following steps:

[0020] Step A211, Feature Extraction: During communication, the client extracts sample pairs from its local training data. The input is fed into the global feature extractor and the local feature extractor to extract global features. and personalized features :

[0021]

[0022]

[0023] in Indicates the client sequence number. Represents a global feature extractor. Indicates a local feature extractor;

[0024] Step A212, dual projection mapping, the client will globalize the features and personalized features Input the global projection module respectively With local projection module ;

[0025] global features With personalized features Through global projection module respectively With local projection module Mapped to:

[0026]

[0027]

[0028] Global features Dimensions Subspace, personalized features Dimensions subspace global features Mapped to dimension The subspace, i.e. , to obtain projection features , Personalized features Mapped to dimension The subspace, i.e. , to obtain projection features ;

[0029] Step A213, stitching and blending, combining the projected features and By concatenating features along their respective dimensions, a fused feature is formed.

[0030]

[0031] in This indicates concatenation along the feature dimension. The characteristics after fusion ;

[0032] Step A214, dual-branch prediction, the client will use the fused features Input global prediction head This leads to a global prediction:

[0033]

[0034] Personalized features By local prediction head Processing yields personalized predictions:

[0035] ,

[0036] Calculate the global prediction separately and personalized prediction Compared with the true labels in the local training data Losses:

[0037] ,

[0038] ,

[0039] ,

[0040] For global prediction With real labels The loss, For personalized prediction With real labels The loss; for and The total client loss is obtained by summing them up;

[0041] Step A215, joint optimization, the total client loss Update the global isomorphic model using gradient descent with an independent learning rate. Local heterogeneous model Global projection module With local projection module :

[0042]

[0043] in , , , Corresponding to global isomorphic models Local heterogeneous model Global projection module With local projection module The learning rate is set. To ensure convergence stability.

[0044] In a further optimized approach, step A2, generating the category prototype includes the following steps:

[0045] Step A221, Local Feature Collection: The client iterates through the local training set to collect categories. The samples are input into the local feature extractor. This yields the corresponding local feature representation;

[0046] Step A222, category prototype generation: The client calculates the category based on the local feature representation. Corresponding category prototype :

[0047] ,

[0048] in Indicates client A subset of local data containing all tags as samples .

[0049] In a further optimized approach, in step A3, the client uploads the fused global feature extractor parameters and category prototype to the server as follows:

[0050] The client will update the global isomorphic model parameters. The client's identifier and training round information are uploaded to the server so that the server can record the set of clients participating in this training round. ;

[0051] The client uploads the category prototype to the server as follows: the client uploads the category prototype... and its category tags The complete set of all category prototypes Send it to the server, along with the necessary round number or client identifier.

[0052] In a further optimized approach, in step A4, the server aggregates the parameters of the global feature extractor uploaded by all clients into: the updated global isomorphic model parameters. The parameters are accumulated layer by layer and averaged according to the number of clients: for each layer of the globally isomorphic model Parameters of each client layer Add them up, then divide by The updated global isomorphic model parameters are obtained. Complete the update of the global isomorphic model parameters. The aggregation of.

[0053] In a further optimized approach, step A4 involves aggregating the category prototypes as follows:

[0054] First, for each category The category prototypes are mapped to a shared latent space using a neural feature transformer T to obtain the latent prototypes:

[0055] ,

[0056] in Indicates having a category A collection of clients, To share the dimensions of the potential space, To control the parameters of the neural feature converter T;

[0057] Then, the potential prototype is processed using a hybrid alignment loss. Close to its category centroid The hybrid alignment loss combines numerical proximity and directional consistency to finally obtain the aligned set of class centers. That is, the global prototype:

[0058]

[0059] in express Norm,

[0060]

[0061] Let be the cosine similarity, where Euclidean inner product, scalar Used to balance directional terms and terms that are close in value.

[0062] In a further optimized approach, in step A5, the server trains the global prediction head parameters using a global prototype as follows:

[0063] In the In round-robin communication, the server will transmit the global prototype. As training samples, their corresponding categories are... As a true label, the class center Input the current global prediction head Conduct training:

[0064] For each category , will class center The prediction output is obtained by inputting the global prediction head, and the prediction output is minimized by the class. loss function between To update the global prediction header parameters:

[0065]

[0066] in For learning rate, Indicates the first Categories after wheel alignment The potential center of mass.

[0067] In a further optimized approach, step A6, where the client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server, is as follows:

[0068] The server saves the updated global feature extractor parameters and the trained global prediction head parameters to the parameter repository according to the communication round number;

[0069] Based on the client selection results in the next round, the server distributes the updated global feature extractor parameters, the trained global prediction head parameters, and the training round information to the client.

[0070] The client updates the local heterogeneous model using the received updated global feature extractor parameters and the trained global prediction head parameters, thus obtaining the updated local heterogeneous model.

[0071] The image classification method based on representation fusion and head training uses a local heterogeneous model obtained by the model update method based on representation fusion and head training to perform image classification. The image classification includes the following steps:

[0072] Step B1: The client performs preprocessing on the image to be classified, including normalization, resizing, and format conversion, to obtain the preprocessed image.

[0073] Step B2: Input the preprocessed image into the updated local feature extractor to obtain the image feature representation;

[0074] Step B3: Input the image feature representation into the updated local prediction head to obtain the class probability or logits and output it.

[0075] The beneficial effects of this invention compared to the prior art are as follows:

[0076] This invention addresses the coexistence of model architecture differences and statistical heterogeneity in heterogeneous federated learning through an adaptive representation fusion mechanism. It effectively coordinates global semantic consistency with local model differences, thereby simplifying the system deployment process in heterogeneous scenarios and enhancing the adaptability and flexibility of the method in diverse hardware environments.

[0077] This invention combines prototype-guided global head training with a learnable feature transformer to optimize the problems of cross-client knowledge alignment and decision boundary unification. This method provides a solution that meets the needs of practical applications without the need for public datasets or complex generative models, and satisfies the strict requirements of modern heterogeneous federated learning environments for communication efficiency and privacy protection.

[0078] The dual-module collaborative design of this invention significantly reduces communication and computation overhead. By exchanging only lightweight projection parameters and category prototypes, it optimizes the time complexity of transmitting complete model parameters in traditional methods, reduces resource consumption, effectively controls costs, and improves the efficiency of knowledge sharing between heterogeneous clients.

[0079] The model update method and image classification method based on representation fusion and head training described in this invention are applicable to fields such as medical image analysis, industrial visual inspection, and edge intelligent devices. Attached Figure Description

[0080] Figure 1 This is a flowchart of the model update method based on representation fusion and head training as described in Implementation Method 1;

[0081] Figure 2 This is a schematic diagram of the neural feature converter T described in Embodiment Seven;

[0082] Figure 3 This is the data flow diagram of the model update method based on representation fusion and head training described in Implementation Method 10;

[0083] Figure 4 This is a schematic diagram showing the comparison of the accuracy of various methods tested on the Cifar100 dataset of this invention in a local heterogeneous environment. Detailed Implementation

[0084] Various embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0085] Implementation Method 1, see Figure 1 This embodiment describes a model update method based on representation fusion and head training. The model update method includes the following steps:

[0086] Step A1: The server initializes the global isomorphic model parameters, which include global feature extractor parameters and global prediction head parameters; the server then distributes the initialized global isomorphic model parameters to the selected clients.

[0087] Step A2: The client uses local training data to generate a category prototype based on the local feature extractor in the local heterogeneous model, and performs feature extraction through the local feature extractor and the initialized global feature extractor parameters respectively, and adaptively fuses the two types of features to obtain the fused global feature extractor parameters.

[0088] Step A3: The client uploads the parameters of the fused global feature extractor and the category prototype to the server;

[0089] Step A4: The server aggregates the parameters of the global feature extractor uploaded by all clients to obtain updated global feature extractor parameters, and aggregates the category prototypes to obtain global prototypes.

[0090] Step A5: The server uses the global prototype to train the global prediction head parameters to obtain the trained global prediction head parameters.

[0091] Step A6: The client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server, which are used to update the local heterogeneous model to obtain the updated local feature extractor and local prediction head.

[0092] Implementation Method 2: This implementation method is a further limitation of Implementation Method 1. It provides an example of step A1, in which the server initializes the global isomorphic model parameters.

[0093] The server initializes the global isomorphic model parameters as follows: the server generates initial parameters based on the global isomorphic model. and the initial parameters Stored on a server as a baseline for a globally isomorphic model;

[0094] The server distributes the initialized global isomorphic model parameters to the selected clients as follows:

[0095] First, the server randomly selects a set of clients from the entire client pool according to the set participation ratio. The selected client:

[0096] ,

[0097] in Indicates the current communication round. The set participation ratio, Indicates from the complete collection The number of clients selected randomly is the participation ratio. Total number of episodes The value of the product rounded down;

[0098] Then, the server updates the global isomorphic model parameters in round (t-1). Distribute to the set Each client in the process.

[0099] Implementation Method 3: This implementation method is a further limitation of Implementation Method 1, and provides an example of the adaptive fusion in step A2.

[0100] The adaptive fusion includes the following steps:

[0101] Step A211, Feature Extraction: During communication, the client extracts sample pairs from its local training data. The input is fed into the global feature extractor and the local feature extractor to extract global features. and personalized features :

[0102]

[0103]

[0104] in Indicates the client sequence number. Represents a global feature extractor. Indicates a local feature extractor;

[0105] This step lays the foundation for subsequent dual-projection mapping, enabling generalized and personalized information to be acquired simultaneously in the same batch.

[0106] Step A212, dual projection mapping, the client will globalize the features and personalized features Input the global projection module respectively With local projection module ;

[0107] global features With personalized features Through global projection module respectively With local projection module Mapped to:

[0108]

[0109]

[0110] Global features Dimensions Subspace, personalized features Dimensions subspace global features Mapped to dimension The subspace, i.e. , to obtain projection features , Personalized features Mapped to dimension The subspace, i.e. , to obtain projection features ;

[0111] Since the server cannot know the specific structure of each client's local model, the client can independently adjust the above dual projection module to adapt to its own architecture and feature dimensions. This design can therefore adapt to clients with diverse models and different feature dimensions, allowing them to achieve a balance between performance, computational efficiency and communication cost based on local resource conditions.

[0112] Step A213, stitching and blending, combining the projected features and By concatenating features along their respective dimensions, a fused feature is formed.

[0113]

[0114] in This indicates concatenation along the feature dimension. The characteristics after fusion ;

[0115] The fused features retain the unified semantic structure of the server-deployed model while embedding client-specific distribution information, providing a unified dimensional input for subsequent joint prediction and alignment.

[0116] Step A214, dual-branch prediction, the client will use the fused features Input global prediction head This leads to a global prediction:

[0117]

[0118] Personalized features By local prediction head Processing yields personalized predictions:

[0119] ,

[0120] Calculate the global prediction separately and personalized prediction Compared with the true labels in the local training data Losses:

[0121] ,

[0122] ,

[0123] ,

[0124] For global prediction With real labels The loss, For personalized prediction With real labels The loss; for and The sum of the client's total loss is assigned equal weights (each weight is 1) to achieve a globally isomorphic model. Local heterogeneous model and end-to-end joint optimization of the two projection modules;

[0125] Step A215, joint optimization, the total client loss Update the global isomorphic model using gradient descent with an independent learning rate. Local heterogeneous model Global projection module With local projection module :

[0126]

[0127] in , , , Corresponding to global isomorphic models Local heterogeneous model Global projection module With local projection module The learning rate is set. To ensure convergence stability;

[0128] Overall, this process achieves a unified representation that combines global and personalized approaches through adaptive projection and fusion learning. On the one hand, it enables global knowledge sharing across clients, and on the other hand, it retains local mode learning capabilities, thereby improving the overall learning performance on heterogeneous clients.

[0129] Implementation Method Four: This implementation method is a further limitation of Implementation Method One, and provides an example of step A2, which involves generating a category prototype.

[0130] The generation of category prototypes includes the following steps:

[0131] Step A221, Local Feature Collection: The client iterates through the local training set to collect categories. The samples are input into the local feature extractor. This yields the corresponding local feature representation;

[0132] Step A222, category prototype generation: The client calculates the category based on the local feature representation. Corresponding category prototype :

[0133] ,

[0134] in Indicates client A subset of local data containing all tags as samples .

[0135] Implementation Method 5: This implementation method further defines Implementation Method 1. For example, in step A3, the client uploads the fused global feature extractor parameters and category prototype to the server.

[0136] The client will update the global isomorphic model parameters. The client's identifier and training round information are uploaded to the server so that the server can record the set of clients participating in this training round. ;

[0137] This equal-weighting strategy ensures that each client contributes equally, effectively integrating multi-source knowledge while avoiding excessive influence of a single client on the global model.

[0138] The client uploads the category prototype to the server as follows: the client uploads the category prototype... and its category tags The complete set of all category prototypes Send to the server, along with the necessary round number or client identifier;

[0139] Since the client uploads category prototypes rather than the original data or complete model parameters, it is difficult to infer the original data from the category prototypes alone, thus significantly reducing the risk of privacy leakage and enabling effective knowledge sharing between clients.

[0140] Implementation Method Six: This implementation method further defines Implementation Method One, and provides an example of step A4, in which the server aggregates the parameters of the global feature extractor uploaded by all clients.

[0141] The server aggregates the parameters of the global feature extractor uploaded by all clients into: the updated global isomorphic model parameters. The parameters are accumulated layer by layer and averaged according to the number of clients: for each layer of the globally isomorphic model Parameters of each client layer Add them up, then divide by The updated global isomorphic model parameters are obtained. Complete the update of the global isomorphic model parameters. The aggregation of.

[0142] Implementation Method Seven, see below Figure 2 This embodiment is a further limitation of embodiment one, and provides an example of step A4, which involves aggregating the category prototypes.

[0143] The process of aggregating the category prototypes is as follows:

[0144] First, for each category The category prototypes are mapped to a shared latent space using a neural feature transformer T to obtain the latent prototypes:

[0145] ,

[0146] in Indicates having a category A collection of clients, To share the dimensions of the potential space, To control the parameters of the neural feature converter T;

[0147] like Figure 2 As shown, the neural feature converter ,Depend on The parameterization is achieved through two fully connected (FC) layers with a batch normalization (BN) layer in between, which maps the class prototypes to a shared latent space and aligns the class structure.

[0148] Then, the potential prototype is processed using a hybrid alignment loss. Close to its category centroid The hybrid alignment loss combines numerical proximity and directional consistency to finally obtain the aligned set of class centers. That is, the global prototype:

[0149]

[0150] in express Norm,

[0151]

[0152] For cosine similarity, Euclidean inner product, scalar Used to balance directional terms and terms that are close in value.

[0153] Implementation Method Eight: This implementation method is a further limitation of Implementation Method One. It provides an example of step A5, in which the server uses a global prototype to train the global prediction head parameters.

[0154] The server trains the global prediction head parameters using a global prototype as follows:

[0155] In the In round-robin communication, the server will transmit the global prototype. As training samples, their corresponding categories are... As a true label, the class center Input the current global prediction head Conduct training:

[0156] For each category , will class center The prediction output is obtained by inputting the global prediction head, and then the prediction output is minimized by the class. loss function between To update the global prediction header parameters:

[0157]

[0158] in For learning rate, Indicates the first Categories after wheel alignment The potential center of mass.

[0159] Implementation Method Nine: This implementation method further defines Implementation Method One, and provides an example of step A6, in which the client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server.

[0160] The server saves the updated global feature extractor parameters and the trained global prediction head parameters to the parameter repository according to the communication round number;

[0161] Based on the client selection results in the next round, the server distributes the updated global feature extractor parameters, the trained global prediction head parameters, and the training round information to the client.

[0162] The client updates the local heterogeneous model using the received updated global feature extractor parameters and the trained global prediction head parameters to obtain the updated local heterogeneous model.

[0163] Intuitively, the global prediction head can accelerate local convergence while retaining the personalized knowledge in the local feature extractor and integrating the global decision-making capabilities of the shared global prediction head, thus improving robustness to statistical heterogeneity.

[0164] Implementation Method 10, see below Figure 3 This embodiment describes the data flow of embodiments one through nine.

[0165] Figure 3 This describes the key collaborative steps in model update methods based on representation fusion and head training:

[0166] The server distributes global isomorphic model parameters to the selected clients. , as model parameters shared by the client;

[0167] Client Holding local heterogeneous models and local non-IID data This will be compared with the received global isomorphic model. Joint parameter training, using two specific projectors on the client side, namely... and Adaptive representation fusion is performed in an end-to-end manner;

[0168] The server aggregates the updated global isomorphic model parameters from the client. Generate new global model parameters This allows for the integration of knowledge from heterogeneous clients;

[0169] Each client uses its heterogeneous feature extractor To belong to category The feature representations generated from the samples are averaged to output the class prototype. The category prototype It was then uploaded to the server;

[0170] The server aggregates the received category prototypes. Through learnable feature transformers The category prototype Align to the shared latent space and output the aligned class centroid. To train global prediction head parameters Then the global prediction header parameters Distribute to clients;

[0171] The client uses the received global prediction header parameters Replace local prediction head .

[0172] Implementation Method Eleven: This implementation method provides an image classification method based on representation fusion and head training.

[0173] Image classification is performed using the local heterogeneous model obtained by the aforementioned model update method based on representation fusion and head training. The image classification includes the following steps:

[0174] Step B1: The client performs normalization, resizing, and format conversion preprocessing on the image to be classified to obtain the preprocessed image;

[0175] Step B2: Input the preprocessed image into the updated local feature extractor to obtain the image feature representation;

[0176] Step B3: Input the image feature representation into the updated local prediction head to obtain the class probability or logits and output it;

[0177] This process can be completed locally without uploading the original image or intermediate features to the server, thus protecting user privacy. Clients can choose to use the classification results locally only, or upload the aggregated statistical information (such as classification accuracy) as needed, to achieve a privacy-friendly image classification service.

[0178] Implementation Method Twelve: This implementation method evaluates the effectiveness of the invention on three widely used image classification benchmark datasets.

[0179] The image classification benchmark data are CIFAR-10, CIFAR-100, and Flowers102.

[0180] CIFAR-10 and CIFAR-100 contain 60,000 images with 10 and 100 categories respectively; Flowers102 contains 8,000 images with 102 categories.

[0181] The method proposed in this invention is compared with representative methods widely used in heterogeneous federated learning, including LG-FedAvg, FedGen, FedGH, FML, FedKD, FedTGP, and FedProto.

[0182] This implementation evaluates the method under two widely adopted statistical heterogeneity settings: a pathological setting and a practical setting.

[0183] In the pathological setting, each client is assigned data from a fixed number of categories (e.g., sampling 2 / 10 / 10 categories from CIFAR-10 / CIFAR-100 / Flowers102 respectively), resulting in a highly non-redundant and imbalanced distribution;

[0184] In the practical settings, the Dirichlet distribution (denoted as Dir=γ) is used to divide the data among clients, and the concentration parameter γ (default 0.1) controls the level of heterogeneity: the smaller γ is, the more obvious the distribution of non-independent and identically distributed data is;

[0185] To comprehensively evaluate the robustness and adaptability of this invention in different heterogeneous federated learning scenarios, experiments were conducted on five heterogeneous model families (HtMF): HtMF1 (containing only 4-layer CNN), HtMF3 (4-layer CNN, GoogleNet, and MobileNet_v2), HtMF5 (ResNet18 / 34 / 50 / 101 / 152), HtMF8 (combining HtMF3 and HtMF5), and HtMF10 (extending DenseNet-121 and EfficientNet-B0 based on HtMF8). These model families cover lightweight and complex convolutional models, allowing for a comprehensive evaluation of the invention's performance under varying model complexities.

[0186] The heterogeneous federated learning image classification method based on representation fusion and head training is named FedARF.

[0187] As shown in Table 1, FedARF outperforms all baselines on all three datasets within the HtMF8 isomorphic model family, achieving an accuracy improvement of up to 23.50%. Specifically, on Cifar-100, it outperforms FedProto by 4.25% in the pathological setting and 6.3% in the practical setting.

[0188] Table 1

[0189]

[0190] This advantage stems from FedARF's adaptive fusion of global and personalized representations on the client side, and its alignment of class prototypes on the server side to optimize the unified global prediction head, thereby imposing consistent decision boundaries while maintaining extremely low communication and computational overhead. Therefore, FedARF can handle non-independent and identically distributed data more effectively; even under severe label skew, the adaptive mechanism can mitigate overfitting to local data and maintain the discriminative power of the learned prototypes.

[0191] To evaluate the robustness of the models under heterogeneity, FedProto and FedARF were compared on Cifar-10, Cifar-100, and Flowers-102 under a utility setting, while the heterogeneity was gradually increased from HtMF1 to HtMF. 10 .

[0192] As shown in Table 2, FedProto's accuracy drops sharply with increasing heterogeneity, while FedARF maintains higher and more stable trajectories across all three datasets. This is even true in the highly heterogeneous regions HtMF8 and HtMF. 10 It maintains approximately 85%, 40%, and 45% accuracy on Cifar-10, Cifar-100, and Flowers-102, respectively. Under low heterogeneity (HtMF1), FedProto may still slightly outperform or be comparable to FedARF on Cifar-10 and Cifar-100, but once heterogeneity increases, FedProto's performance collapses, while FedARF only shows slight fluctuations and continues to lead.

[0193] Table 2

[0194]

[0195] This difference stems from FedProto's direct aggregation of prototypes in heterogeneous feature spaces, leading to semantic drift and distorted class centroids, which amplify with increasing heterogeneity. In contrast, FedARF utilizes adaptive representation fusion and a learnable prototype aligner to obtain low-variance class centroids in a shared latent space, providing stable and discriminative supervision for the global head. Therefore, FedARF mitigates cross-model distribution mismatch, maintains class separability, and enhances cross-client generalization, highlighting its robustness in heterogeneous federated learning.

[0196] Figure 4 This paper demonstrates the test accuracy on Cif ar100 under a practical setting with both statistical and model heterogeneity and the HtMF8 setup. FedARF rises rapidly in the first few rounds, then stabilizes at around 40% accuracy, consistently leading all competing methods throughout training. Model-based methods (such as LG-FedAvg) converge quickly but stabilize a few percentage points below the highest curve. Knowledge distillation-based FedGen shows a similar early rise, then stabilizes at a moderate level of accuracy. Prototype-based methods perform significantly differently: FedProto experiences a long warm-up period and consistently lags behind the leader by more than 10 percentage points; FedGH and FML also lag, indicating their difficulty in coordinating heterogeneous feature spaces; FedKD's accuracy drops significantly after the emergence of complex heterogeneity. In contrast, FedARF maintains a stable, top-tier trajectory after an initial rise, ultimately reaching the highest accuracy across the entire communication budget, highlighting its robustness to heterogeneous architectures and severe label skew.

[0197] This invention proposes FedARF, a fully heterogeneous federated learning framework that integrates a client-side dual lightweight projection branch with a server-side learnable prototype alignment module. The client adaptively fuses global and personalized features using the projection branch, transmitting only class prototypes. The server maps these class prototypes to a shared latent space, strengthening intra-class consistency and inter-class separation, and retrains the prototype-guided global head based on the aligned class centers. Through this coordination process, FedARF explicitly mitigates cross-model feature discrepancies, restores consistent class decision boundaries, and protects privacy while maintaining low communication overhead. Comprehensive empirical and ablation studies under progressively increasing model and statistical heterogeneity and label skew demonstrate that FedARF consistently outperforms prototype-based baselines, achieving a good accuracy-efficiency tradeoff and strong robustness.

[0198] The results show that the present invention solves the technical problem of inconsistent cross-model feature spaces and inconsistent decision boundaries caused by the coexistence of model architecture differences and statistical heterogeneity in heterogeneous federated learning scenarios, which in turn affects the accuracy of image classification. It improves the accuracy and robustness of heterogeneous federated learning models in image classification tasks.

Claims

1. An image classification method based on representation fusion and head training, characterized in that, Image classification is performed using a local heterogeneous model obtained through a model update method based on representation fusion and head training. The image classification includes the following steps: Step B1: The client performs preprocessing on the image to be classified, including normalization, resizing, and format conversion, to obtain the preprocessed image. Step B2: Input the preprocessed image into the updated local feature extractor to obtain the image feature representation; Step B3: Input the image feature representation into the updated local prediction head to obtain the class probability or logits and output it; The model update method includes the following steps: Step A1: The server initializes the global isomorphic model parameters, which include global feature extractor parameters and global prediction head parameters; the server then distributes the initialized global isomorphic model parameters to the selected clients. Step A2: The client uses local training data to generate a category prototype based on the local feature extractor in the local heterogeneous model, and performs feature extraction through the local feature extractor and the initialized global feature extractor parameters respectively, and adaptively fuses the two types of features to obtain the fused global feature extractor parameters. Step A3: The client uploads the parameters of the fused global feature extractor and the category prototype to the server; Step A4: The server aggregates the parameters of the global feature extractor uploaded by all clients to obtain updated global feature extractor parameters, and aggregates the category prototypes to obtain global prototypes. Step A5: The server uses the global prototype to train the global prediction head parameters to obtain the trained global prediction head parameters. Step A6: The client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server, which are used to update the local heterogeneous model to obtain the updated local feature extractor and local prediction head. In step A2, the adaptive fusion includes the following steps: Step A211, Feature Extraction: During communication, the client extracts sample pairs from its local training data. The input is fed into the global feature extractor and the local feature extractor to extract global features. and personalized features ; Step A212, dual projection mapping, the client will globalize the features and personalized features Input the global projection module respectively With local projection module , to obtain projection features ; Step A213, stitching and blending, combining the projected features and The features are combined according to their respective dimensions to form the fused features. ; Step A214, dual-branch prediction, calculate the total client loss. ; Step A215, joint optimization, the total client loss Update the global isomorphic model using gradient descent with an independent learning rate. Local heterogeneous model Global projection module With local projection module .

2. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A1, the server initializes the global isomorphic model parameters by generating initial parameters based on the global isomorphic model. and the initial parameters Stored on a server as a baseline for a globally isomorphic model; The server distributes the initialized global isomorphic model parameters to the selected clients as follows: First, the server randomly selects a set of participating clients from the entire client set according to the set participation ratio. : ， in Indicates the current communication round. The set participation ratio, Indicates from the complete collection The number of clients selected randomly is the participation ratio. Total number of episodes The value of the product rounded down; Then, the server updates the global isomorphic model parameters in round (t-1). Distribute to the set Each client in the process.

3. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A211, the extraction of global features and personalized features for: in Indicates the client sequence number. Represents a global feature extractor. Indicates a local feature extractor; In step A212, the projection features are obtained. for: Global features Dimensions Subspace, personalized features Dimensions subspace global features Mapped to dimension The subspace, i.e. , to obtain projection features , Personalized features Mapped to dimension The subspace, i.e. ; In step A213, the fused features for: in This indicates concatenation along the feature dimension. The characteristics after fusion ; In step A214, the dual-branch prediction is a combination of global prediction and personalized prediction. The client will use the fused features Input global prediction head To obtain a global prediction : Personalized features Input local prediction head Get personalized predictions : ， The calculation of the client's total loss for: ， ， ， For global prediction Compared with the true labels in the local training data The loss, For personalized prediction Compared with the true labels in the local training data The loss; for and The total client loss is obtained by summing them up; In step A215, the total loss of the client Update the global isomorphic model using gradient descent with an independent learning rate. Local heterogeneous model Global projection module With local projection module for: in , , , Corresponding to global isomorphic models Local heterogeneous model Global projection module With local projection module The learning rate is set. To ensure convergence stability.

4. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A2, generating the category prototype includes the following steps: Step A221, Local Feature Collection: The client iterates through the local training set to collect categories. The samples are input into the local feature extractor. This yields the corresponding local feature representation; Step A222, category prototype generation: The client calculates the category based on the local feature representation. Corresponding category prototype : ， in Indicates the client A subset of local data containing all tags as samples .

5. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A3, the client uploads the fused global feature extractor parameters and category prototype to the server as follows: The client will update the global isomorphic model parameters. The client's identifier and training round information are uploaded to the server so that the server can record the set of clients participating in this training round. ; The client uploads the category prototype to the server as follows: the client uploads the category prototype... and its category tags The complete set of all category prototypes Send it to the server, along with the necessary round number or client identifier.

6. The image classification method based on representation fusion and head training according to claim 2, characterized in that, In step A4, the server aggregates the parameters of the global feature extractor uploaded by all clients into: the updated global isomorphic model parameters. The parameters are accumulated layer by layer and averaged according to the number of clients: for each layer of the globally isomorphic model , and the corresponding layer parameters of each client Add them up, then divide by The updated global isomorphic model parameters are obtained. Complete the update of the global isomorphic model parameters. The aggregation of.

7. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A4, the aggregation of the category prototypes is as follows: First, for each category The category prototypes are mapped to a shared latent space using a neural feature transformer T to obtain the latent prototypes: , in Indicates having a category A collection of clients, To share the dimensions of the potential space, To control the parameters of the neural feature converter T; Then, the potential prototype is processed using a hybrid alignment loss. Close to its category centroid The hybrid alignment loss combines numerical proximity and directional consistency to finally obtain the aligned set of class centers. That is, the global prototype: in express Norm, Let be the cosine similarity, where Euclidean inner product, scalar Used to balance directional terms and terms that are close in value.

8. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A5, the server trains the global prediction head parameters using a global prototype as follows: In the In round-robin communication, the server will transmit the global prototype. As training samples, their corresponding categories are... As a true label, the class center Input the current global prediction head Conduct training: For each category , will class center The prediction output is obtained by inputting the global prediction head, and the prediction output is minimized by the class. loss function between To update the global prediction header parameters: in For learning rate, Indicates the first Categories after wheel alignment The potential center of mass.

9. The image classification method based on representation fusion and head training according to claim 1, characterized in that, In step A6, the process by which the client receives the updated global feature extractor parameters and the trained global prediction head parameters from the server is as follows: The server saves the updated global feature extractor parameters and the trained global prediction head parameters to the parameter repository according to the communication round number; Based on the client selection results in the next round, the server distributes the updated global feature extractor parameters, the trained global prediction head parameters, and the training round information to the client. The client uses the received updated global feature extractor parameters and the trained global prediction head parameters.

Images