A federated learning method and system supporting model heterogeneity and data heterogeneity

Abstract

This invention provides a federated learning method and system supporting heterogeneous models and data, relating to the field of artificial intelligence, and aims to address the problem of decreased discriminative power of global knowledge in heterogeneous federated learning. The method includes: a server generating and distributing a global prototype using a trainable generator; a client training its local model based on the global prototype, performing intra-class clustering for each data feature after training to generate multiple local clustering prototypes and uploading them; and the server collecting the local clustering prototypes and training its generator using a composite loss such as focus contrastive loss and orthogonal regularization to proactively generate more discriminative global knowledge. This invention effectively improves classification accuracy and robustness, enhances the discriminative power of global knowledge, and ensures communication efficiency and privacy security through a two-way co-evolutionary mechanism of refined knowledge extraction by the client and proactive knowledge generation by the server.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and federated learning technology, specifically to a federated learning method and system that supports heterogeneous models and data. More particularly, it relates to a collaborative client-server prototype optimization method and system (FedSCP) for heterogeneous federated learning. Background Technology

[0002] Federated learning, as an emerging distributed machine learning paradigm, allows multiple clients to collaboratively train a global model without sharing local private data, thus effectively solving the problems of data silos and privacy protection.

[0003] Traditional federated learning methods, such as parameter averaging-based aggregation methods, typically require all participating clients to have the same model architecture so that the server can directly aggregate the model parameters or gradients uploaded by each client. However, in real-world applications, different clients often deploy models with varying structures due to their hardware conditions, business needs, or historical reasons, resulting in model heterogeneity. This makes traditional parameter averaging aggregation methods unsuitable.

[0004] To address the problem of model heterogeneity, existing technologies have proposed several solutions based on knowledge distillation or prototype aggregation. For example, some methods involve having clients upload "prototypes" of their local data features (typically the average features of each category). The server then aggregates these prototypes to generate a global prototype, which is subsequently distributed to the clients to guide subsequent local model training. However, these existing solutions still face challenges. First, when the data distributions of different clients exhibit significant non-independent and identically distributed characteristics, simply averaging local prototypes from different data distributions can lead to blurred features in the generated global prototype, failing to clearly represent any specific category. This results in unclear boundaries between different categories in the feature space, a problem known as similarity boundary collapse. Second, simply averaging all sample features of a category into a single prototype loses information about the potential multimodal distribution within that category, reducing the richness of the uploaded knowledge. Finally, the server typically acts only as a passive aggregator, performing simple processing on the inconsistent quality of knowledge uploaded by clients. It lacks the ability to actively optimize and refine global knowledge, making it susceptible to the influence of low-quality local prototypes, thus limiting the final performance of the global model.

[0005] Patent document CN115271099A discloses an adaptive personalized federated learning method that supports heterogeneous models, including the following steps: (1) The central server initializes the parameters of the global shared model; (2) The central server distributes the global shared model parameters to each participant in the federated learning. After receiving the global shared model parameters, the participants use the parameters to update their own global shared model; (3) The participants perform adaptive force learning to update the weights of their private models; (4) The participants use the newly obtained private training data to train both the private model and the global shared model simultaneously based on the stochastic gradient descent algorithm; (5) The participants upload the global shared model parameters after one round of iterative training to the central server; (6) After the central server collects enough global shared model parameters, it aggregates these model parameters to obtain new global shared model parameters, and then returns to step (2) to distribute the new global shared model parameters to each participant. This process is repeated until the loss function of all models converges or the maximum number of iterations is reached.

[0006] However, patent document CN115271099A still uses federated averaging (FedAvg) to aggregate the globally shared model. In scenarios with highly heterogeneous data, directly averaging the model parameters of different clients can easily lead to a decline in model performance, resulting in weak discriminative power in the learned feature representations and blurred decision boundaries between different categories. Furthermore, knowledge sharing between clients in this patent document relies entirely on the parameters of the global model. The complex distribution of local client data (such as multiple sub-patterns within a single category) is difficult to fully transmit to the server and other clients through a single model parameter, resulting in a relatively coarse knowledge representation.

[0007] Patent document CN116629376A discloses a federated learning aggregation method based on data-free distillation, including the following steps: Step 1: The server defines the generator and discriminator network structures and sends them to each client participating in the federated learning; the client defines its local classifier network structure; Step 2: Each client uses its local private data to perform several rounds of three-way adversarial training on the generator, discriminator, and local classifier; the trained generator and discriminator parameters are fed back to the server; Step 3: The server receives the generator and discriminator parameters fed back by all clients and calculates the global parameters. Step 1: The server sends global parameters, a predefined noise vector, and batch size to each client. Step 2: After receiving the global parameters, the client loads the global generator and global discriminator, inputs the noise vector into the global generator to obtain generated samples, inputs the generated samples into the local classifier to obtain soft labels, and feeds the soft labels back to the server. Step 3: The server receives the soft labels from all clients, calculates the global average soft label for each client, and sends it back to the corresponding client. Step 4: The client trains the local classifier based on the received global average soft label. Step 5: Repeat steps 2-6 until the local classifier converges.

[0008] However, the server in patent document CN116629376A performs a simple arithmetic average of the soft labels. This operation obscures the unique, discriminative knowledge of different clients, resulting in aggregated knowledge that cannot effectively guide clients to improve their model's discriminative ability, especially in scenarios with vastly different data distributions. Furthermore, in this patent document, each client trains only one generator, and the generated samples may only reflect a single modality of the local data. When the local data itself contains multiple sub-distributions (such as a disease having multiple subtypes), a simple generator struggles to capture this complexity. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the present invention aims to provide a federated learning method and system that supports heterogeneous models and data. It seeks to solve the problems of decreased global prototype discriminativeness and similarity boundary collapse caused by different client model structures and non-independent and identically distributed data in existing heterogeneous federated learning methods, as well as the problems of feature ambiguity and unclear inter-class boundaries caused by simple average aggregation in traditional methods.

[0010] According to the present invention, a federated learning method supporting model heterogeneity and data heterogeneity includes: The server uses a trainable global prototype generator to generate a global prototype for each data category and distributes the global prototype to one or more clients. The client receives the global prototype and guides its heterogeneous local model to train based on the global prototype; After local training is completed, the client performs intra-class clustering on the features of each class of samples in the local dataset to generate multiple local clustering prototypes that can reflect the multimodal distribution of the current class of data. The client uploads the generated local clustering prototype to the server; The server collects local clustering prototypes from one or more clients and uses these local clustering prototypes as training data to train and optimize the global prototype generator.

[0011] Preferably, the client trains the local model using a multi-objective composite loss function; the composite loss function includes at least one of the following: Classification loss is used to minimize the classification error of the local model. Alignment loss is used to align the feature vectors of local samples with the global prototype vectors of the corresponding classes in the feature space. Prototype knowledge distillation loss is used to transform the geometric relationships of global prototypes in the feature space into soft labels to guide the learning of the local classifier. Local distribution correction loss is used to correct long-tail or imbalanced distributions of local data through a self-supervised contrastive learning mechanism.

[0012] Preferably, the server trains the global prototype generator using a composite loss function, which includes at least one of the following: Focus on contrastive loss to enhance the model's ability to distinguish difficult negative samples that are hard to differentiate in the feature space; Orthogonal regularization is used to geometrically constrain global prototype vectors of different classes to remain orthogonal in the feature space.

[0013] Preferably, the focused contrast loss rescales the cosine similarity between prototypes by introducing a non-linear focusing factor, thereby enabling the model to pay more attention to prototypes of different categories with high feature similarity, so as to widen the inter-class boundaries in the crowded feature space.

[0014] Preferably, the orthogonal regularization forces the cosine similarity between global prototype vectors of different categories to approach zero, thereby constructing a decoupled global semantic space.

[0015] Preferably, the client's intra-class clustering uses the K-means clustering algorithm to cluster the sample features within each category, thereby generating a set of local clustering prototypes for each category.

[0016] According to the present invention, a federated learning system supporting model heterogeneity and data heterogeneity includes a server and multiple clients; The server includes a trainable global prototype generator and a model aggregation optimization module; The global prototype generator is used to generate a global prototype vector based on the category index; The model aggregation optimization module is used to receive local clustering prototypes uploaded by the client and train the global prototype generator according to a preset composite loss function. Each client includes: a local model, a local adaptive training module, and a local clustering module; The model structure of the local model differs from that of other clients; The local adaptive training module is used to train the local model using a multi-objective loss function based on the global prototype and local data sent by the server. The local clustering module is used to perform intra-class clustering on the features of the samples obtained after training, in order to generate a local clustering prototype.

[0017] Preferably, the multi-objective loss function of the client includes at least one of the following: classification loss, alignment loss, prototype knowledge distillation loss, and local distribution correction loss; The server's preset composite loss function includes at least one of the following: focus contrast loss and orthogonal regularization.

[0018] Preferably, the focused contrast loss rescales the cosine similarity between prototypes using a nonlinear focusing factor, thereby enhancing the model's ability to distinguish difficult negative samples in the feature space. The orthogonal regularization uses geometric constraints to keep the global prototype vectors of different categories orthogonal in the feature space.

[0019] Preferably, the client's local clustering module uses the K-means clustering algorithm to perform intra-class clustering.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly improves the classification accuracy and robustness of federated learning systems in scenarios with heterogeneous models and data. Through active generation on the server side and refined knowledge extraction on the client side, it effectively solves the feature ambiguity problem caused by traditional aggregation methods.

[0021] 2. This invention enhances the discriminative power of global knowledge. The focused contrast loss and orthogonal regularization adopted on the server side can effectively solve the class confusion problem caused by feature space congestion and construct a structured, decoupled ideal feature space, providing a clear and stable optimization target for all heterogeneous clients.

[0022] 3. This invention retains and utilizes the rich information of local data. The local clustering strategy adopted by the client can capture the multimodal distribution structure of local data more precisely than simple feature averaging calculation, providing the server with input with higher information density for optimizing global knowledge.

[0023] 4. This invention achieves efficient and privacy-preserving collaborative training. The entire federated learning process only needs to transmit lightweight prototype vectors, without transmitting large model parameters or gradients, which reduces communication overhead while protecting the client's local data privacy and model privacy. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall framework of a federated learning system that supports heterogeneous models and data, provided as an embodiment of the present invention. Detailed Implementation

[0025] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0026] Example 1 This embodiment provides a federated learning method that supports both model heterogeneity and data heterogeneity. For example... Figure 1 The diagram illustrates the overall framework of the federated learning system according to an embodiment of the present invention. The system includes a centralized server and multiple distributed clients. In this embodiment, client A and client B are used as examples, but it should be understood that the number of clients is not limited to two. The server and each client are connected via a communication network to jointly execute an iterative collaborative training process.

[0027] The core of this method lies in constructing a bidirectional co-evolutionary mechanism. Specifically, clients upload locally refined knowledge that reflects the multimodal distribution of data to the server; correspondingly, the server, based on the collected high-quality knowledge, actively learns and generates a more discriminative global knowledge system, which is then distributed to guide the local training of each client. This process gradually improves the performance of the entire federated learning system while protecting the data privacy of each client and ensuring compatibility with different model structures.

[0028] In a typical federated learning round, the method specifically includes the following steps: First, at the start of each round of federated learning, the server is responsible for generating and distributing global knowledge, aiming to provide a unified, high-quality reference for all participants. Specifically, a trainable global prototype generator is deployed within the server. The global prototype generator maps category indices to optimized global prototype vectors. Its implementation can be a neural network, for example, consisting of an embedding layer and one or more fully connected layers. For a task with C categories, the global prototype generator receives indices representing categories c (c∈{1, 2, …, C}) as input and outputs a d-dimensional vector, which is the global prototype of category c. At the start of each round, the server invokes its current global prototype generator to generate a corresponding global prototype vector for each category in the task. The global prototype vectors of all categories together constitute the global prototype set GP. Subsequently, the server distributes this global prototype set GP to all clients selected for training in this round via the communication network. Understandably, these global prototypes, as global knowledge, provide a common semantic alignment target for heterogeneous client models.

[0029] Next, upon receiving the global prototype GP from the server, it is used as a "semantic anchor." Each client (e.g., client A and client B) adaptively trains its local model using its private local dataset. The "heterogeneity" here is reflected in the fact that the local models of different clients can differ in network structure, parameter count, etc. For example, client A's local model might be a large residual network, while client B's local model might be a lightweight mobile network. Each local model typically includes a feature extractor (hereinafter also referred to as a feature extractor) for extracting features from the input data. ) and classifiers used for classification based on features (hereinafter also referred to as ).

[0030] To ensure the local model's learning process adapts to the characteristics of local data while aligning with global knowledge, a local adaptive training module is employed on the client side. This module utilizes a global prototype as guidance and optimizes the local model using a multi-objective loss function. As an optional implementation, this loss function may include one or more of the following components: Classification loss: This is the fundamental objective of model training, used to minimize the classification error of the local model on its local dataset. The standard cross-entropy loss function is typically used. The formula is as follows:

[0031] In the formula, This indicates the batch size for local training. Represents a classifier. This indicates that all categories are traversed for Softmax denominator normalization. and Representing the samples respectively Real labels, and samples The feature vector obtained by the feature extractor This represents the Logits vector output by the classifier. This represents the true class in the Logits vector. The value.

[0032] Alignment Loss: This loss term aims to align the features extracted by the local model with the global semantic space. Specifically, the sample feature vectors generated by the local feature extractor are aligned in direction with the global prototype of the corresponding category provided by the server. The formula is as follows:

[0033] In the formula, Indicates a sample, This indicates that the server-issued labels are consistent with the actual labels of the samples. The corresponding global prototype vector, This represents the sample feature vector generated by the client. This indicates the calculation of cosine similarity.

[0034] Prototype knowledge distillation loss (also referred to below as) To facilitate deeper transfer of global knowledge, this method introduces the concept of knowledge distillation, utilizing the global prototype structure for knowledge distillation. The relative positional relationships (such as distance and angle) between different prototypes in the global prototype set GP contain the similarity structure between categories. The goal of prototype knowledge distillation loss is to transform this structured geometric relationship into "soft labels" to guide the learning of the local classifier. Prototype knowledge distillation loss The formula is as follows:

[0035] In the formula, KL represents the distillation temperature coefficient, and KL represents the Kullback-Leibler divergence. Denotes the Softmax function, d Teacher model output.

[0036] Locally distributed corrected loss (hereinafter also referred to as Real-world local datasets often suffer from class imbalance or long-tail distribution. To mitigate the adverse effects of this problem on model training, this invention introduces a local distribution correction loss based on self-supervised contrastive learning to correct local data distribution bias. Within a training batch, for any sample's feature (anchor), features from other samples of the same class are considered positive samples, while features from samples of different classes are considered negative samples. The contrastive learning loss (such as InfoNCE loss) drives the model to bring the anchor feature closer to positive sample features in the feature space, while keeping it further away from negative sample features. This mechanism enhances the compactness of intra-class features and the separability of inter-class features, especially when dealing with sparsely represented tail classes, effectively preventing feature representation collapse and thus correcting the problem caused by imbalanced local data distribution.

[0037]

[0038] In the formula, This represents the set of categories owned by client k. This represents the current local average prototype. Representing categories Local average prototype.

[0039] The client's local adaptive training module sums the above losses in a weighted manner to form the overall optimization objective, and then uses optimization algorithms such as gradient descent to update the parameters of its local model.

[0040] After local training is complete, the client needs to refine the learned knowledge and upload it to the server in a compact and information-rich manner. Traditional prototype aggregation methods typically involve simply calculating the average of all sample features for each category as the local prototype for that category. However, this approach loses distribution information within the category. For example, a category may naturally have multiple sub-patterns (i.e., multimodal distributions), and simply calculating the average will yield a vague result that does not represent the center of any sub-pattern.

[0041] To address this issue, embodiments of the present invention employ a clustering-based knowledge extraction method. Specifically, the client uses the feature extractor in its recently updated local model to traverse all or part of the training samples in its local dataset to extract local sample features for all samples. These local sample features are then fed into a local clustering module. This local clustering module, after local training, clusters each category of features, generating multiple local clustering prototypes that reflect the data distribution. Specifically, it independently executes intra-cluster clustering algorithms for sample features within each category, generating a set of local clustering prototypes that can express the multimodal distribution of that category. Instead of simply calculating an average, K-means clustering can be applied to all sample features of category c. By setting the number of clusters K to an integer greater than 1 (e.g., K=3), the algorithm will find 3 cluster centers for category c. These 3 cluster center vectors together constitute the local cluster prototype set for category c. By summing up the local cluster prototype sets for all categories, we obtain the knowledge that the client will upload in this round, i.e., the local cluster prototype. In other words, this approach uses a set of prototypes to represent a category, effectively capturing and uploading multimodal structural information from local data, providing the server with richer and more refined "raw materials" than a single average prototype.

[0042] This invention abandons traditional model parameter aggregation and instead uses "prototypes" independent of model structure for communication. Furthermore, the client does not simply upload an average prototype, but rather uploads a set of prototypes reflecting the multimodal distribution of the data through local clustering, thus preserving rich data information.

[0043] After this, the client will use the local clustering prototype generated in the previous step. Uploaded to the server. It's important to note that throughout this process, the client does not upload any raw data (such as images or text) or its model parameters; it only uploads lightweight prototype vectors. This not only reduces communication overhead but also fundamentally guarantees the client's data and model privacy, adapting to scenarios with heterogeneous models.

[0044] After collecting the local clustering prototypes uploaded by all participating clients in this round, the server enters its core active optimization phase. Unlike passively weighting the received prototypes, the server in this invention uses these local clustering prototypes from different clients, which may vary in quality, as training data to actively train and optimize its own global prototype generator. During training, the focus-contrast loss is used to maximize the inter-class distance, and orthogonal regularization is used to decouple the directions of different classes, generating a new round of global prototypes with a better structure. This training process is handled by the model aggregation and optimization module on the server. This module also uses a specially designed composite loss function to guide the optimization of the global prototype generator. This composite loss function may include one or more of the following components: Focus on contrast loss (also referred to below) The core idea is to teach the global prototype generator to generate more discriminative global prototypes. In the feature space, prototypes of different classes should ideally be mutually exclusive. Traditional contrastive loss treats all negative samples (i.e., prototypes of different classes) equally. However, in a crowded feature space, the real challenge comes from those "hard negative samples" that are very similar to positive samples.

[0045]

[0046] In the formula, Represents the similarity function. This represents the local prototype vector corresponding to all categories (including positive and negative classes), and M represents the total number of clients. This represents the sum of a specific local prototype vector in the local cluster prototype set. This represents the temperature coefficient for comparative learning.

[0047] Focus contrast loss introduces a nonlinear focusing factor. The cosine similarity between prototypes is rescaled. Specifically, the similarity function can be modified as follows:

[0048] When the original similarity sim(a,b) of two prototypes a and b from different categories is already high, after... The amplification by powers increases the contribution to the loss dramatically. This forces the optimization process to focus on separating the most easily confused difficult negative sample pairs, thereby creating clearer intervals at the class boundaries and effectively solving the problem of similarity boundary collapse.

[0049] Orthogonal regularization (hereinafter also referred to as) To construct a more structurally sound global semantic space, geometric constraints can be introduced, which force global prototype vectors of different categories to remain as orthogonal as possible in the feature space (i.e., their cosine similarity approaches 0). The orthogonal regularization formula is as follows:

[0050] In the formula, C represents the total number of categories. and These represent the global prototype vectors of the i-th and j-th classes, respectively.

[0051] This not only maximizes inter-class differences but also constructs a decoupled semantic space. By forcing prototypes of different categories to be geometrically decoupled (orthogonal), it provides a structurally stable optimization objective for heterogeneous clients.

[0052] The model aggregation optimization module takes all the collected local cluster prototypes as input, calculates the above-mentioned composite loss, and updates the internal parameters of the global prototype generator through the backpropagation algorithm.

[0053] Finally, after proactive optimization on the server side, the global prototype generator is updated, enabling it to generate a new set of global prototypes that are more discriminative and structured than the previous round. The server then redistributes this new set of global prototypes to the next round of selected clients, thus initiating a new cycle of global guidance, local training, knowledge extraction, and global optimization. This process repeats continuously until the entire federated learning system converges or reaches the preset number of training rounds.

[0054] Through the steps described above, the method provided in this embodiment can, in a completely heterogeneous environment, highly aggregate knowledge from all parties through bidirectional co-evolution between the client and server, ultimately training a global knowledge model (represented by an optimized global prototype generator), while simultaneously improving the performance of the local models of all participants. The server-side shifts from "passive aggregation (averaging)" to "active generation." A trainable generator is designed, coupled with a focused contrastive loss, to automatically focus on and distinguish those easily confused difficult samples, ignoring a large number of simple negative samples, thereby reconstructing a clearly defined global feature space.

[0055] Example 2 To better understand the technical solution of the present invention, this embodiment will use a specific application scenario—a heterogeneous medical image-assisted diagnostic system across hospitals—to further illustrate the application of the method described in Embodiment 1.

[0056] Application scenario: Suppose that multiple hospitals (clients A, B, and C) want to jointly train an AI model to identify a certain disease (such as skin disease or fundus lesion). The task is to classify the input dermoscopic images to identify 10 categories of skin lesions, such as "melanoma", "basal cell carcinoma", and "benign nevus".

[0057] In this scenario, the main challenges include: First, according to relevant regulations, patient image data from hospitals is highly sensitive information and is strictly prohibited from being shared outside the hospital, posing challenges to data privacy and security. Second, the models used by different hospitals are heterogeneous. For example, Hospital A's research team uses the ResNet-50 model based on a large server; Hospital B deploys the more efficient EfficientNet-B0 model on its local workstations; and Hospital C uses the lightweight MobileNetV2 model. The model structures and parameter counts of these three hospitals differ significantly. Finally, the data from each party is heterogeneous, specifically manifested in the characteristic of non-independent and identically distributed data. For example, Hospital A, as a dermatology specialist hospital, has very rich case data (local dataset) of "melanoma" and "basal cell carcinoma," but fewer samples of "benign nevi"; while Hospitals B and C, as general hospitals, have relatively balanced data distribution, but the sample size of each category is far less than that of Hospital A.

[0058] The technical solution of this invention can be applied to constructing such a federated learning system, and its specific implementation process is as follows: In the initial rounds of federated learning (e.g., round 1), the cloud server first initializes a global prototype generator. This generator can be implemented as a small neural network, taking class indices from 1 to 10 as input and outputting a 512-dimensional feature vector. At the start of round 1, the generator randomly initializes its parameters and generates 10 initial, random global prototype vectors for the 10 skin disease categories, forming the global prototype set GP. The server then broadcasts these 10 prototype vectors to three hospitals, A, B, and C.

[0059] After receiving the global prototype, each hospital begins local training on its client side. Taking Hospital A as an example, its local adaptive training module begins training its ResNet-50 model as a heterogeneous local model. Its total loss function includes: classification loss, to ensure the model can accurately distinguish the 10 types of skin disease images locally; alignment loss, to align the "melanoma" image features extracted by ResNet-50 with the "melanoma" global prototype sent by the server in terms of orientation; prototype knowledge distillation loss, to learn the implicit class relationships between global prototypes. For example, if the initial "melanoma" and "basal cell carcinoma" prototypes are spatially close, the model will be guided to learn their similarity, making the classifier's decision boundary smoother; and local distribution correction loss. Since Hospital A has few "benign nevus" samples, this loss term forces the features of all "benign nevus" samples to cluster together and distance them from features of other categories (such as a large number of "melanoma" samples) through comparative learning within the training batch, preventing the feature space of "benign nevus" from being occupied by "melanoma". Hospitals B and C also used similar methods, employing their respective local datasets and global prototypes, to train their own EfficientNet-B0 and MobileNetV2 models using a multi-objective loss function.

[0060] After local training, each hospital refined and uploaded its local knowledge. Hospital A used its updated ResNet-50 feature extractor to extract features from all its "melanoma" samples. Due to its rich case volume, these features may exhibit a multimodal distribution, such as corresponding to "early nodular melanoma" and "late-stage ulcerative melanoma." Its local clustering module's K-means clustering algorithm (with K=2) was able to discover these two sub-clusters and generate two local cluster prototypes representing these two morphologies respectively. Similar operations were performed for other categories. Ultimately, Hospital A generated a total of approximately 20-30 local cluster prototypes P for 10 categories. kIn contrast, hospitals B and C, due to their smaller data volume, may have their local clustering modules generate only one prototype per category (i.e., K=1, equivalent to calculating the mean), or generate a small number of prototypes based on the data distribution. The three hospitals will then use their respective generated local clustering prototypes P... k Uploaded to the cloud server.

[0061] Subsequently, the server performs proactive optimization. Its model aggregation optimization module collects all local cluster prototypes and uses these prototypes as training data to optimize the global prototype generator. During this process, it focuses on contrastive loss. The process begins: Suppose a prototype of "early nodular melanoma" from Hospital A is very close to a prototype of "highly malignant benign nevus" from Hospital B in the feature space, forming a pair of difficult negative samples. Focused contrastive loss inflicts a large penalty on this pair of prototypes, forcing the global prototype generator to adjust its parameters so that the global prototypes of "melanoma" and "benign nevus" generated in the next round are spatially separated. Simultaneously, orthogonal regularization... This also plays a role; the loss term penalizes pairwise similarities between all 10 category global prototypes, causing them to tend towards mutual perpendicularity in 512-dimensional space. This is equivalent to constructing an ideal "skin disease semantic coordinate system," where each coordinate axis represents a core feature of a disease and is decoupled from features of other diseases. After optimization, the server's global prototype generator is updated.

[0062] In subsequent rounds, the server redistributes the newly generated, more discriminative, and structurally optimized global prototype set (GP) to the three hospitals. Each hospital's local model will then undergo a new round of training guided by a clearer global objective. For example, Hospital A's ResNet-50 will now learn to align features of both "early" and "late" melanoma to the optimized, more representative global prototype direction for "melanoma."

[0063] This process is repeated dozens of times, until the system finally obtains a very powerful global prototype generator. The generated global prototype can be regarded as a diagnostic standard based on expert consensus, which incorporates the collective wisdom of the three hospitals. Meanwhile, the local models of each hospital, despite their different structures, all show significantly improved diagnostic performance, far exceeding the results of training using only local data. The entire process achieves the construction of a robust and accurate cross-institutional medical image-assisted diagnostic system without sharing any case data.

[0064] Example 3 This invention also provides a federated learning system that supports model heterogeneity and data heterogeneity, for performing the methods described in the foregoing embodiments. (Refer to...) Figure 1The system can include a server and multiple clients in terms of hardware.

[0065] The server can be one or more physical servers, virtual machines, or a cloud server cluster, and its hardware includes at least one or more processors (e.g., central processing unit, graphics processing unit, or dedicated artificial intelligence chip) and memory (e.g., RAM, hard disk). The memory stores computer program instructions, which, when executed by the processor, enable the server to perform corresponding functions. Specifically, the server's functional modules include: a global prototype generator implemented as a trainable neural network model, used to receive category indices and generate a global prototype; a model aggregation optimization module, used to receive local clustering prototypes from multiple clients and update the parameters of the global prototype generator according to a preset composite loss function (e.g., including focus contrast loss and orthogonal regularization); a communication module, used to communicate with each client through a network interface, responsible for distributing the global prototype and receiving local clustering prototypes; and a storage module, used to persistently store the model state, historical prototypes, and other configuration information of the global prototype generator during the federated learning process.

[0066] The multiple clients can be local servers, workstations, edge computing devices, or even mobile devices belonging to hospitals, enterprises, or individual users. Each client also includes at least one processor and memory in its hardware. When the computer program instructions stored in its memory are executed by the processor, each client can perform corresponding functions. Specifically, the functional modules of the client include: a local storage module for securely storing its private, non-shared local dataset; a heterogeneous local model, whose specific network structure (such as the implementation of feature extractors and classifiers) may differ from other clients; a local adaptive training module for training its local model using a multi-objective composite loss function (e.g., including classification loss, alignment loss, etc.) based on the global prototype received from the server and the local dataset; a local clustering module, which is invoked after the local model training is completed to extract sample features and execute intra-class clustering algorithms to generate a local clustering prototype that reflects the multimodal distribution of the data; and a communication module for establishing a connection with the server, receiving the global prototype, and uploading the generated local clustering prototype.

[0067] Understandably, through the collaborative work of its various modules, this system solidifies the method flow described in Examples 1 and 2 into an automated processing system that can efficiently, safely, and accurately complete federated learning tasks in scenarios with both heterogeneous models and data.

[0068] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0069] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A federated learning method supporting model heterogeneity and data heterogeneity, characterized in that, include: The server uses a trainable global prototype generator to generate a global prototype for each data category and distributes the global prototype to one or more clients. The client receives the global prototype and uses the global prototype to guide the training of its heterogeneous local model. After local training is completed, the client performs intra-class clustering on the features of each class of samples in the local dataset to generate multiple local clustering prototypes that can reflect the multimodal distribution of the current class of data. The client uploads the generated local clustering prototype to the server; The server collects local clustering prototypes from one or more clients and uses these local clustering prototypes as training data to train and optimize the global prototype generator.

2. The federated learning method supporting model heterogeneity and data heterogeneity according to claim 1, characterized in that, The client trains the local model using a multi-objective composite loss function; the composite loss function includes at least one of the following: Classification loss, used to minimize the classification error of the local model; Alignment loss is used to align the feature vectors of local samples with the global prototype vectors of the corresponding classes in the feature space. Prototype knowledge distillation loss is used to transform the geometric relationships of global prototypes in the feature space into soft labels to guide the learning of the local classifier. Local distribution correction loss is used to correct long-tail or imbalanced distributions of local data through a self-supervised contrastive learning mechanism.

3. The federated learning method supporting model heterogeneity and data heterogeneity according to claim 1, characterized in that, The server trains the global prototype generator using a composite loss function, which includes at least one of the following: Focus on contrastive loss to enhance the model's ability to distinguish difficult negative samples that are hard to differentiate in the feature space; Orthogonal regularization is used to geometrically constrain global prototype vectors of different classes to remain orthogonal in the feature space.

4. The federated learning method supporting model heterogeneity and data heterogeneity according to claim 3, characterized in that, The focused contrast loss rescales the cosine similarity between prototypes by introducing a non-linear focusing factor, thereby enabling the model to pay more attention to prototypes of different categories with high feature similarity, in order to widen the inter-class boundaries in the crowded feature space.

5. The federated learning method supporting model heterogeneity and data heterogeneity according to claim 3, characterized in that, The orthogonal regularization forces the cosine similarity between global prototype vectors of different categories to approach zero, thereby constructing a decoupled global semantic space.

6. The federated learning method supporting model heterogeneity and data heterogeneity according to claim 1, characterized in that, The client uses the K-means clustering algorithm to cluster the sample features within each category, thereby generating a set of local cluster prototypes for each category.

7. A federated learning system supporting model heterogeneity and data heterogeneity, characterized in that, Includes one server and multiple clients; The server includes a trainable global prototype generator and a model aggregation optimization module; The global prototype generator is used to generate a global prototype vector based on the category index; The model aggregation optimization module is used to receive local clustering prototypes uploaded by the client and train the global prototype generator according to a preset composite loss function. Each client includes: a local model, a local adaptive training module, and a local clustering module; The model structure of the local model differs from that of other clients; The local adaptive training module is used to train the local model using a multi-objective loss function based on the global prototype and local data sent by the server. The local clustering module is used to perform intra-class clustering on the features of the samples obtained after training, in order to generate a local clustering prototype.

8. The federated learning system supporting model heterogeneity and data heterogeneity according to claim 7, characterized in that, The client's multi-objective loss function includes at least one of the following: classification loss, alignment loss, prototype knowledge distillation loss, and local distribution correction loss; The server's preset composite loss function includes at least one of the following: focus contrast loss and orthogonal regularization.

9. The federated learning system supporting model heterogeneity and data heterogeneity according to claim 8, characterized in that, The focused contrast loss rescales the cosine similarity between prototypes using a nonlinear focusing factor, thereby enhancing the model's ability to distinguish difficult negative samples in the feature space. The orthogonal regularization uses geometric constraints to keep the global prototype vectors of different categories orthogonal in the feature space.

10. The federated learning system supporting model heterogeneity and data heterogeneity according to claim 8, characterized in that, The client's local clustering module uses the K-means clustering algorithm to perform intra-class clustering.

Images