A multi-modal federated learning method and system for a modal incomplete scene

Abstract

The application discloses a kind of multi-modal federated learning and system for modal incomplete scene, specifically includes: the client with consistent mode is grouped into a cluster, and corresponding model is distributed according to cluster category;Client trains local model and executes prototype contrast learning, and local prototype set and local model are transmitted to central server;On the central server, cluster step aggregation is executed to derive multi-modal global model, single modal global model;On the central server, global prototype aggregation is executed to derive global prototype set;Central server distributes multi-modal global model, single modal global model and global prototype set to each local client, and iterative cycle until model converges.The scheme of the present application effectively solves the problem of knowledge sharing and collaborative learning between heterogeneous clients in the incomplete modal scene, and improves the effect of model training.

Description

Technical Field

[0001] This invention belongs to the field of federated learning technology in distributed training, specifically involving a multimodal federated learning system for scenarios with incomplete modalities, and proposes a novel multimodal federated framework for scenarios with incomplete modalities. Background Technology

[0002] The development of hospital information technology and artificial intelligence has made the full analysis and utilization of multi-source, heterogeneous, multimodal health and medical data to provide intelligent clinical decision support a current research hotspot. Researchers have conducted a series of multimodal learning studies using data of various modalities, such as medical images, clinical texts, physiological monitoring signals, and laboratory test results. Because medical data contains a large amount of sensitive information and comes from diverse sources, traditional centralized multimodal learning paradigms require aggregating data from different data sources onto a single server for model training, which can easily lead to privacy issues.

[0003] Federated learning is a distributed learning paradigm that allows different clients to collaboratively train models without directly sharing local data. Therefore, it has been widely adopted as a promising privacy-preserving solution. Replacing the unimodal client in traditional federated learning with a multimodal client represents an initial attempt to combine federated learning with multimodal approaches. However, most modality-federated learning methods assume that each client has complete data for all modalities. In reality, some clients may have missing modalities, leading to modality inconsistencies between clients. In fact, unimodal and multimodal clients can supplement each other's information during model training. Effectively combining data resources from these inconsistent clients is particularly important when real-world medical data is relatively scarce. However, existing methods often fail to effectively address information sharing between modally inconsistent clients, resulting in insufficient utilization of data on each client and significantly reducing the effectiveness of multimodal federated learning. For example, some methods rely on public multimodal datasets for knowledge transfer; however, due to the privacy of medical data, access to public multimodal datasets may be impossible. Other methods choose to use generative models to generate missing modalities to address the modality incompleteness problem. However, the modalities generated by the generator may introduce additional noise into the original data, negatively impacting performance. Therefore, the effectiveness of the above methods is significantly limited.

[0004] In summary, modal incompleteness increases the difficulty of collaborative training and information sharing among clients. Addressing modal incompleteness and effectively integrating data resources across heterogeneous clients is crucial for improving the practicality of multimodal federated learning in the healthcare field. Summary of the Invention

[0005] In view of this, the present invention is committed to solving the technical problems caused by modality incompleteness, and thus discloses a multimodal federated learning framework for modality incomplete scenarios, and provides a multimodal federated learning method and system for modality incomplete scenarios, which promotes collaborative training and knowledge exchange between multimodal and unimodal clients without relying on public datasets or generating missing modalities.

[0006] Therefore, the present invention provides the following technical solution:

[0007] On the one hand, the present invention provides a multimodal federated learning method for modal incomplete scenarios, applied to a server, comprising the following steps:

[0008] S1: Divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster type. The clients include multimodal clients and unimodal clients.

[0009] S2: Receive the local prototype set and local model information of each modality transmitted by each client. The local model information and the local prototype set are obtained by the client through local training / computation.

[0010] S3: Execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; and use the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality.

[0011] The cluster-based step-by-step aggregation strategy is divided into two stages. The first stage involves model aggregation and updating within each cluster, while the second stage involves decoupling and aggregation between clusters, i.e., decoupling the model's feature encoder and classifier, and aggregating feature encoders of the same modality.

[0012] S4: Distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to each modality to the corresponding multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client. The distributed global prototype sets are used to participate in the client's update of the local prototype set.

[0013] The iteration loop S2-S4 continues until the iteration termination condition is met, and each multimodal client and single-modal client obtains the final model.

[0014] Optionally, the server executes a cluster-based step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model. In the first phase, the server performs model aggregation and updates within each cluster.

[0015]

[0016] In the formula, φ r+1 (·;w), φ r+1 (·;W) represents the single-modal global model and the multimodal global model obtained in the r-th iteration; This represents the current model trained locally by the 1st and N1st unimodal clients after the unimodal global model obtained in the previous round r-1 is distributed to the clients, where N1 is the number of unimodal clients in the corresponding cluster; w and W are the model parameters of the unimodal client and the multimodal client, respectively. This represents the current model trained locally by the 1st and M1th multimodal clients after the multimodal global model obtained in the previous round r-1 is distributed to the clients, where M1 is the number of multimodal clients in the corresponding cluster;

[0017] The second phase involves decoupling and aggregating servers across the cluster, represented as follows:

[0018]

[0019] In the formula, The feature encoder i represents the single-modal global model in the r-th iteration. The feature encoder i represents the multimodal global model. Represents the global feature encoder i, η obtained from the (r+1)th round of aggregation. i Represents dynamically adjusted aggregate weights, where i corresponds to the modality category.

[0020] Further, optionally, the process in step 3 of using the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality can be represented as follows:

[0021] Receive local prototype sets P from each client local And divide the local prototype set for each modality according to the modality category;

[0022] Then, average aggregation is performed according to modality category to obtain the global prototype set for each modality category.

[0023] Secondly, this invention provides a multimodal federated learning method for modal incomplete scenarios, applicable to systems containing clients and servers, comprising the following steps:

[0024] Step 1: Divide clients with consistent modalities into a cluster, and distribute the corresponding multimodal cluster model or unimodal cluster model to each client according to the cluster type. The clients include multimodal clients and unimodal clients.

[0025] Step 2: Each client trains the model using its local dataset, generates / updates its local prototype set, and transmits it to the server;

[0026] Step 3: The server executes a cluster-based step-by-step aggregation strategy to derive a multimodal global model and a unimodal global model; and uses the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality.

[0027] The cluster step-by-step aggregation strategy is divided into two stages. The first stage is to perform model aggregation and update within the cluster, and the second stage is to perform decoupled aggregation between clusters, that is, to decouple the feature encoder and classifier of the model and aggregate the feature encoders of the same modality.

[0028] Step 4: The server distributes the multimodal global model, the unimodal global model, and the global prototype sets corresponding to each modality to the matching multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client. The distributed global prototype sets participate in updating the local prototype set in Step 2.

[0029] The iteration loop continues from step 2 to step 4 until the iteration termination condition is met, and each multimodal client and single-modal client obtains the final model.

[0030] Further optionally, the update of the local prototype set in step 2 is achieved based on intra-modal prototype comparison and inter-modal prototype comparison;

[0031] Specifically, based on the distance metric between the local prototype and the global prototype, the intra-modal prototype contrast loss and inter-modal prototype contrast loss of multimodal clients and unimodal clients are calculated.

[0032] Each client constructs a total loss function by combining the corresponding intra-modal prototype contrast loss and inter-modal prototype contrast loss. The client's current model is then optimized with the minimum of the total loss function as the optimization objective. The client's local prototype set is then updated based on the updated model.

[0033] Further optionally, the formulas for the intra-modal prototype contrast loss and inter-modal prototype contrast loss of the multimodal client and the unimodal client are as follows;

[0034] The multimodal client includes mode A and mode B, and the loss formula is:

[0035]

[0036] In the formula, These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for the multimodal client m, respectively. Let A be the dataset of category k in the local prototype set A corresponding to modality A of multimodal client m. Let k be the dataset of category k in the local B prototype set corresponding to the B modality of multimodal client m. This is the dataset of category k in the global prototype set A corresponding to modality A. This is the dataset of category k in the global B prototype set corresponding to the B modality;

[0037] If the single-mode is represented as mode A, the loss formula is:

[0038]

[0039] In the formula, These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for a single-modal client n, respectively. This is a dataset of category k in the local prototype set A corresponding to the A modality of a multi-modal client n.

[0040] In three aspects, this aspect provides a server based on the aforementioned multimodal federated learning method applied to a server, including:

[0041] The cluster module is used to: divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. The multimodal cluster model and unimodal cluster model are both composed of feature encoders and classifiers.

[0042] The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein the local model information and the local prototype set are obtained by the client through local training / computation;

[0043] The cluster step-by-step aggregation module is used to execute cluster step-by-step aggregation strategies to derive multimodal global models and unimodal global models;

[0044] The global prototype aggregation module is used to perform global prototype aggregation by modality type using the local prototype sets sent by each client to obtain the global prototype set corresponding to each modality.

[0045] The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the matching multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client, and the distributed global prototype sets are used to participate in the client's update of the local prototype set.

[0046] In four aspects, the present invention provides a system based on the above-mentioned method applied to a system including a client and a server, the client including a multimodal client and a unimodal client;

[0047] The server includes a cluster module, a cluster step-by-step aggregation module, a global prototype aggregation module, and a transmission module.

[0048] The cluster module is used to: divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. The multimodal cluster model and the unimodal cluster model are both composed of a feature encoder and a classifier.

[0049] The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein each client trains the model using the local dataset, generates / updates the local prototype set, and transmits it to the server.

[0050] The cluster step-by-step aggregation module is used to execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model;

[0051] The global prototype aggregation module uses the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality.

[0052] The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the corresponding multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client, and the distributed global prototype sets are used to participate in the client's update of the local prototype set.

[0053] In five aspects, the present invention provides an electronic device, comprising:

[0054] One or more processors;

[0055] A memory that stores one or more computer programs;

[0056] The processor calls the computer program to implement:

[0057] The steps of a multimodal federated learning method for modal incomplete scenarios applied to a server, or the steps of a multimodal federated learning method for modal incomplete scenarios applied to a system containing clients and servers.

[0058] In six aspects, the present invention provides a computer-readable storage medium storing a computer program that is programmed or configured to implement a multimodal federated learning method for modal incomplete scenarios applied to a server or a multimodal federated learning method for modal incomplete scenarios applied to a system including a client and a server.

[0059] Beneficial effects

[0060] Compared with existing methods, the advantages of the present invention are:

[0061] (1) This invention proposes a multimodal federated learning approach for modal incomplete scenarios. Specifically, it proposes a cluster step-by-step aggregation strategy. In the first stage, model aggregation and updating are performed within each cluster. In the second stage, decoupled aggregation is performed between clusters to aggregate feature encoders of the same modality. That is, through model aggregation within the cluster and feature encoder aggregation between the clusters, communication between heterogeneous clients in modal incomplete scenarios is realized, effectively integrating single-peak information, solving the problem of knowledge sharing and collaborative learning between heterogeneous clients in incomplete modal scenarios, and improving the model training effect.

[0062] (2) A preferred embodiment of the present invention proposes a prototype comparison integration strategy, which utilizes prototypes to integrate semantic-level knowledge from heterogeneous clients, thereby enhancing the utilization of client semantic-level knowledge. Specifically, intra-modal and inter-modal prototype comparisons are developed. Intra-modal prototype comparison aims to minimize the difference in feature distribution within the same modality, while inter-modal prototype comparison aims to connect prototypes of different but semantically related modalities, supplementing modal information that may be missing in a single-modal client and enhancing inter-modal interaction.

[0063] (3) The invention is efficient, practical and easy to implement. Experimental results show that the invention can promote collaborative training and information sharing among heterogeneous clients. Attached Figure Description

[0064] Figure 1 This is a system block diagram of multimodal federated learning for scenarios with incomplete modalities, provided by the present invention. Detailed Implementation

[0065] This invention provides a multimodal federated learning framework for scenarios with incomplete modalities, which facilitates collaborative training and knowledge exchange between multimodal and unimodal clients without relying on public datasets or generating missing modalities. The invention will be further described below with reference to embodiments.

[0066] Example 1:

[0067] Step 1: Obtain information on multimodal clients and unimodal clients in the target scenario, form a cluster of clients with consistent modalities, and determine the model based on the cluster category.

[0068] This embodiment considers a modality-mixed distributed multimodal federation setup, with M multimodal clients (each client exhibiting both A and B modalities) and N unimodal A clients. There are no unimodal B clients; however, it should be understood that unimodal B clients may exist in other feasible embodiments. Each multimodal client has a local dataset. in, It is the j-th aligned data pair of the m-th multimodal client (1≤m≤M), with superscript... Represents the dataset The amount of data present in it These are the corresponding labels; it should be understood that in practical applications, all multimodal datasets are applicable, such as ADNI (Alzheimer's Disease Neuroimaging Initiative) multimodal imaging data (PET positron emission tomography + MRI), X-ray + text reports. The labels represent the category of the multimodal dataset. For example, ADNI multimodal imaging data (PET + MRI), with a binary classification task, would have the corresponding labels AD (Alzheimer's Disease) or CN (Cognitively Normal). and It could also be AD or CN.

[0069] Each unimodal A client has a local dataset. in, It is the j-th data from the n-th single-modal client (1≤n≤N). It corresponds to the tag. Represents the dataset The amount of data that exists in it.

[0070] Clients with consistent modalities are grouped into the same cluster. It should be understood that single-modal clients may not necessarily have completely consistent modalities, and if their modalities are not consistent, they will also be grouped into different clusters.

[0071] Both unimodal and multimodal cluster models consist of a feature encoder and a classifier. In this embodiment, the unimodal cluster model φ(·w) is composed of the following feature encoder f. A The classifier g(·) consists of g(·) and g(·), as shown below:

[0072] φ(·;w)=f A (·)∪g(·) (1)

[0073] To convey the correlation at the feature level, the multimodal cluster model in this embodiment employs a feature-level multimodal fusion network. This network merges feature representations extracted from single-modal encoder networks of different modalities before prediction. The feature fusion-based multimodal network can be represented as follows:

[0074] φ(·;W)=(f i (·),i∈(A,B))∪g(·) (2)

[0075] In the formula, i represents the data mode, and f i (·) represents the feature encoder for modality i, and W and w are the model parameters for the multimodal cluster model and the single-modal cluster model, respectively. The model parameters W and w are related to the feature encoder f. A The relationship between (·) and classifier g(·) and feature encoder f A The applications of (·) and classifier g(·) are existing technologies in this field, and therefore will not be described in detail.

[0076] It should be understood that in some embodiments, after the clusters are divided, a single-modal cluster model and a multimodal cluster model are assigned to the single-modal client and the multimodal client according to the cluster category. The specific model can be given in advance, or an optimal multimodal cluster model and a single-modal cluster model can be obtained according to Formula 3 below, or an optimal multimodal cluster model and a single-modal cluster model can be obtained by constructing an objective function with reference to Formula 3.

[0077] The objective set in this embodiment is to obtain an optimal multimodal cluster model and a single-modal cluster model through collaborative training between clients. This objective is defined as follows:

[0078]

[0079] In the formula, This represents the collection of all local datasets from multimodal and unimodal clients. and These represent the sizes of the local datasets for multimodal and unimodal clients, respectively. These represent the i-th data modality in multimodal and unimodal clients, respectively. and are the loss functions for the multimodal and unimodal clients, respectively, and W and w are the model parameters for the multimodal cluster model and the unimodal cluster model, respectively. These represent the expected values ​​of the loss function for data from multimodal clients and unimodal clients, respectively. In some embodiments, an optimal multimodal cluster model and a unimodal cluster model are obtained through collaborative training based on the objective set in Equation 3.

[0080] It should be noted that the formula before the plus sign in Formula 3 represents obtaining the optimal multimodal cluster model, while the formula after the plus sign represents the single modality. The single modality and multimodal cluster models influence each other and cooperate in training. During the training process, both the feature encoder and the classifier will be updated. In this embodiment, the model update method is to update through cluster distribution aggregation and prototype comparison integration strategies.

[0081] Step 2: Each client trains its model using its local dataset and generates / updates its local prototype set, which is then transferred to the server. The process of training the model using the local dataset is existing technology and will not be described in detail here.

[0082] Inspired by prototype learning, this invention leverages prototypes to improve the efficiency of knowledge exchange between clients. Considering the example-driven nature of prototype learning and its ability to facilitate semantic-level knowledge interaction, integrating prototype learning into multimodal federated learning can capture deep internal semantic correlations between multimodal and unimodal clients and effectively utilize intermodal association information. Local prototype updates require generating an initial local prototype set from local data. To capture intramodal and intermodal semantic correlations, this invention develops intramodal and intermodal prototype comparison. Intramodal prototype comparison aims to minimize feature distribution differences within the same modality; intermodal prototype comparison aims to connect prototypes of different but semantically related modalities, supplementing modal information that may be missing in unimodal clients and enhancing intermodal interaction.

[0083] The prototype is the class-averaged feature output by the feature extractor; each local client uses its local dataset. The feature encoder corresponding to the modality is input, and the obtained feature representations are averaged and aggregated to obtain a local prototype set for each client with modality i and category k (label category). Therefore, the multimodal client of this invention obtains a multimodal local prototype set, and the unimodal client obtains a unimodal local prototype set. The calculation equation for the local prototype set can be expressed as:

[0084]

[0085] In the formula, k is a class. It is a local dataset with modality i and category k, f i (·;x) is the feature encoder for mode i.

[0086] The present invention preferably updates the local prototype set based on intra-modal prototype comparison and inter-modal prototype comparison.

[0087] For multimodal clients, the intramodal prototype contrast loss and intermodal prototype contrast loss are as follows:

[0088]

[0089] In the formula, These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for the multimodal client m, respectively. Let A be the dataset of category k in the local prototype set A corresponding to modality A of multimodal client m. Let k be the dataset of category k in the local B prototype set corresponding to the B modality of multimodal client m. This is the dataset of category k in the global prototype set A corresponding to modality A. This is the dataset of category k in the global B prototype set corresponding to the B modality; the global prototype set is obtained through global prototype aggregation in the subsequent step four. In the first global iteration, the prototype contrast loss is not calculated.

[0090] For a single-modal client, the intra-modal prototype contrast loss and the inter-modal prototype contrast loss are as follows:

[0091]

[0092] In the formula, These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for a single-modal client n, respectively. Let d(*) be the dataset of class k in the local prototype set corresponding to modality A of a multi-monomodal client n. d(*) is the distance metric between the local prototype and the global prototype. The distance metric can take various forms, such as L1 distance, L2 distance, and bulldozer distance.

[0093] The ultimate goal of client c as defined in this embodiment It can be represented as:

[0094]

[0095] In the formula, α and β are important weights, l cls It is the local client classification loss.

[0096] It should be understood that the loss function for each client in this embodiment consists of three types: classification loss, intra-modal prototype comparison loss, and inter-modal prototype comparison loss. The goal of the latter two loss functions is to adjust the prototype. The classification loss also implicitly affects the calculation of the prototype. This embodiment uses gradient descent to update the local prototype set (updating model parameters, and then updating the local prototype set based on model updates). In other feasible embodiments, the objective function can be adjusted, with the overall idea being optimization through loss reduction.

[0097] Step 3: The server executes a cluster-based step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; and uses the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality.

[0098] Many federated learning frameworks addressing system heterogeneity (model heterogeneity, modality missing, communication latency) employ outcome-level fusion strategies, using knowledge distillation to aggregate locally predicted logits. While outcome-level fusion strategies solve the aggregation problem caused by system heterogeneity, locally predicted logits cannot convey feature-level correlations, significantly impacting model performance. Inspired by clustered federated learning, this invention proposes a feature-level federated aggregation strategy—the cluster step-by-step aggregation strategy. The first stage uses a traditional federated framework, performing only intra-cluster model aggregation and updates. The second stage performs inter-cluster decoupling aggregation, decoupling the model's feature encoder and classifier, aggregating feature encoders of the same modality, and performing feature-level fusion. The cluster step-by-step aggregation strategy effectively addresses the model heterogeneity problem caused by modality heterogeneity, and compared to outcome-level fusion strategies, feature-level fusion can utilize the correlations and interactions between features of different modalities. The defined cluster step-by-step aggregation strategy includes intra-cluster aggregation and inter-cluster aggregation.

[0099] Although the model architectures differ between clusters, they share the same model architecture within the same cluster. Therefore, a traditional federated framework can be used for weighted averaging within the cluster. This allows for full utilization of local data from similar clients, reducing data bias and correcting the training of local models in federated learning, ensuring that the local objectives of each client are consistent with the optimal value of the cluster's global model.

[0100] Intra-cluster aggregation process: The server performs weighted aggregation of models according to the cluster type to obtain a single-modal global model. and multimodal global model In round r:

[0101]

[0102] Here, FL(*) represents any existing FL algorithm designed to generate a single global model (e.g., FedAvg or FedProx). Therefore, the existing intra-cluster aggregation methods used can be integrated with many state-of-the-art FL algorithms to improve model performance in the case of non-independent and identically distributed models.

[0103] Intra-cluster aggregation effectively learns the data distribution characteristics within the cluster, but it doesn't fully utilize the effective information from other clusters. Inter-cluster aggregation decouples the feature extractor and classifier of the global model, aggregating feature extractors of the same modality to consistently learn unimodal information and enhance the representation of the same modality. Intra-cluster aggregation process: For the r-th round:

[0104]

[0105] In the formula, The feature encoder i represents the r-th round single-modal global model. The feature encoder i represents the multimodal global model. Represents the global feature encoder i, η obtained from the (r+1)th round of aggregation. i The dynamically adjusted aggregation weight reflects the proportion of data in modality i within the total dataset. Its specific value is an empirical value; in this embodiment, it is set to 0.5. It should be understood that obtaining the new... Then, it will be distributed to the corresponding client to replace the client's current feature encoder. After the client trains the model locally, it sends the model to the server, and the server then performs intra-cluster and inter-cluster aggregation.

[0106] This embodiment also obtains a more generalized global prototype set by aggregating local prototype sets from heterogeneous clients. This helps mitigate the knowledge bias between multimodal and unimodal clients. The central server receives the local prototype sets P from each client. local Then, it is divided into local A prototype set. and local B prototype set By performing average aggregation according to each category, we obtain the global prototype set A. and global B prototype set Then, it is distributed to each client.

[0107]

[0108] In the formula, k is a class, and M k N represents the set of multimodal clients with category k. k This represents the set of single-modal clients possessing category k. It should be understood that the above formula is derived using the modality setting of this embodiment as an example; in other feasible embodiments, when the modality differs, the above formula can be adaptively adjusted.

[0109] Step 4: The central server distributes the multimodal global model, the unimodal global model, and the global prototype set to each local client. Steps 2-4 iterate until the model converges, and then each multimodal client and unimodal client obtains the final model. The iteration termination condition or model convergence condition can be set according to application requirements, and this invention does not impose any constraints on it.

[0110] In summary, step 1 of the technical solution of this invention divides the cluster and determines the model structure of each cluster; step 2 is a local execution operation of the client, and steps 3 and 4 are both execution operations of the server. In step 3, the global model and global prototype are updated and sent to the client by step 4. Therefore, even if steps 3 and 4 do not involve local updates, the obtained global model and global prototype will affect the local model and local prototype.

[0111] Example 2:

[0112] This embodiment differs from Embodiment 1, using a multimodal federated learning method applied to a server for illustration. This embodiment provides a multimodal federated learning method for scenarios with incomplete modalities, comprising the following steps:

[0113] S1: Divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster type. The clients include multimodal clients and unimodal clients.

[0114] S2: Receive the local prototype set and local model information of each modality transmitted by each client. The local model information and the local prototype set are obtained by the client through local training / computation.

[0115] S3: Execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; and use the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality.

[0116] The cluster-based step-by-step aggregation strategy is divided into two stages. The first stage involves model aggregation and updating within each cluster, while the second stage involves decoupling and aggregation between clusters, i.e., decoupling the model's feature encoder and classifier, and aggregating feature encoders of the same modality.

[0117] S4: Distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to each modality to the corresponding multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client. The distributed global prototype sets are used to participate in the client's update of the local prototype set. S2-S4 are iterated until the iteration termination condition is met.

[0118] The specific implementation process of each step can be referred to the description in Example 1.

[0119] Example 3:

[0120] Based on Embodiment 1, Embodiment 3 provides a system, which is named FedMI in this embodiment, including: a client and a server, wherein the client includes a multimodal client and a unimodal client; the server includes a cluster module, a cluster step-by-step aggregation module, a global prototype aggregation module, and a transmission module;

[0121] The cluster module is used to divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. Both the multimodal cluster model and the unimodal cluster model are composed of a feature encoder and a classifier.

[0122] The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein each client trains the model using the local dataset, generates / updates the local prototype set, and transmits it to the server.

[0123] The cluster step-by-step aggregation module is used to execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model;

[0124] The global prototype aggregation module is used to perform global prototype aggregation by modality type using the local prototype sets transmitted by each client to obtain the global prototype set corresponding to each modality.

[0125] The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the corresponding multimodal clients and unimodal clients, so as to replace the current model of the multimodal clients and unimodal clients. The distributed global prototype sets are used to participate in the client's update of the local prototype set.

[0126] Example 4:

[0127] Based on Example 2, Example 4 provides a server that includes a cluster module, a transmission module, a cluster step-by-step aggregation module, and a global prototype aggregation module.

[0128] The cluster module is used to divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. The multimodal cluster model and unimodal cluster model are both composed of feature encoders and classifiers.

[0129] The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein the local model information and the local prototype set are obtained by the client through local training / computation;

[0130] The cluster step-by-step aggregation module is used to execute cluster step-by-step aggregation strategies to derive multimodal global models and unimodal global models;

[0131] The global prototype aggregation module is used to perform global prototype aggregation by modality type using the local prototype sets sent by each client to obtain the global prototype set corresponding to each modality.

[0132] The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the matching multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client, and the distributed global prototype sets are used to participate in the client's update of the local prototype set.

[0133] The specific implementation process of each module is described in the above method. This invention will not repeat the details here. The division of the functional modules is only for illustrative purposes. In some embodiments, some functional modules can be merged and some functional modules can be split. Each functional module can be implemented in software, hardware, or a combination of software and hardware. The software and hardware devices include, but are not limited to, general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.

[0134] In some embodiments, the present invention also provides an electronic device, including: one or more processors; and a memory storing one or more computer programs;

[0135] The processor calls the computer program to implement:

[0136] The steps of a multimodal federated learning method for modal incomplete scenarios applied to a server, or the steps of a multimodal federated learning method for modal incomplete scenarios applied to a system containing clients and servers.

[0137] Please refer to the explanation of the method above for the specific implementation process of each step.

[0138] It should be understood that, in the embodiments of the present invention, the processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0139] In some embodiments, the present invention also provides a computer-readable storage medium storing a computer program that is programmed or configured to implement a multimodal federated learning method for modal incomplete scenarios applied to a server or a multimodal federated learning method for modal incomplete scenarios applied to a system including a client and a server.

[0140] Please refer to the explanation of the method above for the specific implementation process of each step.

[0141] The readable storage medium is a computer-readable storage medium, which can be an internal storage unit of the hardware and software device described in any of the foregoing embodiments, such as the hard drive or memory of the controller. The readable storage medium can also be an external storage device of the controller, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, or Flash Card equipped on the controller. Further, the readable storage medium can include both internal storage units and external storage devices of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0142] Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application refers to flowchart illustrations and / or instructions executed by a processor of a method, apparatus (system), and computer program product according to embodiments of this application to create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowchart illustrations and / or one or more block diagrams. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0144] Experimental verification: The verification process was carried out using the research content of multimodal federated learning in a modal incomplete scenario as an example.

[0145] 1. Data Description

[0146] This experiment, based on Alzheimer's Disease Neuroimaging Initiative (ADNI) multimodal imaging data, was conducted to validate and evaluate the effectiveness of the system in Example 2. A brief description follows:

[0147] The Alzheimer's Disease Neuroimaging Initiative (ADNI) data repository contains multimodal imaging, clinical, and genetic data from over 2220 patients across four studies (ADNI1, ADNI2, ADNI GO, and ADNI3). Our study focuses on ADNI1. This experiment used T1-weighted magnetic resonance imaging (T1-MRI) and 18-fluorodeoxyglucose PET (FDG-PET) imaging data. For participants with multiple visits, we used images from the patient's first visit. Data preprocessing of MRI images included intensity normalization, craniotomy, and gray matter (GM) segmentation. Subsequently, T1-MRI images were registered in the MNI-152 standard brain template space. Preprocessing of FDG-PET images included intensity normalization and affine registration with the corresponding MRI images. Finally, all MRI and PET data were downsampled to 96×96×96. The MRI images for this experiment came from 326 participants, including 163 with AD and 163 with CN. MRI-PET paired data were obtained from 118 participants, including 59 with AD and 59 with CN.

[0148] 2. Experimental Setup

[0149] The unimodal cluster model was 3D-ResNet18, and the multimodal cluster model was dual-channel 3D-ResNet18. The traditional federated framework for intra-cluster aggregation was FedAvg. The learning rate for both multimodal and unimodal clusters was set to 0.0001. The batch size for multimodal clusters was set to 4, and for unimodal clusters to 8. The weights for both intramodal and intermodal prototype contrastive loss were set to 1. The multimodal model was trained for 30 epochs, while the unimodal model's parameters were frozen after 15 epochs. The number of clients for both multimodal and unimodal models was 3. This experiment explored two different local data distribution scenarios: Independent Identical Distribution (IID) and Non-Independent Identical Distribution (Non-IID). In the Non-IID scenario, the data distribution for unimodal and multimodal clustering was performed using a Dirichlet(α) distribution, where α∈{1,2}. Both the multimodal and unimodal datasets were partitioned into training and test sets in a 7:3 ratio.

[0150] 3. Comparison with other methods

[0151] This experiment selected several advanced models as baseline models for comparison:

[0152] (1) Single client: Each client trains independently using local data without any information exchange.

[0153] (2) Traditional federation frameworks: FedAvg, FedProx, FedBN. This invention uses the above federation frameworks as intra-cluster aggregation algorithms.

[0154] (3) Unimodal heterogeneous federated frameworks: FedDistall and FedProto. All unimodal heterogeneous federated frameworks extend to multimodal federated scenarios with incomplete modalities by operating on each modality separately. FedDistall achieves scenario extension by sharing the average logistic vector of labels for each modality. FedProto performs prototype learning and global prototype aggregation for each modality separately.

[0155] (4) Modality-incomplete multimodal federated frameworks: CreamFL and Harmony. CreamFL facilitates knowledge exchange between heterogeneous clients through a public multimodal dataset. Since this invention does not require a public dataset, five data points labeled AD and five labeled CN are randomly selected from each multimodal client as the public dataset. Harmony adapts to modality-incomplete scenarios by decoupling the multimodal network. Both CreamFL and Harmony only have a global multimodal model.

[0156] Table 1 shows the results of the comparative experiment.

[0157] Table 1. Performance comparison of different methods under IID and Non-IID

[0158]

[0159]

[0160] Considering the model heterogeneity of single-modal and multimodal clients in modality-incomplete scenarios, this invention compares the test set accuracy of global single-modal and global multimodal models among various methods. It can be seen that this invention achieves the best performance on both IID and Non-IID data. Since each client in a single-client model trains independently using only local data, its performance is the worst among all compared methods. The performance of the three traditional federated learning methods is significantly higher than that of single-client models, indicating that federated learning can effectively improve the overall model performance by fully utilizing data from different clients for collaborative training. The poor performance of FedDistall suggests that in modality-incomplete multimodal federated scenarios, especially in non-IID scenarios, outcome-level fusion strategies may not achieve effective knowledge transfer. FedProto reduces the negative impact of heterogeneous clients through prototype exchange, thereby improving model performance. Therefore, it is even better than multimodal federated learning methods. However, its effectiveness is limited in modality-incomplete multimodal federated learning scenarios due to the lack of intermodal interaction and gradient-level information exchange. Notably, multimodal federated learning methods did not demonstrate a significant performance advantage in our experiments. The primary reason for CreamFL's poor performance is likely its heavy reliance on public datasets with complete modalities and sufficient samples, while the public datasets used in our experiments were relatively small, thus diminishing its effectiveness. Harmony's poor performance may be due to its neglect of local semantic-level exchange between clients, relying solely on gradient-level information exchange. In contrast, our method, through clustered stepwise aggregation and prototype-based comparative ensemble, effectively achieves multimodal information fusion and sharing between clients even with modal inconsistencies, thereby significantly improving the final performance.

[0161] It should be emphasized that the examples described in this invention are illustrative rather than limiting. Therefore, this invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of this invention, without departing from the spirit and scope of this invention, whether modifications or substitutions, are also within the protection scope of this invention.

Claims

1. A multimodal federated learning method for scenarios with incomplete modalities, applied to a central server in a computer cluster, and enabling collaborative training and knowledge interaction between heterogeneous clients on multimodal and unimodal datasets, wherein the data types of the multimodal and unimodal datasets include images, text, and physiological monitoring signals, characterized in that: The method includes the following steps: S1: Divide clients with consistent modalities into a cluster, and distribute the corresponding multimodal cluster model or unimodal cluster model to each client according to the cluster type. Clients in the computer cluster are divided into multimodal clients and unimodal clients according to their modality. S2: Receive the local prototype set and local model information of each modality transmitted by each client. The local model information and the local prototype set are obtained by the client through local training / computation. S3: Execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; and use the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality. The cluster-based step-by-step aggregation strategy consists of two phases. The first phase involves model aggregation and updating within each cluster, while the second phase involves decoupling aggregation between clusters, i.e., decoupling the model's feature encoder and classifier, and aggregating feature encoders with the same modality. The first phase involves model aggregation and updating within each cluster. ； ； In the formula, , This represents the single-modal global model and the multimodal global model obtained in the r-th iteration. , This indicates that after the single-modal global model obtained in the previous round r-1 is distributed to the client, the 1st and 2nd... The current model trained locally by a single-modal client. The number of unimodal clients in the corresponding cluster; w and W are the model parameters of unimodal and multimodal clients, respectively; , This indicates that after the multimodal global model obtained in the previous round r-1 is distributed to the client, the 1st and 2nd... The current model trained locally by a multimodal client. The number of multimodal clients corresponds to the number of clients in the cluster; FL represents any federated learning approach designed to generate a single global model. The second phase involves decoupling and aggregation between clusters, represented as follows: ； In the formula, The feature encoder i represents the single-modal global model in the r-th iteration. The feature encoder i represents the multimodal global model. Represents the global feature encoder i obtained from the (r+1)th round of aggregation. Represents dynamically adjusted aggregate weights, where i corresponds to the modality category; S4: Distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to each modality to the corresponding multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client. The distributed global prototype sets are used to participate in the client's update of the local prototype set. The iteration loop S2-S4 continues until the iteration termination condition is met, and each multimodal client and single-modal client obtains the final model.

2. The method according to claim 1, characterized in that: Step 3, which utilizes the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality, can be represented as follows: Receive local prototype sets from each client. And divide the local prototype set for each modality according to the modality category; Then, average aggregation is performed according to modality category to obtain the global prototype set for each modality category.

3. A multimodal federated learning method for scenarios with incomplete modalities, applied to a computer cluster system including clients and a central server, and enabling collaborative training and knowledge interaction between heterogeneous clients for multimodal and unimodal datasets, wherein the data types of the multimodal and unimodal datasets include images, text, and physiological monitoring signals, characterized in that: The method includes the following steps: Step 1: Divide clients with consistent modalities into a cluster, and distribute the corresponding multimodal cluster model or unimodal cluster model to each client according to the cluster type. Clients in the computer cluster are divided into multimodal clients and unimodal clients according to their modality. Step 2: Each client trains the model using its local dataset, generates / updates its local prototype set, and transmits it to the central server; Step 3: The central server executes a cluster-based step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; and uses the local prototype sets transmitted by each client to perform global prototype aggregation according to modality type to obtain the global prototype set corresponding to each modality. The cluster step-by-step aggregation strategy is divided into two stages. The first stage is to perform model aggregation and update within the cluster, and the second stage is to perform decoupled aggregation between clusters, that is, to decouple the feature encoder and classifier of the model and aggregate the feature encoders of the same modality. The first phase involves model aggregation and updates within each cluster: ； ； In the formula, , This represents the single-modal global model and the multimodal global model obtained in the r-th iteration. , This indicates that after the single-modal global model obtained in the previous round r-1 is distributed to the client, the 1st and 2nd... The current model trained locally by a single-modal client. The number of unimodal clients in the corresponding cluster; w and W are the model parameters of unimodal and multimodal clients, respectively; , This indicates that after the multimodal global model obtained in the previous round r-1 is distributed to the client, the 1st and 2nd... The current model trained locally by a multimodal client. The number of multimodal clients corresponds to the number of clients in the cluster; FL represents any federated learning approach designed to generate a single global model. The second phase involves decoupling and aggregation between clusters, represented as follows: ； In the formula, The feature encoder i represents the single-modal global model in the r-th iteration. The feature encoder i represents the multimodal global model. Represents the global feature encoder i obtained from the (r+1)th round of aggregation. Represents dynamically adjusted aggregate weights, where i corresponds to the modality category; Step 4: The central server distributes the multimodal global model, the unimodal global model, and the global prototype sets corresponding to each modality to the matching multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client. The distributed global prototype sets participate in updating the local prototype sets in Step 2. The iteration loop continues from step 2 to step 4 until the iteration termination condition is met, and each multimodal client and single-modal client obtains the final model.

4. The method according to claim 3, characterized in that: The update of the local prototype set in step 2 is achieved based on intra-modal prototype comparison and inter-modal prototype comparison. Specifically, based on the distance metric between the local prototype and the global prototype, the intra-modal prototype contrast loss and inter-modal prototype contrast loss of multimodal clients and unimodal clients are calculated. Each client constructs a total loss function by combining the corresponding intra-modal prototype contrast loss and inter-modal prototype contrast loss. The client's current model is then optimized with the minimum of the total loss function as the optimization objective. The client's local prototype set is then updated based on the updated model.

5. The method according to claim 4, characterized in that: The formulas for intra-modal prototype contrast loss and inter-modal prototype contrast loss for the multimodal client and the unimodal client are as follows; The multimodal client includes mode A and mode B, and the loss formula is: ； ； In the formula, , These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for the multimodal client m, respectively. Let A be the dataset of category k in the local prototype set A corresponding to modality A of multimodal client m. Let k be the dataset of category k in the local B prototype set corresponding to the B modality of multimodal client m. This is the dataset of category k in the global prototype set A corresponding to modality A. d represents the dataset of category k in the global B prototype set corresponding to the B modality; d is the distance metric. If the single-mode is represented as mode A, the loss formula is: ； ； In the formula, , These represent the intra-modal prototype contrast loss and the inter-modal prototype contrast loss for a single-modal client n, respectively. This is a dataset of category k in the local prototype set A corresponding to the A modality of a multi-modal client n.

6. A central server based on the method of any one of claims 1-2, characterized in that: include: The cluster module is used to divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. Both the multimodal cluster model and the unimodal cluster model are composed of feature encoders and classifiers. The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein the local model information and the local prototype set are obtained by the client through local training / computation; The cluster step-by-step aggregation module is used to execute cluster step-by-step aggregation strategies to derive multimodal global models and unimodal global models; The global prototype aggregation module is used to perform global prototype aggregation by modality type using the local prototype sets sent by each client to obtain the global prototype set corresponding to each modality. The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the matching multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client, and the distributed global prototype sets are used to participate in the client's update of the local prototype set.

7. A computer cluster system based on the method of any one of claims 3-5, characterized in that: It includes a client and a central server, and the client includes multimodal clients and unimodal clients; The central server includes a cluster module, a cluster step-by-step aggregation module, a global prototype aggregation module, and a transmission module. The cluster module is used to: divide clients with consistent modalities into a cluster, and distribute corresponding multimodal cluster models and unimodal cluster models to each client according to the cluster category. The multimodal cluster model and the unimodal cluster model are both composed of a feature encoder and a classifier. The transmission module is used to receive the local prototype set and local model information transmitted by each client, wherein each client trains the model using the local dataset, generates / updates the local prototype set, and transmits it to the central server. The cluster step-by-step aggregation module is used to execute the cluster step-by-step aggregation strategy to derive the multimodal global model and the unimodal global model; The global prototype aggregation module is used to perform global prototype aggregation by modality type using the local prototype sets sent by each client to obtain the global prototype set corresponding to each modality. The transmission module is used to distribute the multimodal global model, the unimodal global model, and the global prototype sets corresponding to various modalities to the corresponding multimodal clients and unimodal clients. The distributed multimodal global model and unimodal global model are used to replace the current model of the multimodal client and the unimodal client, and the distributed global prototype sets are used to participate in the client's update of the local prototype set.

8. An electronic device, characterized in that: include: One or more processors; A memory that stores one or more computer programs; The processor calls the computer program to implement: The steps of any one of claims 1-2 applied to a multimodal federated learning method for modal incomplete scenarios on a central server, or the steps of any one of claims 3-5 applied to a computer cluster system including clients and a central server for a multimodal federated learning method for modal incomplete scenarios.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that is programmed or configured to implement, as claimed in any one of claims 1-2, a multimodal federated learning method for modally incomplete scenarios applied to a central server, or as claimed in any one of claims 3-5, a multimodal federated learning method for modally incomplete scenarios applied to a computer cluster system including clients and a central server.

Images