Hybrid modal federated learning method and system based on modal decoupling and modal memory

Abstract

The invention provides a modal decoupling and modal memory-based hybrid modal federated learning method and system. The method comprises the following steps of: decomposing each mode into modal customization and modal sharing information by a modal decoupling strategy; in the hybrid modal federation updating process, the corresponding modal encoders respectively carry out customization and sharing updating; a modality memory mechanism stores client-shared modality prototypes that are dynamically refreshed from each modality custom encoder to compensate for incomplete modalities in each local client. According to the invention, the technical problem of heterogeneity disorder caused by the dispersibility of model learning in a dispersed mixed mode scene is solved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning of medical imaging data, and specifically to a hybrid-modal federated learning method and system based on modal decoupling and modal memory. Background Art

[0002] Medical MRI image segmentation is crucial in clinical diagnosis and treatment of brain tumors, osteoarthritis of joints, etc. Multi-parametric magnetic resonance imaging (MRI) images usually involve four complementary modalities: T1-weighted (T1), contrast-enhanced T1-weighted (T1c), T2-weighted (T2), and T2 fluid-attenuated inversion recovery (FLAIR). These modalities usually focus on different aspects of clinical diseases. For example, the first two mainly highlight the tumor core of brain tumors, while the latter two are good at highlighting the peritumoral edema of brain tumors.

[0003] In recent years, the rapid development of deep learning has significantly advanced the field of MRI image segmentation. However, most existing MRI image segmentation methods mainly belong to the centralized multi-modal paradigm (posing risks of data and privacy leakage), which is not applicable to non-centralized hybrid-modal medical scenarios. Specifically, due to the sensitivity and privacy of medical data, it is challenging for different customers (hospitals) to share local data, making traditional centralized model training unsuitable for non-centralized scenarios.

[0004] In many actual medical scenarios, due to various reasons (such as device diversity and modal missing), distributed hospitals usually hold different combinations of imaging modalities, and their data distributions in each modality are also different, which brings significant heterogeneity obstacles to decentralized model learning.

[0005] The existing patent application document of invention with publication number CN118379233A, "System and method for deep learning technology for continuous federated learning using a distributed data generation model", the existing method includes: establishing a continuous federated learning framework through multiple processors, the continuous federated learning framework includes a global model at a global site and corresponding local models derived from the global model at corresponding local sites. The method also includes retraining or readjusting the global model and the corresponding local models through the multiple processors without sharing the actual data set between the global site and the corresponding local sites but sharing a synthetic data set generated from the actual data set. However, the foregoing existing technology only targets a single modality and does not make a targeted design for the hybrid-modal scenario (that is, different users hold multiple modal data but different modal combinations), and cannot effectively handle data heterogeneity in the multi-modal scenario.

[0006] The existing invention patent application document "A Multi-modal Federated Learning Method and System for Modal Incomplete Scenarios" with the publication number CN118709805A. The existing method includes: Clients with consistent modalities are grouped into a cluster, and corresponding models are distributed according to the cluster category; The client trains the local model and performs prototype contrast learning, and transmits the local prototype set and the local model to the central server; On the central server, perform cluster step-by-step aggregation to derive a multi-modal global model and a single-modal global model; On the central server, perform global prototype aggregation to derive a global prototype set; The central server distributes the multi-modal global model, the single-modal global model, and the global prototype set to each local client, and iteratively loops until the model converges. However, the aforementioned existing technology still has problems of modal coupling and low static prototype compensation efficiency in modal incomplete scenarios.

[0007] In many actual medical scenarios, due to various reasons (such as device diversity and modal missing), distributed hospitals usually hold different combinations of imaging modalities, and their data distributions in each modality are also different, which brings significant heterogeneity obstacles to decentralized model learning. Therefore, there is an urgent need to develop a decentralized hybrid mode method applicable to decentralized hybrid mode scenarios.

[0008] In summary, the existing technology has the technical problem of heterogeneity obstacles caused by the decentralization of model learning in the decentralized hybrid mode scenario. Summary of the Invention

[0009] The technical problem to be solved by the present invention is: How to solve the technical problem of heterogeneity obstacles caused by the decentralization of model learning in the decentralized hybrid mode scenario.

[0010] The present invention solves the above technical problem by adopting the following technical solutions: The hybrid modal federated learning method based on modal decoupling and modal memory includes:

[0011] S1. Deploy the MDM-MixMFL framework and perform federated update operations. Through the decoupling strategy, perform adaptive federated aggregation operations on the data and modalities of distributed clients, and design a modal memory mechanism to store and refresh modal prototypes to compensate for missing modalities; Among them, the MDM-MixMFL framework includes: the modal decoupling of the hybrid modal federated learning MixMFL, and the overview pipeline of the memory MDM framework;

[0012] S2. Through modal customization operations, use a common encoder and a modal decoupler to decouple modal information; Among them, the modal decoupler includes: a cross-entropy loss modal decoupling branch, a triplet loss modal decoupling branch;

[0013] S3. Utilize the modal memory mechanism to compensate for the incomplete modalities in each local distributed client by memorizing, refreshing, and retrieving modal prototypes, and complete the missing modal information;

[0014] S4. Iterate S2 to S3, use the federated update algorithm for model training and testing to obtain an applicable model, process the given medical image, and obtain a segmentation mask.

[0015] The present invention performs MRI image segmentation based on a hybrid modal federated learning framework (MDM-MixMFL) of modal decoupling and memory (MDM), characterized by a modal decoupling strategy and a modal memory mechanism. Regarding the modal decoupling strategy, the present invention deploys multiple modal custom encoders and a modal shared encoder for each client (hospital), and decouples each modality into modal custom and modal shared components for customized and shared federated updates. This strategy achieves stable hybrid-mode modal fusion and data aggregation across distributed hybrid-mode clients (hospitals), and generates a personalized optimal segmentation model for each client.

[0016] In a more specific technical solution, in S1, no less than 2 modal custom encoders and modal shared encoders are deployed for each distributed client. The modal custom encoder is used to encode the corresponding modal information, and the modal shared encoder is used to encode the modal shared information. Federated updates are performed among the modal encoders of the same category.

[0017] The present invention proposes a new hybrid modal federated learning framework (MDM-MixMFL) of modal decoupling and memory, which can achieve stable hybrid-mode fusion and data aggregation among distributed hybrid-mode clients (hospitals), and generate a personalized segmentation model for each client.

[0018] In a more specific technical solution, the modal shared encoder is used to receive all modalities of the image, encode the invariant information of the modality, and perform federated updates according to all modalities in all clients;

[0019] During the training process, an additional auxiliary modal decoupler is attached, and the decoupling loss is used to guide the auxiliary modal decoupler for adaptive modal decomposition.

[0020] The modal decoupling strategy proposed by the present invention adaptively decouples each modality into modal custom and modal shared information for customized and shared parameter updates respectively. During the hybrid-modal federated update process, the corresponding modal encoders perform customized and shared updates respectively. This helps to stably and adaptively jointly aggregate heterogeneous data and modalities from distributed clients. It promotes the stable and adaptive federated aggregation of heterogeneous data and modalities from distributed clients.

[0021] In a more specific technical solution, modal prototypes are extracted from the modality-specific representations of the modal customization encoder;

[0022] The modal prototypes are stored and refreshed in the memory shared by the clients;

[0023] The modal prototypes are retrieved from the memory to compensate for the incomplete modalities in each distributed client based on the existing modalities;

[0024] The concatenated representations of the modal shared encoder, modal customization encoder, and modal memory are fed into the modal shared decoder in a preset order to predict the final segmentation mask.

[0025] In a more specific technical solution, in S2, the cross-entropy loss modal decoupling branch uses the set of output representations of all modal encoders as input and performs modal classification using the cross-entropy loss. Perform modal classification.

[0026] In a more specific technical solution, the cross-entropy loss modal decoupling branch encourages the modal customization encoder to generate judgment-applicable modal representations and obtain encoded modality-specific information;

[0027] With the help of the Gradient Reversal Layer (GRL), the modal shared encoder is encouraged to generate non-judgment-applicable representations and encode modal shared information.

[0028] In a more specific technical solution, in S2, the triplet loss modal decoupling branch uses the modal fusion representation and the global modal representation as input and performs an indirect modal encoder decoupling operation using the triplet loss. Perform an indirect modal encoder decoupling operation.

[0029] In a more specific technical solution, using the triplet loss, the information gain between the fusion representation Positive formed by any two modal shared representations and any one modal shared representation Anchor is minimized, and the information gain between the fusion representation Negative formed by any modal shared representation and the modality-specific representation and any one modal shared representation Anchor is maximized to perform an indirect modal encoder decoupling operation.

[0030] In a more specific technical solution, in S3, each modality's memory bank is globally shared by the distributed clients, and dynamic refresh and customization operations are performed on the modal prototypes from the corresponding local modalities;

[0031] During the storage and refresh process, the modal customization encoder in each local distributed client generates modality-specific representations for each sample in each local round; lightweight clustering is performed using the modality-specific representations to obtain n clustering centers;

[0032] The clustering centers are used as modal prototypes and written into the repository;

[0033] During model training and inference, a comparison operation is performed on the intermediate corresponding representations of existing modalities and the missing modality memories to obtain and read the missing modality prototype with the highest similarity, so as to complete the missing modality information.

[0034] For the modality memory mechanism, the present invention designs a modality memory module to memorize, refresh and retrieve modality prototypes to make up for the incomplete modalities in local clients. The modality memory mechanism proposed by the present invention buffers the modality prototypes dynamically refreshed from each modality-customized encoder to compensate for the incomplete modalities in each local client, which further facilitates modality aggregation and fusion in the process of hybrid modality federated learning.

[0035] Through the modality decoupling strategy and the dynamic modality memory mechanism, the present invention explicitly separates modality-specific and shared information and completes the missing modalities, so it is more suitable for distributed hybrid modality federated learning, and improves the adaptability and performance of the model in complex medical scenarios.

[0036] In a more specific technical solution, the hybrid modality federated learning system based on modality decoupling and modality memory includes:

[0037] A framework deployment module for deploying the MDM-MixMFL framework, performing federated update operations, adaptively federating and aggregating the data and modalities of distributed clients through a decoupling strategy, and designing a modality memory mechanism to store and refresh modality prototypes to make up for missing modalities; wherein, the MDM-MixMFL framework includes: the modality decoupling of the hybrid modality federated learning MixMFL, and an overview pipeline of the memory MDM framework;

[0038] A modality information decoupling module for decoupling modality information through modality customization operations, using a common encoder and a modality decoupler; wherein, the modality decoupler includes: a cross-entropy loss modality decoupling branch, a triplet loss modality decoupling branch, and the modality information decoupling module is connected to the framework deployment module;

[0039] A modality information incomplete module for compensating for the incomplete modalities in each local distributed client through a modality memory mechanism by memorizing, refreshing and retrieving modality prototypes, and completing the missing modality information. The modality information incomplete module is connected to the modality information decoupling module;

[0040] An iterative training module for iterating S2 to S3, training and testing the model using a federated update algorithm to obtain an applicable model, processing a given medical image to obtain a segmentation mask. The iterative training module is connected to the modality information incomplete module and the modality information decoupling module.

[0041] The present invention has the following advantages compared with the prior art: [[ID=2,6]]

[0042] The present invention performs MRI image segmentation based on a hybrid modal federated learning framework (MDM-MixMFL) of modal decoupling and memory (MDM), which is characterized by a modal decoupling strategy and a modal memory mechanism. Regarding the modal decoupling strategy, the present invention deploys multiple modal-customized encoders and a modal-shared encoder for each client (hospital), and decouples each modality into modal-customized and modal-shared components for customized and shared federated updates. This strategy achieves stable hybrid-mode modal fusion and data aggregation across distributed hybrid-mode clients (hospitals), and generates a personalized optimal segmentation model for each client.

[0043] The present invention proposes a new hybrid modal federated learning framework (MDM-MixMFL) of modal decoupling and memory, which can achieve stable hybrid-mode fusion and data aggregation among distributed hybrid-mode clients (hospitals), and generate a personalized segmentation model for each client.

[0044] The modal decoupling strategy proposed by the present invention adaptively decouples each modality into modal-customized and modal-shared information for customized and shared parameter updates respectively. During the hybrid-modal federated update process, the corresponding modal encoders perform customized and shared updates respectively. This helps to stably and adaptively jointly aggregate heterogeneous data and modalities from distributed clients. It promotes the stable and adaptive federated aggregation of heterogeneous data and modalities from distributed clients.

[0045] Through the explicit modal decoupling strategy and the dynamic memory mechanism, the present invention can more accurately separate modal features, dynamically complete missing modalities, and optimize the federated learning process for medical image segmentation, thereby improving the stability and segmentation accuracy of the model under heterogeneous data.

[0046] Regarding the modal memory mechanism, the present invention designs a modal memory module to memorize, refresh, and retrieve modal prototypes to make up for incomplete modalities in local clients. The modal memory mechanism proposed by the present invention buffers the modal prototypes dynamically refreshed from each modal-customized encoder to compensate for incomplete modalities in each local client, which further facilitates modal aggregation and fusion in the hybrid-modal federated learning process.

[0047] Through the modal decoupling strategy and the dynamic modal memory mechanism, the present invention explicitly separates modal-specific and shared information, and completes missing modalities, thus being more suitable for federated learning of distributed hybrid modalities, and improving the adaptability and performance of the model in complex medical scenarios.

[0048] The present invention solves the technical problem of the dispersion of model learning in the prior art for the decentralized hybrid-mode scenario, resulting in heterogeneity barriers. Description of the Drawings

[0049] Figure 1Schematic diagram of the hybrid modal federated learning method based on modal decoupling and modal memory in Embodiment 1 of the present invention;

[0050] Figure 2 Schematic diagram of the overall network framework of the hybrid modal federated learning method based on modal decoupling and modal memory in Embodiment 1 of the present invention;

[0051] Figure 3 Schematic diagram of the structure of the modal decoupler in Embodiment 1 of the present invention;

[0052] Figure 4 Schematic diagram of the memory bank framework and update mechanism in Embodiment 1 of the present invention. Detailed implementation manners

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0054] Embodiment 1

[0055] As Figure 1 shown, the hybrid modal federated learning method provided by the present invention includes the following basic steps:

[0056] S1. Deployment of the MDM-MixMFL framework;

[0057] As Figure 2 shown, in this embodiment, referring to Figure 2 the federated update part, an overview pipeline of the proposed modal decoupling and memory (MDM) framework for handling hybrid modal federated learning (MixMFL) is shown. To effectively utilize complementary but different modal information in the distributed hybrid modal scenario, multiple modal-customized encoders and one modal-shared encoder are deployed for each client. The modal-customized encoders encode the corresponding modal information, and the modal-shared encoder encodes the modal-shared information. Federated updates are performed among the modal encoders of the same class. Such as Figure 2 the red, green, and blue lines in the federated update part. The joint process of parameter transmission, aggregation, and update of the modal-customized encoders is similar to the process of conventional FL in the single-modal scenario.

[0058] In contrast, the modal-shared encoder receives all modalities of the image and is designed to encode modal-invariant information. It performs federated updates based on all modalities in all clients, for example Figure 2The gray line in the federal update part.

[0059] Federated Learning (FL), as an innovative distributed machine learning paradigm, realizes multi-node collaborative modeling through a centralized coordination mechanism, and effectively integrates scattered computing resources while ensuring data privacy. This technology was first seen in the pioneering research of "Communication-Efficient Learning of Deep Networks from Decentralized Data", and its original design was to adapt to the data scale differences of different user terminals through a distributed training framework. Since then, federated learning technology has developed rapidly and has been successfully applied to core tasks such as image classification, object detection, and semantic segmentation in the field of computer vision. Of particular note is that this decentralized training mechanism provides a new technical path for medical image anatomical segmentation, and can effectively utilize multi-source heterogeneous data resources across institutions while strictly complying with medical data privacy regulations. In the standard federated learning framework, each edge node (such as a medical institution) first encrypts and uploads the locally trained model parameters to the central coordination server. Subsequently, the server securely fuses the model parameters of all participating nodes through a specific aggregation algorithm (such as FedAvg) to generate a global consensus model. Finally, the updated global model weights will be redistributed to each client to form a continuously optimized federated training loop. This working mechanism not only ensures that the original medical image data always remains local, but also realizes cross-institutional knowledge transfer through the parameter sharing mechanism.

[0060] As Figure 2 shown in the network architecture part, in order to ensure that the information in each modality can be effectively decomposed into modality-specific and modality-invariant information, the present invention attaches an auxiliary modality decoupler guided by a decoupling loss during training, which helps with adaptive modality decomposition. Therefore, the decoupling strategy can achieve adaptive federated aggregation of data and modalities of distributed clients, while alleviating the client heterogeneity from data and modalities caused by problems such as training instability and slow convergence speed.

[0061] In this embodiment, a modality memory mechanism is also designed to store and refresh modality prototypes to make up for missing modalities. Specifically, modality prototypes are extracted from the modality-specific representations of the modality-customized encoders, and then stored and refreshed in the memory shared by the clients; the modality prototypes are retrieved from the memory to compensate for the incomplete modalities in each client based on the existing modalities. Therefore, the modality memory mechanism further facilitates modality aggregation and fusion in hybrid modality federated learning; the cascaded representations from the modality-shared encoder, the modality-customized encoder, and the modality memory are sequentially fed into the modality-shared decoder to predict the final segmentation mask.

[0062] S2. Through modal customization, the common encoder and the modal decoupler decouple modal information;

[0063] As Figure 3 shown, in this embodiment, Figure 3 the modal decoupler in [] includes a main branch, along with two different loss functions. These two branches and the corresponding loss functions play a complementary and collaborative role. In this embodiment, the modal decoupler includes: two decoupling branches under different loss function constraints. The upper first branch takes the set of output representations of all modal encoders as input and directly performs modal classification with the help of cross-entropy loss The first branch encourages the modal customization encoder to generate representations that are easy to judge the modality, prompting the first branch to mainly encode modality-specific information. With the help of the Gradient Reversal Layer (GRL), the first branch also encourages the modal sharing encoder to generate representations that are difficult to judge, prompting it to mainly encode modality-shared information. While the lower second branch takes the modal fusion representation and the global modal representation as input and indirectly performs modal decoupling with the help of triplet loss Specifically, denote Anchor as any modality-shared representation, Positive as the fusion representation formed by any two modality-shared representations, and Negative as the fusion representation formed by any modality-shared representation and modality-specific representation. The triplet loss indirectly promotes the decoupling of the modal encoder by minimizing the information gain between Positive and Anchor and maximizing the information gain between Negative and Anchor.

[0064] S3. The modal memory mechanism compensates for incomplete modalities in each local client by memorizing, refreshing, and retrieving modal prototypes;

[0065] As Figure 4 shown, in this embodiment, a memory bank with n slots is allocated for each modality. This memory bank is globally shared by all clients and is dynamically refreshed by the modal prototypes clustered from the corresponding local modal customization representations;

[0066] During the storage and refresh process, the modal customization encoder in the local client will generate modality-specific representations for each sample in each local round. Using these representations for lightweight clustering, n clustering centers are obtained as the corresponding modal prototypes;

[0067] These clustering centers are written into the repository as modal prototypes. During model training or inference, by comparing the intermediate corresponding representations of the existing modalities with the memory of the missing modalities, the missing modal prototype with the highest similarity is read to complete the missing modal information.

[0068] S4. Model training and testing.

[0069] The model is iterated multiple times using the above method. Each iteration first trains locally for a number of local rounds, and then when using the federated update algorithm of the present invention for update testing, given a medical image, the model automatically obtains the segmentation masks of each structure.

[0070] Taking brain tumor segmentation as an example, the present invention formulates the practical non - centralized hybrid - modality MRI image segmentation as a new FL paradigm involving multiple modalities, called Mixed - Modality Federated Learning (MixMFL). The Mixed - Modality Federated Learning (MixMFL) of the present invention is different from the existing Multi - Modality Federated Learning (MulMFL) and Cross - Modality Federated Learning (CroMFL) paradigms. In the MulMFL paradigm, all clients hold multi - modality data with the same image modality but from different data distributions (different hospitals), and only data heterogeneity is involved in the federated aggregation process.

[0071] In the CroMFL paradigm, each client holds a different modality from the same data distribution, and only modality heterogeneity is involved in the federated aggregation process. In contrast, in the proposed MixMF paradigm, each client has multiple hybrid modalities, also from different data distributions, and both modality heterogeneity and data heterogeneity are involved in the federated aggregation process. It aims to utilize the hybrid mode of heterogeneous data from distributed clients for effective federated learning while ensuring a stable and adaptive aggregation process.

[0072] In summary, the present invention performs MRI image segmentation based on the Mixed - Modality Federated Learning framework with Modal Decoupling and Memory (MDM - MixMFL), which is characterized by a modal decoupling strategy and a modal memory mechanism. Regarding the modal decoupling strategy, the present invention deploys multiple modality - specific encoders and a modality - shared encoder for each client (hospital), and decouples each modality into modality - specific and modality - shared components for customized and shared federated updates. This strategy achieves stable hybrid - mode modality fusion and data aggregation across distributed hybrid - mode clients (hospitals) and generates a personalized optimal segmentation model for each client.

[0073] The present invention proposes a new Mixed - Modality Federated Learning framework with Modal Decoupling and Memory (MDM - MixMFL), which can achieve stable hybrid - mode modality fusion and data aggregation among distributed hybrid - mode clients (hospitals) and generate a personalized segmentation model for each client.

[0074] The modal decoupling strategy proposed by the present invention adaptively decouples each modality into modality-specific and modality-shared information, which are respectively used for customizing and sharing parameter updates. During the hybrid modality federated update process, the corresponding modality encoders are respectively updated for customization and sharing. This helps to stably and adaptively jointly aggregate heterogeneous data and modalities from distributed clients, promoting the stable and adaptive federated aggregation of heterogeneous data and modalities from distributed clients.

[0075] Through the explicit modal decoupling strategy and dynamic memory mechanism, the present invention can more accurately separate modal features, dynamically complete missing modalities, and optimize the federated learning process for medical image segmentation, thereby improving the stability and segmentation accuracy of the model under heterogeneous data.

[0076] Regarding the modal memory mechanism, the present invention designs a modal memory module to memorize, refresh, and retrieve modal prototypes to make up for incomplete modalities in local clients. The modal memory mechanism proposed by the present invention buffers the modal prototypes dynamically refreshed from each modality-specific encoder to compensate for incomplete modalities in each local client, which further facilitates modal aggregation and fusion in the hybrid modality federated learning process.

[0077] Through the modal decoupling strategy and dynamic modal memory mechanism, the present invention explicitly separates modality-specific and shared information and completes missing modalities, making it more suitable for federated learning of distributed hybrid modalities and improving the adaptability and performance of the model in complex medical scenarios.

[0078] The present invention solves the technical problem of the dispersion of model learning in the prior art for the dispersed hybrid mode scenario, resulting in the heterogeneity obstacle.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid modal federated learning method based on modal decoupling and modal memory, characterized in that, The method includes: S1. Deploy the MDM-MixMFL framework, perform federated update operations, adaptively perform federated aggregation operations on the data and modalities of distributed clients through decoupling strategies, design a modality memory mechanism to store and refresh modality prototypes to compensate for missing modalities. Among them, the MDM-MixMFL framework includes: modality decoupling of mixed modality federated learning MixMFL, and an overview pipeline of the memory MDM framework; S2. Through modality customization operations, use a common encoder and a modality decoupler to decouple modality information. Among them, the modality decoupler includes: a cross-entropy loss modality decoupling branch and a triplet loss modality decoupling branch; S3. Use the modality memory mechanism to compensate for incomplete modalities in each local distributed client by memorizing, refreshing, and retrieving modality prototypes, and complete missing modality information; S4. Iterate from S2 to S3, use the federated update algorithm to train and test the model, obtain an applicable model, process the given medical image, and obtain a segmentation mask.

2. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 1, characterized in that In S3, deploy at least 2 modality customization encoders and modality sharing encoders for each distributed client, use the modality customization encoders to encode corresponding modality information, use the modality sharing encoders to encode modality sharing information, and perform federated updates among modality encoders of the same category.

3. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 2, wherein Use the modality sharing encoder to receive all modalities of the image, encode invariant information of the modality, and perform federated updates based on all modalities in all clients; During the training process, an additional auxiliary modality decoupler is attached, and the decoupling loss is used to guide the auxiliary modality decoupler for adaptive modality decomposition.

4. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 3, wherein, Extract modality prototypes from the modality-specific representations of the modality customization encoders; Store and refresh the modality prototypes in the memory shared by the clients; Retrieve the modality prototypes from the memory to compensate for incomplete modalities in each distributed client based on the existing modalities; Feed the concatenated representations of the modality sharing encoder, the modality customization encoder, and the modality memory to the modality sharing decoder in a preset order to predict the final segmentation mask.

5. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 1, characterized in that In S2, the cross-entropy loss modality decoupling branch uses the output representation sets of all modality encoders as inputs and performs modality classification using cross-entropy loss. to perform modality classification.

6. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 5, wherein, The cross-entropy loss modality decoupling branch encourages the modality customization encoder to generate judgment-applicable modality representations and obtain encoded modality-specific information; With the help of the gradient reversal layer GRL, encourage the modality sharing encoder to generate non-applicable-to-judgment representations and encode modality sharing information.

7. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 1, characterized in that, In S2, the triplet loss modal decoupling branch takes the modal fusion representation and the global modal representation as inputs and uses the triplet loss to perform an indirect modal encoder decoupling operation.

8. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 7, characterized in that, Use the triplet loss to minimize the information gain between the fusion representation Positive formed by any two modality sharing representations and any one modality sharing representation Anchor, and maximize the information gain between the fusion representation Negative formed by any modality sharing representation and the modality-specific representation and any one modality sharing representation Anchor, and perform the indirect modality encoder decoupling operation.

9. The hybrid modal federated learning method based on modal decoupling and modal memory according to claim 1, characterized in that, In S3, enable the distributed clients to globally share the memory bank of each modality, and perform dynamic refresh customization operations on the modality prototypes from the corresponding local modalities; During the storage and refresh process, the modality customization encoders in the local distributed clients generate modality-specific representations for each sample in each local round; use the modality-specific representations for lightweight clustering to obtain n clustering centers; Write the clustering center as the modality prototype into the repository; During model training and inference, perform a comparison operation on the intermediate corresponding representations of existing modalities and the missing modality memories, and obtain and read the missing modality prototype with the highest similarity to complete the missing modality information.

10. A hybrid modal federated learning system based on modal decoupling and modal memory, characterized in that The system includes: A framework deployment module for deploying the MDM-MixMFL framework, performing federated update operations, adaptively aggregating the data and modalities of distributed clients through a decoupling strategy, designing a modality memory mechanism to store and refresh modality prototypes to compensate for missing modalities; where the MDM-MixMFL framework includes: modality decoupling of mixed modality federated learning MixMFL and an overview pipeline of the memory MDM framework; A modality information decoupling module for decoupling modality information through modality customization operations using a common encoder and a modality decoupler; where the modality decoupler includes: a cross-entropy loss modality decoupling branch and a triplet loss modality decoupling branch, and the modality information decoupling module is connected to the framework deployment module; A modality information incomplete module for compensating for incomplete modalities in each local distributed client and completing missing modality information by using a modality memory mechanism to memorize, refresh, and retrieve modality prototypes, and the modality information incomplete module is connected to the modality information decoupling module; An iterative training module for iterating from S2 to S3, training and testing the model using a federated update algorithm to obtain an applicable model, processing a given medical image to obtain a segmentation mask, and the iterative training module is connected to the modality information incomplete module and the modality information decoupling module.

Images