Training method and device of encoder model, electronic equipment and storage medium

Abstract

The application provides a training method and device of an encoder model, electronic equipment and a storage medium. The method comprises the following steps: a server device first receives encoder parameters respectively sent by a plurality of training terminal devices after the local encoders are trained in a first round; then the plurality of received encoder parameters are aggregated according to modalities to obtain a plurality of global encoder parameters, and the plurality of global encoder parameters obtained by aggregation, private decoder parameters of the server device, and a training image of a full modality are used to train a plurality of global encoders deployed by the server device in a second round; subsequently, the global encoder parameters of the plurality of trained global encoders are obtained, and a plurality of multi-modal anchor points are obtained; finally, the plurality of multi-modal anchor points and the global encoder parameters of the plurality of trained global encoders are sent to the plurality of training terminal devices respectively, so that the local encoders are trained in a third round. Through the application, the communication overhead in the model training process can be saved.

Description

Technical Field

[0001] This application relates to the field of federated learning technology, and more particularly to a method, apparatus, electronic device, and storage medium for training an encoder model. Background Technology

[0002] Federated learning is a distributed machine learning technique that builds a global model based on virtual fused data without exchanging local individual or sample data, but only by exchanging model parameters or intermediate results, thereby achieving a balance between data privacy protection and data sharing computation.

[0003] Taking multimodal image segmentation tasks in a distributed federated learning scenario as an example, in order to address the new intermodal heterogeneity problem brought about by multimodal federated learning, the solutions provided by related technologies require the server to calculate the relevant models for all blocks for each client in each round. This requires a lot of computing resources, and the transmission of complete model parameters also brings considerable communication overhead. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training an encoder model, which can save communication overhead during model training.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for training an encoder model, including:

[0007] The system receives encoder parameters sent by multiple training devices, wherein each training device includes training images of a partial modality, and the encoder parameters are obtained by the training device after training the local encoder in the first round based on the training images of the partial modality.

[0008] Multiple encoder parameters are aggregated according to their corresponding modes to obtain multiple global encoder parameters;

[0009] Based on the parameters of the multiple global encoders, the private decoder parameters of the server device, and the training images of the full modality, the multiple global encoders deployed on the server device are trained in the second round.

[0010] Obtain the global encoder parameters of the trained global encoders and obtain multiple multimodal anchor points;

[0011] The training devices send the multimodal anchor points and the global encoder parameters of the trained global encoders to the training devices respectively, so that the training devices can train the local encoder in the third round.

[0012] This application provides a training apparatus for an encoder model, comprising:

[0013] A receiving module is used to receive encoder parameters sent by multiple training end devices respectively, wherein each training end device includes a training image of a partial modality, and the encoder parameters are obtained by the training end device after training the local encoder in the first round based on the training image of the partial modality.

[0014] The aggregation module is used to aggregate multiple encoder parameters according to their corresponding modes to obtain multiple global encoder parameters;

[0015] The training module is used to train the multiple global encoders deployed on the server device in the second round based on the parameters of the multiple global encoders, the private decoder parameters of the server device, and the training images of the full modality.

[0016] The acquisition module is used to acquire the global encoder parameters of the multiple global encoders after training, and to acquire multiple multimodal anchor points;

[0017] The sending module is used to send the multiple multimodal anchor points and the global encoder parameters of the multiple trained global encoders to the multiple training terminal devices respectively, so that the training terminal devices can train the local encoder in the third round.

[0018] This application provides a method for training an encoder model, including:

[0019] The encoder parameters are sent to the server device, wherein the encoder parameters are obtained after the local encoder is trained in the first round based on the training images of partial modalities. The server device includes training images of full modalities and is equipped with multiple global encoders that correspond one-to-one with multiple modalities in the full modalities.

[0020] The server receives multiple multimodal anchor points and global encoder parameters of the multiple global encoders after training, wherein the global encoder parameters of the multiple global encoders after training and the multiple multimodal anchor points are obtained by the server based on the encoder parameters sent by the multiple training devices respectively after the second round of training.

[0021] Based on the multiple multimodal anchor points, the global encoder parameters of the multiple trained global encoders, and the training images of the partial modalities, the local encoder is trained in the third round.

[0022] This application provides a training apparatus for an encoder model, comprising:

[0023] The sending module is used to send encoder parameters to the server device. The encoder parameters are obtained after the local encoder is trained in the first round based on the training images of partial modalities. The server device includes training images of full modalities and is equipped with multiple global encoders that correspond one-to-one with multiple modalities in the full modalities.

[0024] The receiving module is used to receive multiple multimodal anchor points sent by the server device and global encoder parameters of the multiple trained global encoders, wherein the global encoder parameters of the multiple trained global encoders and the multiple multimodal anchor points are obtained by the server device after the second round of training based on the encoder parameters sent by the multiple training devices respectively.

[0025] The training module is used to train the local encoder in the third round based on the multiple multimodal anchor points, the global encoder parameters of the multiple trained global encoders, and the training images of the partial modalities.

[0026] This application provides an electronic device, including:

[0027] Memory, used to store executable instructions;

[0028] The processor, when executing executable instructions stored in the memory, implements the encoder model training method provided in the embodiments of this application.

[0029] This application provides a computer-readable storage medium storing computer-executable instructions for implementing the encoder model training method provided in this application when executed by a processor.

[0030] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the training method for the encoder model provided in this application.

[0031] The embodiments of this application have the following beneficial effects:

[0032] When training the encoder model, the training device only needs to send encoder parameters to the server device, and the server device only needs to send the global encoder parameters of the multiple global encoders after training, as well as the multimodal anchors extracted from the training images of all modalities, to each training device. In this way, the transmission of partial parameters and highly abstract multimodal anchors can save communication overhead to a certain extent. At the same time, the embodiments of this application can allow the training device to have incomplete modalities, which aims to effectively compensate for the information loss caused by the lack of modalities in each training device through the multimodal anchors sent by the server device. In addition, by sending encoder parameters obtained from the training images of the included partial modalities to the server device, the training device can also provide the server device with more diverse modal information, thereby improving the performance of the models of each participating party. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the architecture of the training system for the encoder model provided in the embodiments of this application;

[0034] Figure 2A This is a schematic diagram of the structure of the server-side device provided in the embodiments of this application;

[0035] Figure 2B This is a schematic diagram of the structure of the training terminal device provided in the embodiments of this application;

[0036] Figure 3 This is a flowchart illustrating the training method for the encoder model provided in the embodiments of this application;

[0037] Figure 4 This is a flowchart illustrating the training method for the encoder model provided in the embodiments of this application;

[0038] Figure 5 This is a flowchart illustrating the training method for the encoder model provided in the embodiments of this application;

[0039] Figure 6A This is a schematic diagram of client modality missing provided in an embodiment of this application;

[0040] Figure 6B This is a schematic diagram of the segmentation result provided in an embodiment of this application;

[0041] Figure 7A This is a schematic diagram illustrating an application scenario of the encoder model training method provided in this application embodiment;

[0042] Figure 7B This is a schematic diagram illustrating an application scenario of the encoder model training method provided in this application embodiment;

[0043] Figure 8This is a schematic diagram of the overall architecture of the encoder model training system provided in this application embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0047] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0050] 1) Federated learning: A distributed machine learning technique that builds a global model based on virtual fused data by exchanging model parameters or intermediate results without exchanging local individual or sample data, thereby achieving a balance between data privacy protection and data sharing computation.

[0051] 2) Personalized Federated Learning: This method is proposed to address the performance degradation of federated learning on highly heterogeneous data. It does not require all participants to use the same model in the end, but allows each participant to fine-tune the model according to their own data, thereby generating a unique personalized model for themselves.

[0052] 3) Server-side equipment: Equipment used to provide services to training equipment, wherein the server-side equipment can be a server or a terminal device.

[0053] 4) Training device: The device used to train the encoder model. The training device can be a server or a terminal device.

[0054] 5) Attention Mechanism: Originating from research on human vision. In cognitive science, due to information processing bottlenecks, humans selectively focus on a portion of all information while ignoring other visible information. This mechanism is commonly referred to as the attention mechanism. In neural networks, the attention mechanism is a resource allocation scheme that, under limited computational power, allocates computational resources to more important tasks while addressing the problem of information overload. In neural network learning, generally speaking, the more parameters a model has, the stronger its expressive power and the greater the amount of information it stores, but this can lead to information overload. By introducing the attention mechanism, focusing on information more critical to the current task from a large amount of input information, reducing attention to other information, and even filtering out irrelevant information, the problem of information overload can be solved, and the efficiency and accuracy of task processing can be improved.

[0055] 6) Responding to: used to indicate the conditions or states on which the operation is performed. When the conditions or states on which the operation is performed are met, one or more operations may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0056] 7) Multimodal MRI images: i.e. information in multiple modalities. In the embodiments of this application, multimodal MRI images refer to the four imaging modes of magnetic resonance imaging (MRI), namely T1 weighted (T1), enhanced T1 weighted (T1c), T2 weighted (T2), and liquid attenuation inversion sequence (FLAIR).

[0057] 8) Masked Average Pooling (MAP): This refers to using a segmentation mask (or segmentation annotation) to mask the features of the foreground region (or background region) extracted by the encoder, thereby obtaining the background prototype (or foreground prototype). In this embodiment, a segmentation mask can be used to perform masked average pooling on the multimodal fusion feature map output by the multimodal fusion decoder. That is, the segmentation mask masks regions other than those identified from the training image (e.g., edema regions), meaning that only the features of the specific regions identified from the training image are retained as the corresponding category features.

[0058] In multimodal federated learning scenarios, such as multimodal MRI brain tumor segmentation tasks, related technologies provide the following solutions: modality-based normalization methods, contrastive representation-based integration and aggregation methods, correlation-adaptive methods based on separate networks, and semi-supervised multimodal aggregation methods.

[0059] Among them, modal-based normalization methods (such as FedNorm) use modal normalization (MN) technology to privatize the normalization parameters and not participate in the aggregation of modal parameters.

[0060] Among them, contrastive representation ensemble and aggregation methods (such as CreamFL) also utilize a public multimodal dataset, and knowledge exchange between the client (i.e., the training device) and the server (i.e., the server device) is completed only on this public dataset. A global-local cross-modal ensemble strategy is designed to aggregate the client's representations, and further, intra-modal and inter-modal contrastives are proposed to normalize local training, compensating for the missing modal information of single-modal clients and further improving client performance.

[0061] Among these, correlation-adaptive methods based on split networks (such as FedMSplit) first split the client model into multiple smaller, shareable blocks and adaptively obtain the correlations between multimodal clients using a dynamic multi-view graph structure, thereby enabling local model training. Specifically, in each round, the server receives the client model parameters, updates the current client relationships, and sends the aggregated relationship information to the client. For each client, potential client relationships are considered and used as regularization terms in the local dataset for training and updates, thereby improving model performance.

[0062] Among them, semi-supervised multimodal aggregation methods (such as FedIoT) mainly consider semi-supervised scenarios. Given that the client's local data is unlabeled, an autoencoder is trained locally on the client and uploaded. A labeled auxiliary dataset from the server is then used to train the model for the final task. Ultimately, the client can use the server-trained model to complete the local task and achieve performance improvements.

[0063] However, during the implementation of the embodiments of this application, the applicant discovered that: the modality-based normalization method (FedNorm) is mainly proposed for computed tomography (CT) and MRI modality images in federated multimodal liver segmentation. Its modality normalization technique is highly specific and cannot be well generalized to other multimodal federated learning scenarios. Methods based on contrastive representation ensemble and aggregation (e.g., CreamFL) rely solely on a public dataset for knowledge exchange between the client and server, which to some extent undermines the privacy protection mechanism under federated learning. Furthermore, the client must train not only on its local private data but also on public data, increasing the client's training burden. Relevance adaptation methods based on separate networks (e.g., FedMSplit) require the server to compute the relevance model for all blocks corresponding to each client in each round, consuming significant computational resources. The transmission of complete model parameters also incurs considerable communication overhead. Semi-supervised multimodal aggregation methods (e.g., FedIoT) require some clients to have aligned multimodal data, which places even stricter demands on the clients.

[0064] In other words, when faced with the new intermodal heterogeneity problem brought about by multimodal federated learning, the above four technical solutions only consider the performance of the client model, obtaining a personalized model for each client, while ignoring the optimization of the global model. Furthermore, relevance adaptation methods based on separate networks (e.g., FedMSplit) and semi-supervised multimodal aggregation methods (e.g., FedIoT) rely heavily on the participation of multimodal clients to achieve performance improvements, which is impractical in real-world scenarios.

[0065] In view of this, embodiments of this application provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for training an encoder model, which can save communication overhead during model training while protecting privacy. The electronic device provided in the embodiments of this application is described below. The electronic device (including a server device and a training device) provided in the embodiments of this application can be implemented as a server, or as a terminal device, or implemented collaboratively by a server and a terminal device. For example, the server device can be implemented as a server, and the training device as a terminal device.

[0066] For example, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the encoder model training system provided in this application embodiment. To support applications that complete model training in a resource-efficient manner, such as... Figure 1 As shown, the encoder model training system includes: a server device 200, a network 300, and multiple training devices (e.g., training device 400-1, training device 400-2, ..., training device 400-N, where N is a positive integer greater than 1). The server device 200 includes training images for all modalities and deploys multiple global encoders that correspond one-to-one with the multiple modalities in the full modality. The network 300 can be a local area network (LAN) or a wide area network (WAN), or a combination of both. Each training device includes training images for only a portion of the modalities and deploys at least one encoder that corresponds one-to-one with at least one modality in the portion of the modalities. Furthermore, different training devices belong to different training parties, such as different individuals, enterprises, institutions, or organizations. For example, taking a multimodal MRI image segmentation task as an example, server device 200 may belong to a large regional hospital and store MRI images of complete modalities (including FLAIR, T1c, T1, and T2 modalities) in server device 200. Then, server device 200 can deploy four global encoders corresponding one-to-one with the four different modalities. Multiple training devices may belong to multiple different local health units. Due to the limitations of imaging equipment, it is difficult to obtain MRI images of complete modalities. That is, each training device only includes MRI images of a portion of the modalities. For example, taking training device 400-1 as an example, assuming that training device 400-1 belongs to local health unit 1, due to the limitations of imaging equipment, training device 400-1 only includes MRI images of T1c and T2 modalities. Then, training device 400-1 only deploys two local encoders corresponding to these two modalities respectively.

[0067] In some embodiments, training devices 400-1, 400-2, ..., 400-N train local encoders in the first round based on training images of their respective partial modalities, and send the trained local encoder parameters to the server device 400. After receiving the encoder parameters sent by the multiple training devices, the server device 200 aggregates the multiple encoder parameters according to their corresponding modalities to obtain multiple global encoder parameters. Then, based on the multiple global encoder parameters, the server device 200's private decoder parameters, and the full-modal training images, the server device 200 trains the multiple global encoders deployed in the server device 200 in the second round. Subsequently, the server device 200 can obtain the global encoder parameters of the trained multiple global encoders and obtain multiple multimodal anchor points. Finally, the server device 200 can send multiple multimodal anchors and global encoder parameters of multiple trained global encoders to training devices 400-1, 400-2, ..., 400-N respectively. This allows each training device to train its local encoder in the third round based on the multiple multimodal anchors returned by the server device 200, the global encoder parameters of multiple trained global encoders, and training images of some modalities. In this way, only encoder parameters and multimodal anchors are transmitted between the server device and the training devices. The partial parameters and highly abstract multimodal anchors can save communication overhead to a certain extent. At the same time, the training devices can use the multimodal anchors extracted from the full-modal training images by the server device to make up for the information loss caused by the lack of modalities, thereby improving the performance of the models of each participant.

[0068] In other embodiments, the embodiments of this application can also be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0069] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0070] Example, Figure 1The server-side device 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The training-side devices (including training-side devices 400-1, 400-2, ..., 400-N) can be smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, etc., but are not limited to these. The training-side devices (e.g., training-side devices 400-1, 400-2, ..., 400-N) and the server-side device 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0071] The following continues... Figure 1 The structure of the server device 200 shown in the diagram will be explained. For example, the server device 200 is a server; see [link to documentation]. Figure 2A , Figure 2A This is a schematic diagram of the structure of the server-side device provided in the embodiments of this application. Figure 2A The server-side device shown includes at least one processor 210, a memory 240, and at least one network interface 220. The various components in the server-side device 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to implement communication between these components. In addition to a data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 230.

[0072] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The memory 240 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 240 may optionally include one or more storage devices physically located away from the processor 210.

[0074] The memory 240 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 240 described in this application embodiment is intended to include any suitable type of memory.

[0075] In some embodiments, memory 240 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0076] Operating system 241 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0077] The network communication module 242 is used to reach other computing devices via one or more (wired or wireless) network interfaces 220, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0078] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A A training device 243 for an encoder model stored in memory 240 is shown. This device can be software in the form of programs and plug-ins, including the following software modules: a receiving module 2431, an aggregation module 2432, a training module 2433, an acquisition module 2434, a sending module 2435, an update module 2436, and a determination module 2437. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions implemented. It should be noted that... Figure 2A For ease of explanation, all the above modules are shown at once, but this should not be interpreted as excluding the implementation of the encoder model training device 243, which may only include the receiving module 2431, the aggregation module 2432, the training module 2433, the acquisition module 2434, and the sending module 2435. The functions of each module will be described below.

[0079] The structure of the training terminal device provided in the embodiments of this application will be described below. For example, a server will be used as the training terminal device; see [link to relevant documentation]. Figure 2B , Figure 2B This is a schematic diagram of the structure of the training terminal device provided in the embodiments of this application. Figure 2BThe training device shown includes at least one processor 410, a memory 440, and at least one network interface 420. The various components in the training device 400 are coupled together via a bus system 430. It is understood that the bus system 430 is used to implement communication between these components. In addition to a data bus, the bus system 430 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2B The general labeled all buses as Bus System 430.

[0080] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0081] The memory 440 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 440 may optionally include one or more storage devices physically located away from the processor 410.

[0082] The memory 440 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 440 described in this application embodiment is intended to include any suitable type of memory.

[0083] In some embodiments, memory 440 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0084] Operating system 441 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0085] The network communication module 442 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0086] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2B A training device 443 for an encoder model stored in memory 440 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a transmitting module 4431, a receiving module 4432, and a training module 4433. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0087] The training method of the encoder model provided in this application embodiment will be specifically described below from the perspective of the interaction between the training end device and the server end device.

[0088] See Figure 3 , Figure 3 This is a flowchart illustrating the training method for the encoder model provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0089] It should be noted that, Figure 3 The illustrated server-side device includes training images for all modalities and deploys multiple global encoders corresponding one-to-one with the multiple modalities in the full modality. The training-side device includes training images for only a subset of modalities and deploys at least one local encoder corresponding one-to-one with at least one modality in the subset. Furthermore, it should be noted that there should be multiple training-side devices, and the steps performed by each training-side device are similar. For ease of description, Figure 3 The following explanation uses a training device as an example.

[0090] In step 101, the training device trains the local encoder in the first round based on the training images of the included partial modalities.

[0091] In some embodiments, at least one local encoder corresponding one-to-one with at least one modality among the included partial modalities can be deployed in the training device. That is, the number of local encoders deployed in the training device is the same as the number of modalities of the training images included in the training device. For example, taking MRI images as training images, there are four different modalities of MRI images, namely T1, T1c, T2, and FLAIR. Assuming that the training device only has MRI images of modalities T1 and T2, two local encoders (let's say local encoder 1 and local encoder 2) can be deployed in the training device, and these two local encoders can be trained using MRI images of these two modalities respectively. For example, local encoder 1 can be trained based on MRI images of modality T1, and local encoder 2 can be trained based on MRI images of modality T2.

[0092] In step 102, the training device sends encoder parameters to the server device.

[0093] In some embodiments, after the training device trains the local encoder for the first round based on the training images of the included partial modalities, it can obtain the encoder parameters of the trained local encoder and send the obtained encoder parameters to the server device.

[0094] For example, taking partial modal training images as T1 and T2 modal MRI images, the training device deploys two local encoders corresponding to these two modalities, assuming they are local encoder 1 and local encoder 2. After training local encoder 1 based on T1 modal MRI images and local encoder 2 based on T2 modal MRI images, the training device can obtain the encoder parameters of local encoder 1 (e.g., encoder parameter 1) and the encoder parameters of local encoder 2 (e.g., encoder parameter 2) after training, and send encoder parameter 1 and encoder parameter 2 to the server device.

[0095] In step 103, the server device aggregates multiple encoder parameters according to their corresponding modes to obtain multiple global encoder parameters.

[0096] In some embodiments, multiple global encoder parameters may correspond one-to-one with multiple modes in the full modality. Then, the server device can implement step 103 in the following way: aggregate multiple encoder parameters according to their corresponding modes to obtain global encoder parameters corresponding to each mode.

[0097] For example, the server device can achieve the above aggregation of multiple encoder parameters according to their corresponding modes to obtain the global encoder parameters corresponding to each mode in the following way: For each mode, perform the following processing: filter out the target encoder parameters corresponding to the mode from the multiple encoder parameters; sum the multiple target encoder parameters to obtain the summation result; determine the number of multiple target encoder parameters; divide the summation result by the number to determine the global encoder parameter corresponding to the mode.

[0098] For example, taking MRI images as training images, there are four different modalities of MRI images: T1, T1c, T2, and FLAIR. Assume there are five training devices: Training Device 1, Training Device 2, Training Device 3, Training Device 4, and Training Device 5. Training Device 1 only includes MRI images of the T1 and T2 modalities, and therefore has two local encoders deployed in it, let's say Local Encoder 1 and Local Encoder 2. Training Device 2 only includes MRI images of the T1c and FLAIR modalities, and therefore has... The backup device 2 has two local encoders, let's assume they are local encoder 3 and local encoder 4; the training device 3 only includes MRI images of T1 and FLAIR modalities, so it has two local encoders, let's assume they are local encoder 5 and local encoder 6; the training device 4 only includes MRI images of T1c and T2 modalities, so it has two local encoders, let's assume they are local encoder 7 and local encoder 8; the training device 5 only includes MRI images of T1 modalities, so it has only one local encoder, let's assume it is local encoder 9.

[0099] Taking training device 1 as an example, after training local encoder 1 and local encoder 2 based on MRI images of T1 and T2 modalities respectively, training device 1 can send the encoder parameters of local encoder 1 (assumed to be encoder parameter 1) and the encoder parameters of local encoder 2 (assumed to be encoder parameter 2) after training to the server device. Similarly, training device 2 can send the encoder parameters 3 after training local encoder 3 based on MRI images of T1c modality and the encoder parameters 4 after training local encoder 4 based on MRI images of FLAIR modality to the server device. Training device 3 can send the encoder parameters 5 after training local encoder 5 based on MRI images of T1 modality and the encoder parameters 6 after training local encoder 6 based on MRI images of FLAIR modality to the server device. Training device 4 can send the encoder parameters 7 after training local encoder 7 based on MRI images of T1c modality and the encoder parameters 8 after training local encoder 8 based on MRI images of T2 modality to the server device. Training device 5 can send the encoder parameters 9 after training local encoder 9 based on MRI images of T1 modality to the server device.

[0100] It should be noted that the encoder parameters sent from the training device to the server device can also carry the identifier of the corresponding modality. For example, taking the encoder parameter 1 mentioned above as an example, encoder parameter 1 can carry the identifier of T1 modality to indicate that encoder parameter 1 is trained based on MRI images of T1 modality.

[0101] After receiving the encoder parameters sent by the five training devices, the server can aggregate multiple encoder parameters according to their corresponding modes. For example, the server can aggregate encoder parameter 1 sent by training device 1, encoder parameter 5 sent by training device 3, and encoder parameter 9 sent by training device 4 (these three encoder parameters all carry the identifier corresponding to mode T1). That is, it can sum encoder parameter 1, encoder parameter 5, and encoder parameter 9, and divide the sum by 3 to obtain the global encoder parameter corresponding to mode T1. Similarly, it can aggregate encoder parameter 2 sent by training device 1 and encoder parameter 8 sent by training device 4 (these two encoder parameters both carry the identifier corresponding to mode T2). The encoder parameters are summed with encoder parameter 8 and divided by 2 to obtain the global encoder parameters corresponding to the T2 mode. The encoder parameters 3 sent by training device 2 and 7 sent by training device 4 are aggregated (both of these encoder parameters carry the identifier corresponding to the T1c mode), that is, the encoder parameters 3 and 7 are summed and the sum is divided by 2 to obtain the global encoder parameters corresponding to the T1c mode. The encoder parameters 4 sent by training device 2 and 6 sent by training device 3 are aggregated (both of these encoder parameters carry the identifier corresponding to the FLAIR mode), that is, the encoder parameters 4 and 6 are summed and the sum is divided by 2 to obtain the global encoder parameters corresponding to the FLAIR mode.

[0102] In step 104, the server device trains multiple global encoders deployed on the server device in the second round based on multiple global encoder parameters, server device-private decoder parameters, and full-modal training images.

[0103] In some embodiments, multiple global encoder parameters may correspond one-to-one with multiple modalities in the full modality, and multiple global encoders may correspond one-to-one with multiple global encoder parameters. Then, the server device can implement step 104 in the following way: based on the global encoder parameters corresponding to each modality, the private decoder parameters of the server device, and the training images of the full modality, train the global encoder corresponding to each modality in the second round.

[0104] For example, a multimodal fusion decoder can also be deployed in the server device. The server device can implement the above-mentioned training of the global encoder corresponding to each modality in the second round based on the global encoder parameters corresponding to each modality, the server device's private decoder parameters, and the training images of the whole modality. In the second round, the following processing is performed: For each modality, the global encoder parameters corresponding to the modality are used as the initial parameters of the global encoder corresponding to the modality, and the training images corresponding to the modality in the training images of the whole modality are input into the global encoder corresponding to the modality to obtain the feature map of the training images corresponding to the modality; the private decoder parameters are used as the decoder parameters of the multimodal fusion decoder, and the feature maps output by the global encoder corresponding to each modality are input into the multimodal fusion decoder to obtain the prediction result for the training images of the whole modality; the prediction result and the label data of the training images of the whole modality are substituted into the loss function, and the error calculated based on the loss function is backpropagated to train the global encoder corresponding to each modality.

[0105] For example, taking MRI images as training images, there are four different modalities of MRI images: T1, T1c, T2, and FLAIR. The server-side device stores MRI images of these four modalities and deploys four global encoders corresponding to each of these four modalities. Let's assume they are global encoder 1 for T1 modality, global encoder 2 for T1c modality, global encoder 3 for T2 modality, and global encoder 4 for FLAIR modality. After aggregating the encoder parameters sent from multiple training devices to obtain the global encoder parameters corresponding to each modality, the server-side device can use the global encoder parameters of the corresponding modality as the initial parameters of the global encoder for that modality. For example, the global encoder parameters corresponding to T1 modality can be used as the initial parameters of global encoder 1, the global encoder parameters corresponding to T1c modality as the initial parameters of global encoder 2, the global encoder parameters corresponding to T2 modality as the initial parameters of global encoder 3, and the global encoder parameters corresponding to FLAIR modality as the initial parameters of global encoder 4. Subsequently, the T1 modality MRI images from the full-modality training images can be input into global encoder 1, causing global encoder 1 to output feature maps obtained after feature extraction of the T1 modality MRI images. Similarly, the T1c modality MRI images can be input into global encoder 2, causing global encoder 2 to output feature maps obtained after feature extraction of the T1c modality MRI images. The T2 modality MRI images can be input into global encoder 3, causing global encoder 3 to output feature maps obtained after feature extraction of the T2 modality MRI images. The FLAIR modality MRI images can be input into global encoder 4, causing global encoder 4 to output feature maps obtained after feature extraction of the FLAIR modality MRI images. The feature maps output by these four global encoders can then be processed separately. Figure 1 The input is a multimodal fusion decoder, which outputs segmentation results for the MRI image. For example, the multimodal fusion decoder can aggregate the modality-specific feature maps output by the four global encoders, outputting a multimodal fusion feature map and the probability that each pixel in the multimodal fusion feature map belongs to a different region. Finally, the segmentation results output by the multimodal fusion decoder and the labeled data for the MRI image are substituted into a loss function, and the error calculated by the loss function is backpropagated to optimize the global encoder parameters of the four global encoders.

[0106] It should be noted that the loss function mentioned above can be of various types, such as cross-entropy loss function, hinge loss function, etc. The embodiments of this application do not specifically limit the type of loss function.

[0107] Furthermore, it should be noted that, for the sake of distinction, the encoder deployed on the server-side device is referred to as the global encoder, and the encoder deployed on the training-side device is referred to as the local encoder. In other words, the global encoder refers to the encoder deployed on the server-side device, and the local encoder refers to the encoder deployed on the training-side device.

[0108] In step 105, the server device obtains the global encoder parameters of multiple trained global encoders and obtains multiple multimodal anchor points.

[0109] In some embodiments, after the server device trains multiple global encoders deployed on the server device in the second round based on multiple global encoder parameters, the server device's private decoder parameters, and full-modal training images, the server device can obtain the global encoder parameters of the trained multiple global encoders (i.e., the updated global encoder parameters).

[0110] For example, taking MRI images as training images, there are four different modalities of MRI images: T1, T1c, T2, and FLAIR. That is, there are four global encoders deployed on the server device that correspond one-to-one with these four modalities. After the server device trains the four global encoders deployed on the server device in the second round, it can obtain the global encoder parameters of the four global encoders after training, based on the aggregated parameters of the four global encoders, the server device's private decoder parameters, and the full-modal MRI images.

[0111] In other embodiments, a multimodal fusion decoder can also be deployed in the server device. The server device can then obtain multiple multimodal anchor points by: acquiring multiple multimodal fusion feature maps of different scales output by the multimodal fusion decoder; performing masked average pooling on the multiple multimodal fusion feature maps of different scales using a segmentation mask to obtain multiple categorical features of different scales; clustering the multiple categorical features of different scales (e.g., using K-means clustering, mean-shift clustering, density-space-based clustering, hierarchical clustering, etc.) to obtain multiple clusters; and determining the cluster center corresponding to each cluster as a corresponding multimodal anchor point. For example, taking cluster 1 as an example, the average value of at least one categorical feature included in cluster 1 can be calculated, and the calculated average value can be used as the cluster center corresponding to cluster 1.

[0112] For example, a multimodal fusion decoder may include multiple region-aware fusion modules (RFM), where each RFM module outputs a multimodal fusion feature map at one scale. The above-mentioned acquisition of multiple multimodal fusion feature maps at different scales output by the multimodal fusion decoder can be achieved as follows: acquire multiple multimodal fusion feature maps at different scales output by multiple RFM modules, where each RFM module corresponds one-to-one with a different scale, and the scales of the multimodal fusion feature maps output by different RFM modules are different.

[0113] For example, taking MRI images as training images, which have four different modalities—T1, T1c, T2, and FLAIR—the multimodal fusion decoder provided in this embodiment can also consist of four region-aware fusion modules corresponding one-to-one with the four global encoders. These are assumed to be region-aware fusion module 1, region-aware fusion module 2, region-aware fusion module 3, and region-aware fusion module 4, where each region-aware fusion module outputs a multimodal fusion feature map at one scale. For instance, the feature maps output by the four global encoders can be input into region-aware fusion module 1 for aggregation to obtain a multimodal fusion feature map at one scale. Then, the multimodal fusion feature map output by region-aware fusion module 1 is input into region-aware fusion module 2, causing region-aware fusion module 2 to output a multimodal fusion feature map with half the channel dimension and double the scale. This process can be repeated to obtain four multimodal fusion feature maps at different scales.

[0114] In other words, the embodiments of this application can construct the network using an encoder-decoder architecture, where four global encoders extract features from MRI images of different modalities. To establish the relationship between image morphology and tumor regions, a region-aware fusion module can also be introduced. The region-aware fusion module first divides the modal features into different regions (i.e., tumor substructures) using the learned probability map. The probability map represents the probability of a tumor region at each pixel. Then, the region-aware fusion module generates corresponding attention weights for each region, adaptively controlling the contribution of different image morphologies.

[0115] For example, embodiments of this application can aggregate feature maps output by multiple global encoders (i.e., feature representations from specific modal encoders) using a multimodal fusion decoder to output a multimodal fusion feature map. Since the multimodal fusion feature map includes features from all regions of the training image, a segmentation mask can be used to perform masked average pooling on the multimodal fusion feature map. This masks regions other than those identified from the training image, meaning only the features of the specific regions identified from the training image are retained as the corresponding category features. For example, taking an MRI image as the training image, masked average pooling can be used to segment edema regions or enhanced tumor regions from the multimodal fusion feature map as the corresponding category features.

[0116] In step 106, the server device sends multiple multimodal anchor points and global encoder parameters of multiple trained global encoders to the training device.

[0117] In some embodiments, after obtaining the global encoder parameters (i.e., the updated global encoder parameters) of the trained global encoders and the multimodal anchors, the server device can send the multimodal anchors and the global encoder parameters of the trained global encoders to the training devices respectively, so that the training devices can train the local encoder in the third round.

[0118] It should be noted that the first round, second round, and third round in the embodiments of this application do not refer to a specific round, but are used to distinguish different rounds. For example, the first round can be the previous round, the second round can be the current round, and the third round can be the next round. In addition, the rounds of the training device and the server device are carried out alternately. For example, if the current round is training on the training device, then the next round is training on the server device.

[0119] In other embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the training method of the encoder model provided in this application embodiment, as shown below. Figure 4 As shown, the server device is executing Figure 3 Before step 106 shown, the following steps can also be performed: Figure 4 Step 108 shown will combine Figure 4 The steps shown are explained.

[0120] In step 108, the server device updates multiple multimodal anchor points using an exponential moving average method to obtain updated multimodal anchor points.

[0121] In some embodiments, the server device may implement step 108 in the following manner: obtaining multiple historical multimodal anchor points obtained in the fourth round of training, wherein the fourth round is earlier than the second round; multiplying the multiple historical multimodal anchor points with weighting coefficients to obtain a first multiplication result, wherein the value of the weighting coefficients is between 0 and 1; determining the difference between 1 and the weighting coefficients, and multiplying the difference with the multiple multimodal anchor points to obtain a second multiplication result; adding the first multiplication result and the second multiplication result to obtain a summed result, and determining the summed result as the updated multiple multimodal anchor points.

[0122] For example, to avoid training crashes caused by frequent centroid jumps during each round of re-clustering, this embodiment of the application can use multimodal anchors as a memory bank and update them using an exponential moving average method. The exponential moving average method refers to a prediction method that uses the actual and predicted (or estimated) values ​​from the previous period, assigns them different weights, and obtains an exponentially smoothed value as the predicted value for the next period. For example, in the current round, the multimodal anchors (let's call them 'x', 'y', 'x', 'y', are obtained as follows: After that, we can also obtain the multimodal anchor points obtained in the previous round (let's call them 'x'). Then, the multimodal anchor points obtained in the previous round can be multiplied by the weighting coefficients (let's say β, where β can be 0.999) to obtain the first multiplication result (i.e. Then, the difference between 1 and the weighting coefficient can be determined, and this difference is multiplied by the multimodal anchor point obtained in the current round to obtain the second multiplication result (i.e. Finally, the first multiplication result and the second multiplication result are added together, and the result is used as the multimodal anchor point after the current round update.

[0123] In step 107, the training device trains the local encoder in the third round based on multiple multimodal anchor points, global encoder parameters of multiple trained global encoders, and training images of some modalities.

[0124] In some embodiments, the training device may implement step 107 by: selecting target global encoder parameters corresponding to a partial modality from the global encoder parameters of multiple trained global encoders; and training a local encoder based on the target global encoder parameters, multiple multimodal anchors, and training images of the partial modality.

[0125] For example, taking MRI images as training images, there are four different modalities in MRI images: T1, T1c, T2, and FLAIR. The server device can deploy four global encoders corresponding to these four modalities, assuming they are global encoder 1 (T1 modality), global encoder 2 (T1c modality), global encoder 3 (T2 modality), and global encoder 4 (FLAIR modality). That is, the global encoder parameters of the trained global encoders include: the global encoder parameters of global encoder 1 (e.g., updated global encoder parameter 1), the global encoder parameters of global encoder 2 (e.g., updated global encoder parameter 2), and the global encoder parameters of global encoder 3 (e.g., updated global encoder parameter 4). The training device includes global encoder parameters 3 and global encoder parameters 4 (e.g., updated global encoder parameters 4). Assuming the training device only includes MRI images of the T1 and T2 modalities, two local encoders corresponding to these two modalities are deployed in the training device, such as local encoder 1 corresponding to the T1 modality and local encoder 2 corresponding to the T2 modality. After receiving the global encoder parameters of the multiple trained global encoders from the server device, the training device can replace the original encoder parameters of local encoder 1 with the global encoder parameters of trained global encoder 1 (i.e., updated global encoder parameters 1), and replace the original encoder parameters of local encoder 2 with the global encoder parameters of trained global encoder 3 (i.e., updated global encoder parameters 3).

[0126] It should be noted that the global encoder parameters of multiple global encoders after training can carry the identifier of the corresponding mode. For example, the updated global encoder parameter 1 mentioned above can carry the identifier corresponding to the T1 mode to indicate that the global encoder parameter corresponds to the T1 mode. Thus, the training device can use the global encoder parameter to replace the encoder parameter of the local encoder corresponding to the T1 mode.

[0127] For example, the training device can implement the above-mentioned training of the local encoder based on the target global encoder parameters, multiple multimodal anchors, and training images of some modalities in the following way: replace the encoder parameters of the local encoder with the target global encoder parameters; encode the training images of some modalities through the local encoder to obtain feature maps at different scales; for each scale feature map, perform the following processing: reshape the dimensions of the feature map at that scale and use the reshaped feature map as a query; obtain the multimodal anchors corresponding to that scale from multiple multimodal anchors as keys and values; input the query, key, and value into the cross-attention function to obtain a calibration feature map; reshape the dimensions of the calibration feature map and add the obtained reshaped feature map to the feature map at that scale to obtain the addition result, wherein the addition result is used for backpropagation to update the encoder parameters of the local encoder.

[0128] It should be noted that the local encoder in this embodiment may include multiple convolutional layers. Each convolutional process doubles the scale of the feature map and halves the channel dimension. Thus, after multiple convolutional layers, multiple feature maps of different scales can be output. Furthermore, since the dimension of the feature map does not match the cross-attention function, it is necessary to first reshape the feature map's dimensions. Dimension reshaping refers to changing the dimension of the vector, for example, reshaping the vector from two dimensions to three dimensions. This embodiment can reshape the feature map's dimension from... Remodeling This allows it to be used as a query for subsequent cross-attention functions. Similarly, after obtaining the calibration feature map, its dimensions can be reshaped back, allowing it to be added to the original feature map.

[0129] In other embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the training method of the encoder model provided in this application embodiment, as shown below. Figure 5 As shown, after execution Figure 3 After step 107 shown, you can also execute... Figure 5 Steps 109 to 111 shown will combine Figure 5 The steps shown are explained.

[0130] In step 109, the training device sends the updated encoder parameters to the server device.

[0131] In some embodiments, after the training device has trained the local encoder parameters (i.e. updated encoder parameters) for the third time, based on multiple multimodal anchor points sent by the server device, the global encoder parameters of multiple trained global encoders, and training images of some modalities, the training device can send the encoder parameters of the trained local encoder (i.e., updated encoder parameters) to the server device.

[0132] In step 110, the server device aggregates multiple updated encoder parameters to obtain multiple updated global encoder parameters.

[0133] In some embodiments, after receiving updated encoder parameters from multiple training devices, the server device can aggregate the updated encoder parameters according to their corresponding modes to obtain updated global encoder parameters for each mode. It should be noted that the process of the server device aggregating multiple updated encoder parameters is similar to the process of aggregating multiple encoder parameters in step 103, and can be referred to the implementation of step 103 above. This embodiment will not be repeated here.

[0134] In step 111, in response to the training rounds reaching the round threshold, the server device determines the multiple updated global encoder parameters as the final global encoder parameters of the multiple global encoders.

[0135] In some embodiments, after the server device aggregates multiple updated encoder parameters according to their corresponding modalities to obtain the updated global encoder parameters corresponding to each modal, it can determine whether the current round has reached the round threshold. If the round threshold (e.g., 500 rounds) has been reached, the multiple updated global encoder parameters can be used as the final global encoder parameters of the multiple global encoders. If the current round has not reached the round threshold, step 104 above can be repeated to perform the next round of training until the number of training rounds reaches the round threshold.

[0136] It should be noted that the trained global encoder (or local encoder) provided in this application embodiment can be used to extract features of an image and output the corresponding feature map. The multimodal fusion decoder can be used to aggregate the feature maps output by multiple global encoders (i.e., aggregate the feature representations from a specific modality encoder) and output the segmentation result for the image. For example, it can output a multimodal fusion feature map and the probability that each pixel in the multimodal fusion feature map belongs to a specific region.

[0137] The encoder model training method provided in this application allows the server device to transmit only the multimodal anchor points extracted from the full-modal training images and the global encoder parameters of the trained global encoder to the training device during the encoder model training process. This effectively avoids privacy leaks. Furthermore, the transmission of some parameters and highly abstract multimodal anchor points also saves communication overhead to some extent. In addition, the training device can effectively compensate for information loss caused by missing modalities through the multimodal anchor points sent by the server device, thereby improving the performance of the models of all participating parties.

[0138] The following example, using multimodal MRI brain tumor image segmentation in a distributed federation scenario, illustrates an exemplary application of the embodiments of this application in a real-world application scenario.

[0139] This application addresses a multimodal federated learning scenario where multiple participants have incomplete data modalities. Considering the heterogeneity between modalities, it assigns a modality-specific encoder to each modality and proposes a personalized federated learning framework (FedMEMA, Federated Modality-specific Encoders and Multimodal Anchors) that effectively utilizes complete multimodal data from the server. First, the server (corresponding to the server-side device) uses a multimodal fusion decoder to aggregate feature representations from specific modality encoders (corresponding to the global encoder) to complete its own training. Simultaneously, it extracts multiple multimodal anchors from the fused multimodal representations and distributes them, along with the updated global encoder parameters, to the clients (corresponding to the training-side device). The clients then perform local calibration through an attention mechanism, utilizing the multiple multimodal anchors sent by the server to adapt to existing modality representations while compensating for information loss from missing modality representations, thereby improving segmentation accuracy on their respective private data. Furthermore, the server also benefits from richer modality information due to the sharing of encoder parameters trained on more diverse data by each client, resulting in performance improvements.

[0140] The technical solutions provided in the embodiments of this application will be described in detail below.

[0141] The technical solution provided in this application can be applied to multimodal MRI brain tumor image segmentation tasks in a distributed federated environment, and can also be applied to other multimodal medical image segmentation tasks, such as various combinations of modalities like CT, MRI, and positron emission tomography (PET), as well as the segmentation of other body parts such as abdominal organs. Furthermore, different clients may have different modality loss situations, for example, such as... Figure 6AAs shown, for the four given modalities, the client may lack one or more modalities (black boxes in the figure indicate missing modalities). That is, the client only has MRI images of the T1c modality, lacking MRI images of the FLAIR, T1, and T2 modalities. The application target of this embodiment is clients with incomplete modalities. By participating in the personalized federated learning framework provided in this embodiment, without disclosing data privacy, they can ultimately obtain an updated model that significantly outperforms local training and adapts to their data modalities, thereby completing the local brain tumor segmentation task. In other words, after training the model using the technical solution provided in this embodiment, the client can use the trained model to segment the received local data modal images and directly output the results, such as... Figure 6B The results of tumor sub-region segmentation are shown, where region 601 represents the core of necrotic and non-enhancing tumor (NET), region 602 represents enhancing tumor (ET), and region 603 represents edema.

[0142] In other embodiments, the technical solutions provided in this application can be used for tumor region segmentation of multimodal brain MRI data, and allow for clients with arbitrary missing modalities. Each client obtains a better model adapted to its local data modality by participating in the personalized federated learning framework provided in this application.

[0143] For example, for MRI brain tumor images, the complete modality set can be represented by M∈{T1,T1c,T2,FLAIR}. Figure 7A As shown, consider a federated learning scenario with a server possessing a set of full-modal data and multiple clients with incomplete data modalities. Corresponding to a real-world medical scenario, server 701 might be a large regional hospital, while client 702 might be a smaller local healthcare unit. Due to limitations in imaging equipment, obtaining full-modal data is difficult. Therefore, the goal is to compensate for the information loss caused by missing local data modalities by participating in distributed collaboration within federated learning, without disclosing local patient privacy data, and by transmitting only partial parameters and multimodal anchors. This would result in a better model than the performance of training solely on local data. The objective of this application's embodiment is that, with all clients and servers participating in the personalized brain tumor segmentation federated learning framework provided in this application, without direct data sharing, each ultimately obtains a better model than the performance of training solely on local data. This means an optimal global model that can handle full-modal data well, and several optimal personalized models adapted to local data under specific modality missing conditions.

[0144] In other embodiments, such as Figure 7BAs shown, after each client obtains a better model by participating in the personalized federated learning framework provided in this application embodiment, it can receive images of local data modalities (i.e., input, where any one or more modalities may be missing) based on the trained model and directly provide the segmented tumor sub-regions (i.e., output), where region 703 represents the necrotic and non-enhanced tumor core; region 704 represents the enhanced tumor; and region 705 represents edema.

[0145] In some embodiments, such as Figure 8 As shown, this application embodiment primarily considers scenarios where each client possesses only single-modal data, while the server possesses a certain amount of complete multimodal data. Based on this, it can be extended to other client scenarios with incomplete modalities. Under the federated modality-specific encoder architecture, the server with full modality data can use a multimodal fusion decoder to aggregate feature representations from all modality-specific encoders and extract multiple multimodal anchors from the fused multimodal representations, sending them to the client along with the updated global encoder parameters. Single-modal clients can then utilize the multimodal anchors sent by the server based on an attention mechanism to perform local calibration, compensating for missing modality information and improving model performance. In local calibration, this application embodiment can insert a Localized Adaptive Calibration via Cross-Attention (LACCA) module into the existing model architecture to perform local adaptive calibration.

[0146] See also Figure 8 The algorithm flow of the technical solution provided in this application embodiment is as follows:

[0147] (1) Initialize the full-modal dataset D using the server. M (i.e., training images from all modalities, such as MRI images from four modalities) to train a global model (i.e., multiple global encoders), and generate multimodal anchor points A. {l} .

[0148] (2) During each round of training:

[0149] (a) For client i belonging to modality m (i.e., the client whose training image has modality m), receive the global encoder parameters of the corresponding modality. and multimodal anchor point A {l} Then, using the local model (i.e., the local encoder) and the calibration module LACCA on the local training dataset D m Training and updates are performed on the training images (i.e., training images of some modalities).

[0150] (b) Modality-specific encoder parameters uploaded by the client.

[0151] (c) The server receives modality-specific encoder parameters sent by multiple clients and aggregates the received encoder parameters according to their corresponding modalities to obtain global encoder parameters corresponding to each modality.

[0152] (d) The server uses the aggregated global encoder parameters. and private decoder parameters in the full-modal dataset D M Update the training and generate the current multimodal anchor point A. {l} .

[0153] (e) The server updates the multimodal anchor point A using an exponential moving average method. {l} and with the updated global encoder parameters Send it to each client at the same time.

[0154] The steps described above will be explained in detail below.

[0155] In some embodiments, under the specified scenarios of this application, the federated learning paradigm provided by related technologies does not offer advantages due to the high heterogeneity among multimodal MRI data. Therefore, to address this issue, embodiments of this application employ a modal-specific encoder to process different image modalities. For example... Figure 8 As shown, for the server side, this embodiment can employ a late-stage fusion strategy to construct a global model, including a modality-specific encoder corresponding to each modality, and a multimodal fusion decoder for aggregating multimodal features. For the client side, since its data modalities are incomplete, it only includes the modality-specific encoder (i.e., the local encoder) corresponding to the modality, and a private decoder for generating a segmentation mask.

[0156] The process of generating multimodal anchor points will be explained below.

[0157] In some embodiments, the server receives encoder parameters sent by each client. Next, the encoder parameters for the corresponding mode m are aggregated, and the specific formula is as follows:

[0158]

[0159] in, This represents the global encoder parameters obtained after aggregation. N represents the encoder parameters sent by the client. mThis represents the number of clients with data modality m.

[0160] In obtaining modality-specific global encoder parameters Then, the server can first analyze the full-modality dataset D. M Training is completed on the above, and a multimodal fusion decoder is used to align and fuse the multimodal features. Simultaneously, to further enhance representation capabilities, embodiments of this application can also extract category features from the fused multimodal feature maps of different scales l using a segmentation mask, and obtain N using the K-means clustering method. k There are multiple modal anchor points, where the value of K can be set to 3 based on empirical evidence. Specifically, the multimodal fusion decoder provided in this embodiment can consist of four region-aware fusion modules. Each module outputs a multimodal fusion feature map at one scale. Then, a segmentation mask is used to perform masked average pooling on the multimodal fusion feature map to obtain multiple categorical features. Subsequently, the multiple categorical features can be clustered to obtain multiple clusters, and the cluster center of each cluster is used as a corresponding multimodal anchor point.

[0161] In other embodiments, to avoid training crashes caused by frequent centroid jumps during each round of re-clustering, this application embodiment can treat multimodal anchors as a memory bank and update them using an exponential moving average method, as shown in the following formula:

[0162]

[0163] in, This represents the multimodal anchor point obtained during training in round t (e.g., the current round). Let A represent the multimodal anchor points obtained in the (t-1)th round of training (e.g., the previous round), β represent the weighting coefficients (which can be taken as 0.999 here), and A... l This represents the updated multimodal anchor point in round t.

[0164] The following section continues to explain the local adaptive calibration and training process on the client side.

[0165] In some embodiments, in each round, the client receives the global encoder parameters for the corresponding modality from the server. and multimodal anchor point A {l} On the local training dataset D mWhile completing training and updates, multimodal anchors can be used to calibrate local missing modality representations. For the local calibration module LACCA, this embodiment inserts the LACCA module into the four stages of the local model decoder. Similar to the encoder, the four stages of the decoder are all modules composed of convolutions, etc. With each stage, the feature map scale doubles and the channel dimension halves, resulting in four feature maps of different scales. Specifically, for the decoding operation of a certain stage l, this embodiment can first obtain the feature map of the corresponding scale. This is then dimensionally reshaped and used as a query in the attention operation. The corresponding multimodal anchor point A is... l As keys and values, local feature representations are calibrated through a cross-attention mechanism, as shown in Equation (3).

[0166]

[0167] Among them, F l cal F represents the calibrated feature map. l A represents the feature map output by the local encoder. l C represents the multimodal anchor point at the corresponding scale. l is the channel dimension of the feature map, and the subscript l is the corresponding scale.

[0168] Finally, the calibrated feature map F l cal The dimensions are reshaped back and added to the feature map of the same scale before being used in subsequent forward propagation to further optimize the model parameters.

[0169] The algorithm flow of the technical solution provided in the embodiments of this application will be described below.

[0170] Input: M = {T1, T1c, T2, FLAIR}: the set of modes indexed by m, D M The server-side full-modality training dataset, N m The number of clients with data modality m. A single-modal training dataset with client i having modality m, and N r : Number of communication rounds, N e Number of training sessions per round.

[0171] Output: A modality-specific set of global encoder parameters, A {l} ={A l}: Multimodal anchor points of different scales l.

[0172] Server-side execution:

[0173] initialization A {l} In D M Training N e One cycle;

[0174] For the first to the Nth time r For each communication, the following processing is performed:

[0175] For mode m in the mode set, perform the following processing:

[0176] For each client with data modality m, perform the following processing:

[0177] According to the server renew

[0178] Modal aggregation modal-specific encoder parameters

[0179] renew And in D M Training N e One cycle;

[0180] A is updated using K-means clustering and exponential moving average. {l} .

[0181] Client execution:

[0182] Use the data sent from the server. Replace local encoder parameters

[0183] Training dataset locally N is trained using the local model and calibration module (LACCA) e Each cycle, to update

[0184] Upload updated encoder parameters

[0185] In summary, the technical solutions provided by the embodiments of this application have the following beneficial effects:

[0186] 1) Each client can obtain an optimal personalized model that adapts to its local data modality;

[0187] 2) The server can obtain an optimal global model that adapts to all modal data inputs;

[0188] 3) The transmission of some parameters and multimodal anchor points reduces communication overhead to some extent.

[0189] The beneficial effects of the technical solutions provided in the embodiments of this application will be further explained below in conjunction with experimental data.

[0190] This application's embodiments are tested on benchmark datasets for MRI brain tumor image segmentation (e.g., Brats 2020). The experiments follow the overall framework of federated learning. The client employs a modality-specific encoder and a personalized decoder architecture corresponding to the corresponding modality, while the server employs a modality-specific encoder for all modalities and a fusion decoder architecture for multimodal fusion. During training, the input data size is 80*80*80, the optimizer is Adam, the learning rate and weight decay are 0.0002 and 1e-5 respectively, the EMA smoothing coefficient is set to 0.999, and the loss function uses the commonly used dice loss and cross-entropy loss for image segmentation. Simultaneously, due to memory limitations, the batch size for client training is set to 3, and the batch size for server training is set to 1, with 500 training epochs, and each participant training for only one cycle within each epoch.

[0191] For example, the Brats2020 dataset contains 369 MRI data points, each including four complete MRI scan sequences: T1, T1c, T2, and FLAIR. The segmentation target is three nested tumor sub-regions: 1. Whole Tumor (WT), including all tumor regions; 2. Tumor Core (TC), consisting of enhancing tumor, necrotic areas, and a non-enhancing tumor core; 3. Enhancing Tumor (ET). In this embodiment, the dataset is divided into training, validation, and test sets according to a 219:50:100 ratio.

[0192] To avoid loss of generality, this application's embodiments discuss two settings in a federated scenario. In setting 1, the training set is randomly and evenly distributed among the five participants (i.e., four clients and one server), with no overlap in patient data between clients. In setting 2, the server-side data remains unchanged, while the clients exhibit some data overlap (randomly divided into 4+1 parts, where four parts are unique to each client and the remaining part is shared by all clients). This setting corresponds to a real-world scenario where a patient may have undergone relevant examinations at multiple hospitals within a given region. Each client can only use data specific to its modality, while the server uses complete modality data. The final model's segmentation accuracy is evaluated using the Dice coefficient (%), where the Dice coefficient is a set similarity metric function typically used to calculate the similarity between two samples, with a value ranging from [0, 1].

[0193] This embodiment first compares the encoder-decoder architecture provided by related technologies and the modality-specific encoder architecture provided by this embodiment on the Brats validation set, according to the federated scenario setting in Setting 1, and examines the impact of different feature anchor strategies on the final segmentation performance. This embodiment sets up four clients with different modalities, denoted as FLAIR, T1c, T1, and T2, and a full-modality server S, comparing their average segmentation performance (Dice coefficient) on their respective modality test sets. As shown in Table 1, the performance of the modality-specific encoder architecture provided by this embodiment is significantly better than that of the encoder-decoder architecture provided by related technologies in the second row, demonstrating the superiority of the modality-specific encoder architecture provided by this embodiment. The last two rows employ a multimodal feature anchor strategy for local calibration to compensate for missing modality information. It can be seen that for fused multimodal features, multiple anchors provide richer information than single anchors and are better able to replace multimodal brain tumor data. Simultaneously, the comparison of different shared parameters in the first two rows also verifies the rationality and effectiveness of the shared encoder parameters selected in this embodiment.

[0194] Table 1. Comparison of segmentation accuracy of model architecture and multimodal anchoring strategy on the Brats2020 validation set.

[0195]

[0196] Wherein, E&D represents the encoder-decoder architecture provided by related technologies. 4E&D represents the modality-specific encoder architecture provided in the embodiments of this application. N / A indicates that this method is not used or has not been implemented. FLAIR, T1c, T1, and T2 represent clients with corresponding data modalities, S represents the server, and Overall represents the average segmentation performance of all participants.

[0197] Table 2 shows the segmentation performance of the embodiments of this application on the Brats dataset, as well as the results of eight other comparative methods, including four benchmark methods: training with only local data (Local_models), modality-specific architecture network (RFNet), federated aggregation method (FedAvg), personalized federated learning method (perFL), and four related techniques: modality-based normalization method (FedNorm), method based on contrastive representation ensemble and aggregation (CreamFL), relevance adaptation method based on separate networks (FedMSplit), and semi-supervised multimodal aggregation method (FedIoT). It is evident that, under both experimental settings, the average performance of the technique proposed in the embodiments of this application outperforms the other methods, with optimal performance from all participants, including the client and server. This demonstrates the advanced nature and effectiveness of the technical solution provided in the embodiments of this application, which is attributed to the architecture of the modality-specific encoder provided in the embodiments of this application and the effective utilization of the server's full-modality data.

[0198] Table 2. Comparison of segmentation accuracy of different methods applied to the Brats2020 test set.

[0199]

[0200] The following description continues to illustrate the exemplary structure of the encoder model training device 243 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2A As shown, the software modules in the encoder model training device 243 stored in the memory 240 may include: a receiving module 2431, an aggregation module 2432, a training module 2433, an acquisition module 2434, and a sending module 2435.

[0201] The receiving module 2431 is used to receive encoder parameters sent by multiple training end devices, wherein each training end device includes training images of partial modalities, and the encoder parameters are obtained by the training end device based on the training images of partial modalities after training the local encoder in the first round; the aggregation module 2432 is used to aggregate the multiple encoder parameters according to their corresponding modalities to obtain multiple global encoder parameters; the training module 2433 is used to train the multiple global encoders deployed on the server device in the second round based on the multiple global encoder parameters, the private decoder parameters of the server device, and the full-modal training images; the acquisition module 2434 is used to acquire the global encoder parameters of the multiple global encoders after training and acquire multiple multimodal anchors; the sending module 2435 is used to send multiple multimodal anchors and the global encoder parameters of the multiple global encoders after training to the multiple training end devices respectively, so that the training end devices can train the local encoder in the third round.

[0202] In some embodiments, multiple global encoder parameters correspond one-to-one with multiple modes in the full mode; the aggregation module 2432 is further configured to aggregate multiple encoder parameters according to their corresponding modes to obtain global encoder parameters corresponding to each mode.

[0203] In some embodiments, the aggregation module 2432 is further configured to perform the following processing for each mode: filter out the target encoder parameters corresponding to the mode from a plurality of encoder parameters; sum the plurality of target encoder parameters to obtain a summation result; determine the number of the plurality of target encoder parameters; and divide the summation result by the number to determine the global encoder parameter corresponding to the mode.

[0204] In some embodiments, multiple global encoder parameters correspond one-to-one with multiple modalities in the full modality, and multiple global encoders correspond one-to-one with multiple global encoder parameters; the training module 2433 is further configured to train the global encoder corresponding to each modality in the second round based on the global encoder parameters corresponding to each modality, the private decoder parameters of the server device, and the training images of the full modality.

[0205] In some embodiments, the server device is further deployed with a multimodal fusion decoder; the training module 2433 is further configured to perform the following processing in the second round: for each modality, the global encoder parameters corresponding to the modality are used as the initial parameters of the global encoder corresponding to the modality, and the training images corresponding to the modality in the training images of the whole modality are input into the global encoder corresponding to the modality to obtain the feature map of the training image corresponding to the modality; the private decoder parameters are used as the decoder parameters of the multimodal fusion decoder, and the feature maps output by the global encoder corresponding to each modality are input into the multimodal fusion decoder to obtain the prediction result for the training images of the whole modality; the prediction result and the label data labeled for the training images of the whole modality are substituted into the loss function, and the error calculated based on the loss function is backpropagated to train the global encoder corresponding to each modality.

[0206] In some embodiments, the server device is further deployed with a multimodal fusion decoder; the acquisition module 2434 is further configured to acquire multiple multimodal fusion feature maps of different scales output by the multimodal fusion decoder; perform masked average pooling on the multiple multimodal fusion feature maps of different scales through a segmentation mask to obtain multiple class features of different scales; cluster the multiple class features of different scales to obtain multiple clusters; and determine the cluster center corresponding to each cluster as a corresponding multimodal anchor point.

[0207] In some embodiments, the multimodal fusion decoder includes multiple region-aware fusion modules; the acquisition module 2434 is further configured to acquire multiple multimodal fusion feature maps of different scales output by the multiple region-aware fusion modules, wherein the multiple region-aware fusion modules correspond one-to-one with multiple different scales, and the scales of the multimodal fusion feature maps output by different region-aware fusion modules are different.

[0208] In some embodiments, the encoder model training device 243 further includes an update module 2436, which is used by the sending module 2435 to update the multiple multimodal anchors using an exponential moving average method before sending the multiple multimodal anchors and the global encoder parameters of the multiple trained global encoders to the multiple training end devices respectively, to obtain the updated multiple multimodal anchors; the sending module 2435 is also used to send the updated multiple multimodal anchors and the global encoder parameters of the multiple trained global encoders to the multiple training end devices respectively.

[0209] In some embodiments, the update module 2436 is further configured to obtain multiple historical multimodal anchor points obtained in the fourth round of training, wherein the fourth round is earlier than the second round; multiply the multiple historical multimodal anchor points with weighting coefficients to obtain a first multiplication result; determine the difference between 1 and the weighting coefficients, and multiply the difference with the multiple multimodal anchor points to obtain a second multiplication result; add the first multiplication result and the second multiplication result, and determine the added result as the updated multiple multimodal anchor points.

[0210] In some embodiments, multiple multimodal anchors and global encoder parameters of multiple trained global encoders are used by the training device to perform the following processes: selecting target global encoder parameters corresponding to a subset of modalities from the global encoder parameters of the multiple trained global encoders; and training a local encoder in a third round based on the target global encoder parameters, multiple multimodal anchors, and training images of the subset of modalities.

[0211] In some embodiments, multiple multimodal anchors and the global encoder parameters of multiple trained global encoders are used by the training device to perform the following processes: In the third round, the following processes are performed: replacing the encoder parameters of the local encoder with the target global encoder parameters; encoding the training images of some modalities using the local encoder to obtain feature maps at different scales; for each scale feature map, the following processes are performed: reshaping the scale feature map and using the reshaped feature map as a query; obtaining the multimodal anchors corresponding to the scale from multiple multimodal anchors as keys and values; inputting the query, key, and value into a cross-attention function to obtain a calibration feature map; reshaping the calibration feature map and adding the resulting reshaped feature map to the scale feature map to obtain a sum, wherein the sum is used for backpropagation to update the encoder parameters of the local encoder.

[0212] In some embodiments, the receiving module 2431 is further configured to receive updated encoder parameters sent by multiple training end devices respectively, wherein the updated encoder parameters are obtained by the training end devices after training the local encoder in the third round based on multiple multimodal anchor points, the global encoder parameters of multiple trained global encoders, and the training images of the included partial modalities; the aggregation module 2432 is further configured to aggregate the multiple updated encoder parameters to obtain multiple updated global encoder parameters; the training device 243 of the encoder model further includes a determining module 2437, configured to determine the multiple updated global encoder parameters as the final global encoder parameters of the multiple global encoders in response to the training round reaching a round threshold.

[0213] The following description continues to illustrate the exemplary structure of the encoder model training device 443 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2B As shown, the software modules in the encoder model training device 443 stored in the memory 440 may include: a transmitting module 4431, a receiving module 4432, and a training module 4433.

[0214] The sending module 4431 is used to send encoder parameters to the server device. The encoder parameters are obtained after the local encoder is trained in the first round based on the partial modality training images. The server device includes full modality training images and deploys multiple global encoders that correspond one-to-one with the multiple modalities in the full modality. The receiving module 4432 is used to receive multiple multimodal anchors and global encoder parameters of multiple trained global encoders sent by the server device. The global encoder parameters of multiple trained global encoders and multiple multimodal anchors are obtained by the server device after the second round of training based on the encoder parameters sent by multiple training devices respectively. The training module 4433 is used to train the local encoder in the third round based on multiple multimodal anchors, global encoder parameters of multiple trained global encoders, and partial modality training images.

[0215] In some embodiments, the training module 4433 is further configured to perform the following processing in the third round: selecting target global encoder parameters corresponding to a partial modality from the global encoder parameters of multiple trained global encoders; replacing the encoder parameters of the local encoder with the target global encoder parameters; encoding the training images of the partial modality using the local encoder to obtain feature maps at different scales; and performing the following processing for each scale feature map: reshaping the dimensions of the scale feature map and using the reshaped feature map as a query; obtaining the multimodal anchor points corresponding to the scale from multiple multimodal anchor points as keys and values; inputting the query, key, and value into a cross-attention function to obtain a calibration feature map; reshaping the dimensions of the calibration feature map and adding the obtained reshaped feature map to the scale feature map to obtain a summing result, wherein the summing result is used for backpropagation to update the encoder parameters of the local encoder.

[0216] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For technical details not covered in the encoder model training apparatus provided in this application embodiment, please refer to... Figure 3 , Figure 4 ,or Figure 5 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0217] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform the encoder model training method described above in this application.

[0218] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the training method for the encoder model provided in this application. For example, ... Figure 3 , Figure 4 ,or Figure 5 The training method for the encoder model is shown.

[0219] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0220] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0221] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0222] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training an encoder model, characterized in that, The method includes: The system receives encoder parameters sent by multiple training devices, wherein each training device includes training images of a partial modality, and the encoder parameters are obtained by the training device after training the local encoder in the first round based on the training images of the partial modality. Multiple encoder parameters are aggregated according to their corresponding modes to obtain multiple global encoder parameters; Based on the parameters of the multiple global encoders, the private decoder parameters of the server device, and the training images of the full modality, the multiple global encoders deployed on the server device are trained in the second round. Obtain the global encoder parameters of the trained global encoders; The server-side device is also deployed with a multimodal fusion decoder to acquire multiple multimodal fusion feature maps of different scales output by the multimodal fusion decoder; By performing masked average pooling on the multimodal fusion feature maps of multiple different scales through segmentation masking, multiple category features of different scales are obtained. Clustering is performed on the category features at multiple different scales to obtain multiple clusters; The cluster center corresponding to each cluster is determined as a corresponding multimodal anchor point; Multiple multimodal anchor points and global encoder parameters of the multiple trained global encoders are sent to the multiple training terminal devices respectively, so that the training terminal devices can train the local encoder in the third round.

2. The method according to claim 1, characterized in that, The multiple global encoder parameters correspond one-to-one with the multiple modes in the full modality; The aggregation of multiple encoder parameters according to their corresponding modes yields multiple global encoder parameters, including: The multiple encoder parameters are aggregated according to their corresponding modes to obtain the global encoder parameters corresponding to each mode.

3. The method according to claim 2, characterized in that, The aggregation of multiple encoder parameters according to their corresponding modes to obtain global encoder parameters corresponding to each mode includes: For each of the aforementioned modes, the following processing is performed: Filter out the target encoder parameters corresponding to the mode from a plurality of encoder parameters; The multiple target encoder parameters are summed to obtain a summation result, and the number of the multiple target encoder parameters is determined. The summation result is divided by the quantity to obtain the division result, which is then determined as the global encoder parameter corresponding to the mode.

4. The method according to claim 1, characterized in that, The multiple global encoder parameters correspond one-to-one with the multiple modes in the full modality, and the multiple global encoders correspond one-to-one with the multiple global encoder parameters; The second round of training of the multiple global encoders deployed on the server device, based on the parameters of the multiple global encoders, the private decoder parameters of the server device, and the full-modal training images, includes: Based on the global encoder parameters corresponding to each modality, the private decoder parameters of the server device, and the training images of the entire modality, the global encoder corresponding to each modality is trained in the second round.

5. The method according to claim 4, characterized in that, The second round of training of the global encoder corresponding to each modality, based on the global encoder parameters corresponding to each modality, the private decoder parameters of the server device, and the training images of the entire modality, includes: The following processing will be performed in the second round: For each modality, the global encoder parameters corresponding to the modality are used as the initial parameters of the global encoder corresponding to the modality, and the training images corresponding to the modality in the training images of the whole modality are input into the global encoder corresponding to the modality to obtain the feature map of the training image corresponding to the modality. The private decoder parameters are used as the decoder parameters of the multimodal fusion decoder, and the feature maps output by the global encoder corresponding to each modality are input into the multimodal fusion decoder to obtain the prediction results for the training images of the full modality. The prediction results and the label data of the training images for the full modality are substituted into the loss function, and the error calculated based on the loss function is backpropagated to train the global encoder corresponding to each modality.

6. The method according to claim 1, characterized in that, The multimodal fusion decoder includes multiple region-aware fusion modules; The process of obtaining multiple multimodal fusion feature maps of different scales output by the multimodal fusion decoder includes: Multiple multimodal fusion feature maps of different scales output by the multiple region perception fusion modules are obtained, wherein the multiple region perception fusion modules correspond one-to-one with the multiple different scales, and the scales of the multimodal fusion feature maps output by different region perception fusion modules are different.

7. The method according to claim 1, characterized in that, Before sending the multiple multimodal anchor points and the global encoder parameters of the multiple trained global encoders to the multiple training terminal devices respectively, the method further includes: The multiple multimodal anchor points are updated using an exponential moving average method to obtain the updated multiple multimodal anchor points; The step of sending multiple multimodal anchor points and global encoder parameters of the multiple trained global encoders to the multiple training terminal devices includes: The updated multimodal anchor points and the trained global encoder parameters of the multiple global encoders are sent to the multiple training terminal devices respectively.

8. The method according to claim 7, characterized in that, The method of updating the plurality of multimodal anchor points using an exponential moving average to obtain the updated plurality of multimodal anchor points includes: Obtain multiple historical multimodal anchor points obtained during the fourth round of training, wherein the fourth round is earlier than the second round; Multiply the multiple historical multimodal anchor points with the weighting coefficients to obtain the first multiplication result; Determine the difference between 1 and the weighting coefficient, and multiply the difference by the plurality of multimodal anchor points to obtain a second multiplication result; The first multiplication result and the second multiplication result are added together, and the result of the addition is determined as the updated plurality of multimodal anchor points.

9. The method according to claim 1, characterized in that, The plurality of multimodal anchor points and the global encoder parameters of the plurality of trained global encoders are used by the training device to perform the following processing: From the global encoder parameters of the trained multiple global encoders, select the target global encoder parameters corresponding to the partial modalities; The local encoder is trained in the third round based on the target global encoder parameters, the multiple multimodal anchors, and the training images of the partial modalities.

10. The method according to claim 9, characterized in that, The training of the local encoder in the third round, based on the target global encoder parameters, the multiple multimodal anchors, and the training images of the partial modalities, includes: The following processing is performed in the third round: Replace the encoder parameters of the local encoder with the target global encoder parameters; The training images of the partial modalities are encoded using the local encoder to obtain feature maps at different scales; For each of the aforementioned feature maps at each scale, the following processing is performed: The feature map at the specified scale is reshaped in dimensions, and the reshaped feature map is used as a query. Obtain the multimodal anchor point corresponding to the scale from the plurality of multimodal anchor points, and use it as the key and value; The query, the key, and the value are input into a cross-attention function to obtain a calibration feature map; The calibration feature map is dimensionally reshaped, and the resulting reshaped feature map is added to the feature map at the scale to obtain a summing result, wherein the summing result is used for backpropagation to update the encoder parameters of the local encoder.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: The system receives updated encoder parameters sent by the multiple training devices, wherein the updated encoder parameters are obtained by the training devices after the third round of training the local encoder based on the multiple multimodal anchors, the global encoder parameters of the multiple global encoders after training, and the training images of the included partial modalities. The updated encoder parameters are aggregated to obtain multiple updated global encoder parameters; In response to the training epoch reaching the epoch threshold, the multiple updated global encoder parameters are determined as the final global encoder parameters of the multiple global encoders.

12. A method for training an encoder model, characterized in that, The method includes: The encoder parameters are sent to the server device, wherein the encoder parameters are obtained after the local encoder is trained in the first round based on the training images of partial modalities. The server device includes training images of full modalities and is equipped with multiple global encoders that correspond one-to-one with multiple modalities in the full modalities. The server receives multiple multimodal anchor points and trained global encoder parameters of multiple global encoders sent by the server device. The trained global encoder parameters and the multiple multimodal anchor points are obtained by the server device after a second round of training based on the encoder parameters sent by multiple training devices. The multiple multimodal anchor points are cluster centers corresponding to multiple clusters. The multiple clusters are obtained by clustering multiple category features at different scales. The multiple category features at different scales are obtained by performing masked average pooling on the multimodal fusion feature maps at multiple scales output by the multimodal fusion decoder deployed by the server device using a segmentation mask. Based on the multiple multimodal anchor points, the global encoder parameters of the multiple trained global encoders, and the training images of the partial modalities, the local encoder is trained in the third round.

13. The method according to claim 12, characterized in that, The training of the local encoder in the third round, based on the multiple multimodal anchor points, the global encoder parameters of the trained multiple global encoders, and the training images of the partial modalities, includes: The following processing will be performed in the third round: From the global encoder parameters of the trained multiple global encoders, select the target global encoder parameters corresponding to the partial modalities; Replace the encoder parameters of the local encoder with the target global encoder parameters; The training images of the partial modalities are encoded using the local encoder to obtain feature maps at different scales; For each of the aforementioned feature maps at each scale, the following processing is performed: The feature map at the specified scale is reshaped in dimensions, and the reshaped feature map is used as a query. Obtain the multimodal anchor point corresponding to the scale from the plurality of multimodal anchor points, and use it as the key and value; The query, the key, and the value are input into a cross-attention function to obtain a calibration feature map; The calibration feature map is dimensionally reshaped, and the resulting reshaped feature map is added to the feature map at the scale to obtain a summing result, wherein the summing result is used for backpropagation to update the encoder parameters of the local encoder.

14. A training device for an encoder model, characterized in that, The device includes: A receiving module is used to receive encoder parameters sent by multiple training end devices respectively, wherein each training end device includes a training image of a partial modality, and the encoder parameters are obtained by the training end device after training the local encoder in the first round based on the training image of the partial modality. The aggregation module is used to aggregate multiple encoder parameters according to their corresponding modes to obtain multiple global encoder parameters; The training module is used to train the multiple global encoders deployed on the server device in the second round based on the parameters of the multiple global encoders, the private decoder parameters of the server device, and the training images of the full modality. The acquisition module is used to acquire the global encoder parameters of the multiple global encoders after training. The server device is also deployed with a multimodal fusion decoder. The acquisition module is also used to acquire multiple multimodal fusion feature maps of different scales output by the multimodal fusion decoder. Masked average pooling is performed on the multiple multimodal fusion feature maps of different scales through segmentation mask to obtain multiple class features of different scales. The multiple class features of different scales are clustered to obtain multiple clusters. The cluster center corresponding to each cluster is determined as a corresponding multimodal anchor point. The sending module is used to send multiple multimodal anchor points and global encoder parameters of the multiple trained global encoders to the multiple training terminal devices respectively, so that the training terminal devices can train the local encoder in the third round.

15. The apparatus according to claim 14, characterized in that, The multiple global encoder parameters correspond one-to-one with the multiple modes in the full modality; The aggregation module is further configured to aggregate multiple encoder parameters according to their corresponding modes to obtain global encoder parameters corresponding to each mode.

16. The apparatus according to claim 15, characterized in that, The aggregation module is further configured to perform the following processing for each of the modalities: Filter out the target encoder parameters corresponding to the mode from a plurality of encoder parameters; The multiple target encoder parameters are summed to obtain a summation result, and the number of the multiple target encoder parameters is determined. The summation result is divided by the quantity to obtain the division result, which is then determined as the global encoder parameter corresponding to the mode.

17. The apparatus according to claim 14, characterized in that, The multiple global encoder parameters correspond one-to-one with the multiple modes in the full modality, and the multiple global encoders correspond one-to-one with the multiple global encoder parameters; The training module is further configured to train the global encoder corresponding to each modality in the second round based on the global encoder parameters corresponding to each modality, the private decoder parameters of the server device, and the training images of the entire modality.

18. The apparatus according to claim 17, characterized in that, The training module is also used to perform the following processing in the second round: For each modality, the global encoder parameters corresponding to the modality are used as the initial parameters of the global encoder corresponding to the modality, and the training images corresponding to the modality in the training images of the whole modality are input into the global encoder corresponding to the modality to obtain the feature map of the training image corresponding to the modality. The private decoder parameters are used as the decoder parameters of the multimodal fusion decoder, and the feature maps output by the global encoder corresponding to each modality are input into the multimodal fusion decoder to obtain the prediction results for the training images of the full modality. The prediction results and the label data of the training images for the full modality are substituted into the loss function, and the error calculated based on the loss function is backpropagated to train the global encoder corresponding to each modality.

19. The apparatus according to claim 14, characterized in that, The multimodal fusion decoder includes multiple region-aware fusion modules; The acquisition module is further configured to acquire multiple multimodal fusion feature maps of different scales output by the multiple region perception fusion modules, wherein the multiple region perception fusion modules correspond one-to-one with the multiple different scales, and the scales of the multimodal fusion feature maps output by different region perception fusion modules are different.

20. The apparatus according to claim 14, characterized in that, The device further includes an update module, which is used to update the multiple multimodal anchors by an exponential moving average method before the sending module sends multiple multimodal anchors and the global encoder parameters of the multiple global encoders after training to the multiple training terminal devices respectively, so as to obtain the updated multiple multimodal anchors. The sending module is further configured to send the updated multimodal anchor points and the trained global encoder parameters of the multimodal encoders to the multiple training terminal devices respectively.

21. The apparatus according to claim 20, characterized in that, The update module is further configured to obtain multiple historical multimodal anchor points obtained during the fourth round of training, wherein the fourth round is earlier than the second round; multiply the multiple historical multimodal anchor points with weighting coefficients to obtain a first multiplication result; determine the difference between 1 and the weighting coefficients, and multiply the difference with the multiple multimodal anchor points to obtain a second multiplication result; add the first multiplication result and the second multiplication result, and determine the added result as the updated multiple multimodal anchor points.

22. The apparatus according to claim 14, characterized in that, The receiving module is further configured to receive updated encoder parameters sent by the plurality of training end devices respectively, wherein the updated encoder parameters are obtained by the training end device after the third round of training the local encoder based on the plurality of multimodal anchor points, the global encoder parameters of the plurality of trained global encoders, and the training images of the included partial modalities. The aggregation module is also used to aggregate multiple updated encoder parameters to obtain multiple updated global encoder parameters; The training device for the encoder model further includes a determination module, used to determine the multiple updated global encoder parameters as the final global encoder parameters of the multiple global encoders in response to the training rounds reaching a round threshold.

23. A training device for an encoder model, characterized in that, The device includes: The sending module is used to send encoder parameters to the server device. The encoder parameters are obtained after the local encoder is trained in the first round based on the training images of partial modalities. The server device includes training images of full modalities and is equipped with multiple global encoders that correspond one-to-one with multiple modalities in the full modalities. The receiving module is configured to receive multiple multimodal anchor points and global encoder parameters of the multiple trained global encoders sent by the server device. The global encoder parameters of the multiple trained global encoders and the multiple multimodal anchor points are obtained by the server device after a second round of training based on the encoder parameters sent by multiple training devices. The multiple multimodal anchor points are cluster centers corresponding to multiple clusters. The multiple clusters are obtained by clustering multiple category features at different scales. The multiple category features at different scales are obtained by performing masked average pooling on the multimodal fusion feature maps output by the multimodal fusion decoder deployed on the server device using a segmentation mask. The training module is used to train the local encoder in the third round based on the multiple multimodal anchor points, the global encoder parameters of the multiple trained global encoders, and the training images of the partial modalities.

24. The apparatus according to claim 23, characterized in that, The training module is also used to perform the following processing in the third round: From the global encoder parameters of the trained multiple global encoders, select the target global encoder parameters corresponding to the partial modalities; The target global encoder parameters are used to replace the encoder parameters of the local encoder; the training images of the partial modalities are encoded by the local encoder to obtain feature maps of different scales. For each feature map at the aforementioned scale, the following processing is performed: the feature map at the aforementioned scale is reshaped in dimensions, and the reshaped feature map is used as the query. Obtain the multimodal anchor point corresponding to the scale from the plurality of multimodal anchor points, and use it as the key and value; The query, the key, and the value are input into a cross-attention function to obtain a calibration feature map; The calibration feature map is dimensionally reshaped, and the resulting reshaped feature map is added to the feature map at the scale to obtain a summing result, wherein the summing result is used for backpropagation to update the encoder parameters of the local encoder.

25. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the training method of the encoder model according to any one of claims 1 to 11 or any one of claims 12 to 13.

26. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the training method of the encoder model according to any one of claims 1 to 11 or any one of claims 12 to 13.

27. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, they implement the training method of the encoder model according to any one of claims 1 to 11 or any one of claims 12 to 13.

Images