A multi-modal medical target prediction method based on federal prototype under modal loss

Abstract

The application discloses a kind of modal missing based on federal prototype under multi-modal medical target prediction method, comprising: step 1, construct missing multi-modal medical data set, extract single mode feature;Data in data set is private multi-modal data, from N Individual client;Step 2, each single mode feature is fused and projected to shared space to obtain multi-modal feature, calculate local prototype;Step 3, the similarity between global prototype and local prototype is calculated as the degree of semantic consistency, start iterative training, construct global prototype;Step 4, send new global model and global prototype to each client, iteratively execute step 3 until global model converges or reaches the preset global iteration number;Step 5, the client saves global model, and carries out medical target prediction to missing modal data.The application can alleviate medical data heterogeneity, handle modal missing problem, and improve the discriminant ability of cross-modal semantic representation, improve medical target prediction accuracy.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence, distributed machine learning, and privacy computing, and specifically relates to a multimodal medical goal prediction method based on a federated prototype under modality missing conditions. This method is particularly suitable for handling multimodal collaborative learning tasks in real-world scenarios where medical institution data is heterogeneous and some modal data may be missing, such as medical and health data analysis, assisted diagnostic decision-making, and cross-institutional collaborative modeling. Background Technology

[0002] With the continuous development of artificial intelligence and big data technologies, an increasing amount of multimodal medical data is being used to improve the recognition performance of large models and applications (such as disease identification and healthcare). However, this data is usually collected from a single medical institution, making it difficult to reflect the comprehensiveness of the data. Meanwhile, the rise of federated learning provides a promising distributed learning paradigm, allowing multiple medical institutions to collaboratively train models without disclosing private data. Therefore, due to the need to simultaneously process medical data from different clients and protect privacy, multimodal federated learning (MFL) has become an effective solution. However, existing methods still face the following key challenges in practical applications, which constitute the technical problems that this invention aims to solve: (1) Data heterogeneity problem: In federated learning, distributed data is non-independent and identically distributed (non-IID), which leads to a significant decrease in model performance. Although existing federated learning methods can alleviate data heterogeneity to some extent, they are mainly geared towards single-modal data scenarios and have not been effectively extended to multimodal federated learning.

[0003] (2) Modality Missing Problem: In practical applications, due to equipment failure or performance limitations, data for some modalities may be completely missing or contain noise, making it impossible for medical institutions to provide complete modal information. Previous studies typically assume that the modalities of each multimodal patient data are sufficient and complete. However, when faced with incomplete multimodal medical data, model performance deteriorates significantly. For example, studies have shown that models trained with incomplete modalities achieve nearly 20% lower accuracy than those trained with complete modalities.

[0004] In summary, existing methods have significant limitations in handling data heterogeneity and modality missingness in multimodal federated learning. This invention aims to propose an integrated solution to overcome these limitations, enabling efficient and robust multimodal federated learning under conditions of incomplete medical data modalities, ultimately improving prediction accuracy. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, the present invention aims to provide a multimodal medical target prediction method based on federated prototypes under modality missing conditions, which aims to alleviate the heterogeneity of medical data, handle the problem of modality missing, improve the discriminative ability of cross-modal semantic representation, and improve the accuracy of medical target prediction.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Firstly, a multimodal healthcare goal prediction method based on a federated prototype under modality missing conditions includes the following steps: Step 1: Construct a missing multimodal medical dataset and extract unimodal features; the data in the dataset is private multimodal data. , originating from N One client, of which Indicates the first The first patient's Medical data in multiple modalities Indicates the first The labels corresponding to the medical goal assessment results of each patient. , The total number of modes; Step 2: Fuse the single-modal features and project them into a shared space to obtain multimodal features, and calculate the local prototype; Step 3: Calculate the similarity between the global prototype and the local prototype as the degree of semantic consistency, start iterative training, aggregate the semantic consistency degree to obtain the next round of global model parameters, and construct the global prototype on the server. Step 4: Send the new global model and global prototype to each client, and iteratively execute Step 3 until the global model converges or reaches the preset number of global iterations; Step 5: The client saves the global model and uses this global model to predict medical objectives based on the missing modality data.

[0007] In one embodiment, step 1, private multimodal data refers to the private data of a certain client; the missing modality refers to... In each client, at least one modality of data is missing, or all modality data are present, but at least one modality of data is missing to some extent.

[0008] In one embodiment, the multimodality includes: At least two of the following: electronic medical records, medical images, and physiological signals; The medical goal predictions include disease-aided diagnosis, disease risk level prediction, prognostic risk prediction, hospitalization risk prediction, readmission risk prediction, and treatment effect evaluation.

[0009] In one embodiment, the extraction of unimodal features is achieved through a unimodal embedding network, as follows: Based on different modalities, select the appropriate modal encoder to extract single-modal features; Each single-modal feature is input into a recurrent neural network to capture temporal physiological feature changes within an individual patient and between different patients; The outputs of each recurrent neural network are projected into a shared space to align the single-modal features with the complete prototype, resulting in the final single-modal features used for fusion.

[0010] In one embodiment, step 2 involves fusing the individual single-modal features using a multimodal fusion network based on a cross-modal attention module to obtain a fused feature. The multimodal features are then projected onto a shared space to obtain the multimodal features. , means as follows: in, This represents the shared space between different clients. Indicates a projection operation; The local prototype is the mean of the projected multimodal feature representations of patient samples under the same client and the same medical target category, and can be regarded as the initial centroid of the category in the local feature space. The local prototype categories for different clients are defined as follows: in, Indicates the first The first client Local prototype of the category Indicates the first The first client The dataset is categorized, where the category refers to the assessment result of the medical goal, such as low risk, medium risk, high risk, or disease positive, disease negative, etc.

[0011] In one embodiment, in step 2, the local prototype in the feature space of a single client is not unique. The same client can form multiple local prototypes according to different medical target categories. Each local prototype is used to characterize the feature distribution center of the corresponding category of patient samples in the client.

[0012] In one embodiment, step 3, calculating the similarity between the global prototype and the local prototype, is as follows: in, Indicates the first The degree of semantic consistency among clients Indicates the first The client contains a set of medical target categories; Indicates the first The first client The local prototype of the category; Indicates the first in the global prototype The global prototype of the category; Represents the cosine similarity function; The aggregation yields the global model parameters for the next round, and a global prototype for the server is constructed. The implementation method is as follows: First, the sum of the semantic consistency levels of all clients participating in the aggregation is normalized: in, Indicates the first Normalized semantic consistency aggregation weights for each client. Indicates the first The degree of semantic consistency among clients.

[0013] Secondly, calculate the next round of global model parameters obtained after aggregation: in, This represents the total number of clients participating in model parameter aggregation. Indicates the first The model parameters obtained from the first round of training, Indicates the first A collection of client patient samples, This indicates the number of patient samples in the set; This represents the sum of all client samples. Finally, the global prototype of the server is constructed using the following formula: Indicates the first generation built by the server. Class global prototype, ,in Indicates the total number of categories. This represents the global prototype set consisting of all global prototypes of all categories.

[0014] In one embodiment, the following loss function is used during training: in, , For hyperparameter weights, Represents the cross-entropy loss function. This represents the adaptive modal weighted prototype regularization loss function. The dynamic temperature prototype contrast loss function is expressed as follows: in, Indicates the true label in the first place The probability of a class The model predicts the first Class probability, Indicates the first The final adaptive weights for each modality, Indicates the first Multimodal features of a class Represents the prototype set after projection. express and similarity, It is a similarity scaling factor. It is a single-modal feature used for fusion.

[0015] In one embodiment, the first The final adaptive weights for each modality are expressed as follows: in, Indicates the first The first client The first patient Attention scores for each modality Indicates the first The first client The first patient A mask for the absence of a modality. This represents the query vector obtained by mapping the features of this modality. This represents the key vector obtained from the feature mapping of this modality. Indicates the first The first client The first patient The final adaptive weights for each modality, This is the scale scaling factor. The feature dimension of a single attention head. This indicates the number of heads of attention.

[0016] In one embodiment, the similarity scaling factor Represented as in, Indicates the first Sample frequency of class Let be the initial temperature, representing the th Dynamic temperature parameters corresponding to each medical target category This represents a hyperparameter used to adjust the degree of influence of category frequencies. The initial temperature is a preset hyperparameter in contrastive learning, used to control the smoothness of the similarity distribution. In practical applications, It can be set based on the validation set results or empirical values.

[0017] A second aspect of the present invention also provides a multimodal medical goal prediction system based on a federated prototype under modality missing conditions, comprising: The data acquisition end can be various commonly used medical testing and examination equipment, or a scanner, which can be used to read the patient's medical images, physiological signals and electronic medical records respectively; The data transmission module can be any feasible data communication module, such as wireless or wired, used to transmit the collected data to the processor; The processor stores a global model and performs medical target prediction on the received data. This global model is either the global model already obtained through steps 1 to 4 of the first aspect, or it can be the global model obtained by the processor re-executing steps 1 to 4 of the first aspect. The data output module is used to display the medical target prediction results of the global model.

[0018] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention effectively alleviates the representational differences between local and global models through adaptive modality weight prototype regularization, and can dynamically adjust the regularization strength according to the semantic contribution of each modality. It realizes adaptive knowledge transfer from complete prototypes to medical institutions with missing modalities, improves the consistency of medical target identification in cross-medical institution environments, and reduces the risk of medical target prediction errors caused by data heterogeneity.

[0019] 2. This invention utilizes a dynamic temperature prototype comparison mechanism to optimize cross-modal similarity of existing contrastive learning by using dynamic temperature factors related to class frequency. This aligns the representations of different modalities with the complete prototypes into a unified semantic space, reducing intra-class distance and increasing inter-class distance, effectively solving the discrimination problem in class imbalance scenarios.

[0020] 3. The prototype-aware aggregation algorithm proposed in this invention jointly models the local features of medical institutions and the global prototype, fully capturing the shared semantic structure across modalities and medical institutions. This not only reduces the information loss caused by modality missingness, but also enhances the generalization ability of the global prototype in heterogeneous data environments. Overall, it achieves more accurate multimodal medical data analysis and assisted diagnosis. The prediction results can be used to assist doctors in making clinical decisions or as a diagnostic reference. Attached Figure Description

[0021] Figure 1 This is the framework of the multimodal medical goal prediction method based on the federated prototype under modality missing conditions of this invention.

[0022] Figure 2 This is a flowchart of the multimodal medical target prediction method based on the federated prototype under modality missing conditions according to the present invention. Detailed Implementation

[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings and examples.

[0024] Multimodal federated learning aims to facilitate collaborative training of a shared global model among multiple healthcare institutions, leveraging medical data from different clients while ensuring data privacy. In this invention, we consider data with private multimodal characteristics. Source One client, of which Indicates the first The first patient's Medical data in multiple modalities Indicates the first The medical goal assessment results for each patient , This represents the total number of modes. In the case of a lack of modes, The combination of modes leads to This invention considers three possible types of clients, as each client may possess data corresponding to a unique subset of patterns. For example, in medical application scenarios with medical images and electronic medical records, there are three types of medical institutions: those lacking image data, those lacking electronic medical records, and those possessing both medical image and electronic medical record data. Furthermore, this invention considers different missing rates in medical institutions possessing both medical image and electronic medical record data.

[0025] In this invention, private multimodal data refers to the private data of a specific client. Missing modality refers to... Each client may lack at least one modality of data, or have all modalities of data, but at least one modality is missing. Multimodal medical data consists of at least two of the following: electronic medical records, medical images, and physiological signals. Standardized medical preprocessing, including image normalization, structured coding of medical records, and filtering of physiological signals, is required before inputting the data into the model to ensure consistency in medical semantics across different medical institutions.

[0026] In multimodal federated learning frameworks, efficient embedding of multimodal medical data features is necessary to achieve better model training and inference performance, especially in scenarios where some modalities are missing. In medical institutions, embedding vectors for feature representation are obtained through an embedding layer network, and then these embedding vectors are clustered to generate prototypes. This invention presents a multimodal medical auxiliary diagnosis method based on federated prototypes under modality missing conditions, mainly making the following three improvements: (1) Prototype-aware Aggregation (PAA). This method jointly models the local features of the client and the global prototype, fully capturing the shared semantic structure across modalities and clients. This aggregation strategy not only alleviates information loss caused by modality absence but also enhances the generalization ability of the global prototype in heterogeneous data environments.

[0027] (2) Adaptive Modal Weight Prototype Regularization (AMWPR). By shortening the distance between the shared representation of local modalities and the complete prototype, this method effectively alleviates the representational differences between local and global models. It introduces modal contribution as an adaptive weight, enabling different modalities to dynamically adjust according to their semantic value during the regularization process. This achieves more efficient knowledge transfer between the complete prototype and the missing modal client, improving the robustness of the model.

[0028] (3) Dynamic Temperature Prototype Comparison (DTPC). This method leverages complete prototypes to optimize the similarity of cross-modal features in existing contrastive learning methods. A dynamic temperature factor related to class frequency is introduced, allowing representations of specific modalities to more closely approximate similar prototypes while distancing themselves from dissimilar ones. A projection layer maps modality-specific representations to complete prototypes in a unified latent space, achieving consistent semantic representation across modalities and improving discriminative ability in imbalanced class scenarios.

[0029] The specific process of this invention is as follows: Figure 1 and Figure 2 As shown, the description is as follows: First, the original missing multimodal medical data (including electronic medical record text, medical image data, and physiological signals) is input into a modal encoder to obtain single-modal features. After extracting the single-modal features, they can be input into a recurrent neural network module to capture temporal physiological feature changes within and between different patients. Next, the interaction and fusion of multimodal information is achieved through a cross-modal attention module to obtain multimodal fusion features. Finally, the multimodal fusion features are mapped to a shared space for representing unified features to obtain multimodal features. In this space, data of patients with similar characteristics are clustered as closely as possible, and data of patients with dissimilar characteristics are separated as much as possible, thereby calculating the prototype.

[0030] In embodiments of the present invention, the extraction of single-modal features can be independently completed by a single-modal embedding network, and the specific implementation consists of three steps: Step (1): Encoding and feature extraction based on modal characteristics.

[0031] For different physiological signal modalities (such as electrocardiograms, electroencephalograms, blood pressure waveforms, and respiratory curves), appropriate modal encoders are selected or designed. For example, convolutional neural networks or temporal convolutional networks can be used for one-dimensional time series signals, Transformer encoders can be used for multi-dimensional time series signals, and ResNet or other image encoders can be used for image modalities (such as ultrasound images). The encoder output is a high-dimensional representation of the original input, i.e., single-modal features. This process aims to preserve key physiological and pathological information in each modality while removing noise and redundancy.

[0032] Step (2): Recurrent neural networks capture hierarchical temporal dependencies.

[0033] The feature sequences extracted from each modality (either time-step aligned or unaligned) are input into independent recurrent neural networks (such as LSTM, GRU, or bidirectional LSTM). RNNs not only model long-term and short-term physiological dynamics within the same patient (such as heart rate variability and respiratory rhythm fluctuations), but also implicitly learn common temporal evolution patterns among different patients through shared parameters or cross-sample training. Through memory units and gating mechanisms, the network can effectively mitigate the vanishing / exploding gradient problem and exhibits robustness to irregularly sampled clinical data.

[0034] Step (3): Align the shared space projection with the prototype.

[0035] The hidden state sequence of each RNN output (or the hidden state at the last time step) is projected onto a shared embedding space through a learnable linear transformation or a multilayer perceptron. In this space, an ideal representation after multimodal fusion or a global template learned from health-disease samples is introduced as an alignment target. By minimizing the difference (e.g., cosine distance or mean squared error) between each unimodal projection vector and this prototype, each modality's representation is forced to retain its own specificity while maintaining semantic consistency with the complete multimodal information. This ultimately yields aligned unimodal feature vectors that can be used for subsequent multimodal fusion, providing a semantically unified and time-sensitive representation foundation for multimodal fusion.

[0036] In embodiments of the present invention, to ensure alignment between local client training and the global prototype, multimodal fusion features from different clients need to be projected into the unified feature sharing space. Specifically, this is achieved by fusing the individual single-modal features using a multimodal fusion network based on a cross-modal attention module. The multimodal features are then projected onto a shared space to obtain the multimodal features. , means as follows: in, This represents the shared space between different clients. This indicates a projection operation.

[0037] The local prototype is the mean of the projected multimodal feature representations of patient samples under the same client and the same medical target category. It can be regarded as the initial centroid of that category in the local feature space. It is worth noting that for a single client, the local prototype in its feature space is not unique. Depending on different medical target categories, the same client may have multiple local prototypes. Each local prototype represents the feature distribution center of the corresponding category of patient samples in that client. The local prototype categories for different clients are defined as follows: in, Indicates the first The first client Local prototype of the category Indicates the first The first client The dataset is categorized, where the category refers to the assessment result of the medical goal, such as low risk, medium risk, high risk, or disease positive, disease negative, etc.

[0038] In federated learning, traditional aggregation algorithms mostly rely on the amount of patient data for weighting. However, in multimodal scenarios, the distribution of medical institution data is heterogeneous and may contain semantic biases. Relying solely on the amount of patient data is insufficient to accurately reflect the contribution of medical institution training results to the global model. To make the aggregation process more aligned with the semantic consistency requirements of the model and improve the performance of the global model, a prototype-based semantic consistency aggregation algorithm is presented in this embodiment of the invention as follows.

[0039] First, calculate the similarity between the global prototype and the local prototype: in, Indicates the first The degree of semantic consistency among clients Indicates the first The client contains a set of medical target categories; Indicates the first The first client The local prototype of the category; Indicates the first in the global prototype The global prototype of the category; This represents the cosine similarity function. This invention uses the calculated similarity between the global prototype and the local prototype as the degree of semantic consistency to begin iterative training.

[0040] Subsequently, the aggregated semantic consistency level yields the parameters for the next round of global model construction, building the global prototype for the server. Specifically, the sum of the semantic consistency levels of all clients participating in the aggregation is normalized: in, Indicates the first Normalized semantic consistency aggregation weights for each client. Indicates the first The degree of semantic consistency among clients.

[0041] The next round of global model parameters obtained after aggregation are calculated as follows: in, This represents the total number of clients participating in model parameter aggregation. Indicates the first The model parameters obtained from the first round of training, Indicates the first A collection of client patient samples, This indicates the number of patient samples in the set; This represents the sum of all client samples. The final global prototype of the server is constructed using the following formula: Indicates the first generation built by the server. Class global prototype, ,in Indicates the total number of categories. This represents the global prototype set consisting of all global prototypes of all categories.

[0042] A new global model and global prototype will be sent to each client, initiating a new round of iterations until the global model converges or reaches a preset number of global iterations. Finally, the client saves the global model and uses it to predict medical objectives based on missing modality data, such as disease-aided diagnosis, disease risk level prediction, prognostic risk prediction, hospitalization risk prediction, readmission risk prediction, and treatment effect evaluation.

[0043] For medical institutions lacking modalities, diverse modal knowledge can be obtained from the global prototype through single modalities. The modality-specific attributes of patient data within a local medical institution should be highly correlated with the corresponding complete prototype within the same category, and should reflect the differences between the complete prototypes associated with different semantic categories. In embodiments of this invention, for the local training process of different medical institutions under modality-deficient conditions, the client not only needs to complete the medical target classification task, but also needs to maintain semantic consistency between local features and the global prototype in the shared representation space. Therefore, this invention reduces the total training loss of the client... Designed to use cross-entropy loss Adaptive modal weighted prototype regularization loss and dynamic temperature prototype contrast loss The loss consists of three parts, each characterized by a function, and its overall form is as follows: Specifically, the cross-entropy loss function Also known as classification loss, it is used to constrain the model's prediction results for medical target categories; This represents the adaptive modality-weighted prototype regularization loss function, used to constrain the distance between different modal features and the corresponding global prototype of the class. This represents the dynamic temperature prototype contrast loss function, used to enhance the similarity between similar samples and their corresponding prototypes, and to expand the discriminative power between prototypes of different categories. , The hyperparameter weights are used to adjust the influence of the prototype regularization term and the prototype comparison term on the total loss.

[0044] Specifically, the cross-entropy loss function The definition is as follows: in, Indicates the total number of medical target categories. Indicates the true label in the first place Values ​​on the class This indicates that the model predicts the sample belongs to the first... The probability of a class.

[0045] To fully utilize existing modal information under modal missing conditions, this invention further introduces an adaptive modal weighted prototype regularization loss. For the... The first client The first patient's For each modality, the semantic contribution of that modality is first calculated based on the attention mechanism: in, Indicates by the first The query vector obtained by mapping modal features This represents the key vector obtained from the feature mapping of this modality. This represents the feature dimension of a single attention head. Considering the possibility of missing modalities, a modality missing mask is introduced. When this mode exists When this mode is missing Therefore, the first The final adaptive weights for each modality can be expressed as: in, Indicates the first The first client The first patient Attention scores for each modality Indicates the first The first client The first patient A mask for the absence of a modality. This represents the query vector obtained by mapping the features of this modality. This represents the key vector obtained from the feature mapping of this modality. Indicates the first The first client The first patient The final adaptive weights for each modality, This is the scale scaling factor. The feature dimension of a single attention head. This indicates the number of attention heads. Using this weight, the model can automatically reduce the influence of missing or low-contribution modes during training, while increasing the role of effective modes in prototype constraints.

[0046] Based on this, adaptive modality-weighted prototype regularization loss Defined as: in, Indicates the first The final adaptive weights for each modality, Indicates the first Multimodal features of a class This indicates the category of the patient's actual medical goals. The corresponding global prototype. This loss term enables local unimodal or incomplete multimodal representations to approach the complete global prototype of the corresponding category, thereby achieving knowledge transfer from the global prototype to the missing modality client.

[0047] Furthermore, to alleviate the common category imbalance problem in medical data, this invention introduces dynamic temperature prototype contrast loss. For the first... The dynamic temperature parameter for each medical target category is defined as follows: Among them, among them, Indicates the first Sample frequency of class Let be the initial temperature, representing the th Dynamic temperature parameters corresponding to each medical target category This represents a hyperparameter used to adjust the degree of influence of category frequencies. The initial temperature is a preset hyperparameter in contrastive learning, used to control the smoothness of the similarity distribution. In practical applications, The parameters can be set based on validation set results or empirical values. The lower the frequency of class samples, the smaller the corresponding temperature parameter, which makes the model pay more attention to the discrimination boundary of minority class samples during comparative learning.

[0048] Dynamic temperature prototype contrast loss The definition is as follows: in, express and similarity, It is a similarity scaling factor. It is a single-modal feature used for fusion. The similarity function can be represented by cosine similarity. This loss term improves the model's discriminative ability in scenarios with class imbalance and modality loss by narrowing the distance between sample features and global prototypes of the same class, and widening the distance between them and global prototypes of different classes.

[0049] Taking a joint cardiovascular disease risk prediction effort by three medical institutions as an example, there are three client applications, denoted as client applications. Client and client Medical goals are categorized into three levels: low-risk, medium-risk, and high-risk. Multimodal medical data includes three modalities: electronic medical records, electrocardiogram signals, and medical images. .

[0050] Among them, the client A medical institution that simultaneously possesses electronic medical records, electrocardiogram signals, and medical imaging data can be considered to have a relatively complete modality; Client Due to equipment limitations, only electronic medical records and electrocardiogram (ECG) signal data are available, lacking medical imaging modalities; client-side... Because some examinations were not conducted, only electronic medical records and medical imaging data were available, lacking ECG signal modalities. Missing modalities were marked using a modality missing mask during training. For example, when a patient has both electronic medical records and ECG signals but lacks medical imaging, their modality missing mask can be represented as: The first element indicates the presence of the electronic medical record modality, the second element indicates the presence of the electrocardiogram signal modality, and the third element indicates the absence of the medical imaging modality.

[0051] During the local training phase, each client first extracts features using its corresponding modal encoder. For example, electronic medical record data can be used to extract medical record features through a text encoder or a structured feature encoder, electrocardiogram signals can be used to extract temporal physiological features through a one-dimensional convolutional neural network or a recurrent neural network, and medical images can be used to extract image features through a convolutional neural network. Subsequently, the client inputs the existing modal features into the cross-modal attention module for fusion and projects them onto a shared representation space to obtain the patient's multimodal fused features.

[0052] For the client Because of its relatively complete modalities, its local model can learn more comprehensive multimodal representations and calculate three local prototypes based on low-risk, medium-risk, and high-risk patient samples respectively. For the client... and client Although modalities are missing, local prototypes for the corresponding categories can still be calculated based on existing modalities. After receiving the model parameters and local prototypes uploaded by each client, the server calculates the semantic consistency weight based on the cosine similarity between the local prototype and the global prototype, and uses this weight to complete the aggregation of global model parameters.

[0053] For example, in a certain round of federated training, if the client If the local prototype has a high mean cosine similarity to the current global prototype, it indicates that the client's local representation is relatively consistent with the global semantic space, and therefore its model update receives higher weight during aggregation; if the client If the lack of medical images leads to a significant shift in the representation of a certain category, its semantic consistency weight will be relatively reduced. In this way, the server considers not only the amount of data from each client, but also the semantic consistency between their local representation and the global prototype, thereby reducing the adverse effects of heterogeneous data and modality loss on the global model.

[0054] During the loss function optimization process, for clients lacking medical images... The missing mask for the medical imaging modality is 0, therefore this modality will not participate in the adaptive modality weight calculation. The model automatically assigns weights based on the attention scores of electronic medical records and electrocardiogram signals, allowing modalities that contribute more to play a more important role in prototype regularization. Simultaneously, the dynamic temperature prototype contrast loss adjusts the temperature parameter according to the frequency of low-risk, medium-risk, and high-risk samples. For example, if the number of high-risk patient samples is small, the temperature parameter corresponding to the high-risk category is smaller, causing the model to pay more attention to the alignment relationship between high-risk samples and the high-risk global prototype during training, thereby improving the recognition ability of a few categories.

[0055] After multiple rounds of federated iterative training, the server obtains the final global model and global prototype, which are then distributed to various medical institutions. Each client, after saving this global model, can predict cardiovascular disease risk for locally located patient data with missing modalities without uploading the original patient data. The final output can be low-risk, medium-risk, or high-risk, used to assist doctors in clinical risk assessment and treatment decisions.

Claims

1. A multimodal healthcare goal prediction method based on a federated prototype under modality missing conditions, characterized in that, Includes the following steps: Step 1: Construct a missing multimodal medical dataset and extract unimodal features; the data in the dataset is private multimodal data. , originating from N One client, of which Indicates the first The first patient's Medical data in multiple modalities Indicates the first The labels corresponding to the medical goal assessment results of each patient. , The total number of modes; Step 2: Fuse the single-modal features and project them into a shared space to obtain multimodal features, and calculate the local prototype; Step 3: Calculate the similarity between the global prototype and the local prototype as the degree of semantic consistency, start iterative training, aggregate the semantic consistency degree to obtain the global model parameters for the next round, and construct the global prototype on the server. Step 4: Send the new global model and global prototype to each client, and iteratively execute Step 3 until the global model converges or reaches the preset number of global iterations; Step 5: The client saves the global model and uses this global model to predict medical objectives based on the missing modality data.

2. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, In step 1, private multimodal data refers to the private data of a specific client; the missing modality refers to the... In each client, at least one modality of data is missing, or all modality data are present, but at least one modality of data is missing to some extent.

3. The multimodal medical goal prediction method based on a federated prototype under modality missing conditions according to claim 1 or 2, characterized in that, The multimodality includes: At least two of the following: electronic medical records, medical images, and physiological signals; The medical goal predictions include disease-aided diagnosis, disease risk level prediction, prognostic risk prediction, hospitalization risk prediction, readmission risk prediction, and treatment effect evaluation.

4. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, The extraction of unimodal features is achieved through a unimodal embedding network, as follows: Based on different modalities, select the appropriate modal encoder to extract single-modal features; Each single-modal feature is input into a recurrent neural network to capture temporal physiological feature changes within an individual patient and between different patients; The outputs of each recurrent neural network are projected into a shared space to align the single-modal features with the complete prototype, resulting in the final single-modal features used for fusion.

5. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, In step 2, the single-modal features are fused together using a multimodal fusion network based on a cross-modal attention module to obtain fused features. The multimodal features are then projected onto a shared space to obtain the multimodal features. , means as follows: in, This represents the shared space between different clients. Indicates a projection operation; The local prototype is the mean of the projected multimodal feature representations of patient samples under the same client and the same medical target category. The local prototype categories for different clients are defined as follows: in, Indicates the first The first client Local prototype of the category Indicates the first The first client The dataset is categorized, where the category refers to the assessment results of medical goals.

6. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, In step 2, for a single client, the local prototype in its feature space is not unique. The same client forms multiple local prototypes according to different medical target categories. Each local prototype is used to represent the feature distribution center of the corresponding category of patient samples in the client.

7. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, Step 3, calculating the similarity between the global prototype and the local prototype, is as follows: in, Indicates the first The degree of semantic consistency among clients Indicates the first The client contains a set of medical target categories; Indicates the first The first client The local prototype of the category; Indicates the first in the global prototype The global prototype of the category; Represents the cosine similarity function; The aggregation yields the global model parameters for the next round, and a global prototype for the server is constructed. The implementation method is as follows: First, the sum of the semantic consistency levels of all clients participating in the aggregation is normalized: in, Indicates the first Normalized semantic consistency aggregation weights for each client. Indicates the first The degree of semantic consistency among clients; Secondly, calculate the next round of global model parameters obtained after aggregation: in, This represents the total number of clients participating in model parameter aggregation. Indicates the first The model parameters obtained from the first round of training, Indicates the first A collection of client patient samples, This indicates the number of patient samples in the set; This represents the sum of all client samples. Finally, the global prototype of the server is constructed using the following formula: Indicates the first generation built by the server. Class global prototype, ,in Indicates the total number of categories. This represents the global prototype set consisting of all global prototypes of all categories.

8. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 1, characterized in that, The following loss function is used in the training: in, , For hyperparameter weights, Represents the cross-entropy loss function. This represents the adaptive modal weighted prototype regularization loss function. The dynamic temperature prototype contrast loss function is expressed as follows: in, Indicates the true label in the first place The probability of a class The model predicts the first Class probability, Indicates the first The final adaptive weights for each modality, Indicates the first Multimodal features of a class Represents the prototype set after projection. express and similarity, It is a similarity scaling factor. It is a single-modal feature used for fusion.

9. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 8, characterized in that, No. The final adaptive weights for each modality are expressed as follows: in, Indicates the first The first client The first patient Attention scores for each modality Indicates the first The first client The first patient A mask for the absence of a modality. This represents the query vector obtained by mapping the features of this modality. This represents the key vector obtained from the feature mapping of this modality. Indicates the first The first client The first patient The final adaptive weights for each modality, This is the scale scaling factor. The feature dimension of a single attention head. This indicates the number of heads of attention.

10. The multimodal medical goal prediction method based on federated prototype under modality missing conditions according to claim 8, characterized in that, The similarity scaling factor Represented as in, Indicates the first Sample frequency of class Let be the initial temperature, representing the th Dynamic temperature parameters corresponding to each medical target category This represents a hyperparameter used to adjust the degree of influence of category frequencies. The initial temperature is a preset hyperparameter in contrastive learning, used to control the smoothness of the similarity distribution. In practical applications, It can be set based on the validation set results or empirical values.

Images