Medical image reconstruction method in missing mode
By using an encoder decoder network and an adversarial neural network in medical image reconstruction, combining modal feature extraction and missing perception modules to process multimodal data, the interaction and integration problems of multimodal methods in missing mode are solved. High-quality PET image synthesis is achieved, improving the fidelity and robustness of the image.
Patent Information
- Application Number
- CN202410853331.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-06-28
AI Technical Summary
Existing multimodal methods are difficult to effectively interact and integrate multimodal data in the absence of modality, resulting in limited quality of PET image synthesis and neglecting the information provided by the patient's electronic health records, affecting the accuracy of the results.
Using encoder decoder network and adversarial neural network, low-dose PET images, NMR MRI images and patient electronics are processed through modal feature extraction, modal interactive encoder and modal fusion decoder, combined with missing perception modules and transformers. Health record data to generate high-quality SPET images.
High-quality interaction and integration of multimodal data under missing modes is achieved, the synthesis quality and fidelity of PET images are improved, the adversarial training strategy is enhanced, and the robustness and flexibility of the synthesis results are ensured.
Smart Images

Figure CN118898546A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to a medical image reconstruction method under missing modality. Background Art
[0002] Positron emission tomography (PET) is a widely used nuclear imaging technique. However, radiation exposure in standard PET (SPET) imaging raises health concerns. To reduce radiation hazards, various methods have been proposed to synthesize SPET images from low-dose PET (LPET) images. Recent efforts aim to improve the quality of PET synthesis by using multimodal inputs. However, these multimodal methods are limited in two aspects. First, they require complete multimodal data for training and validation, which is not always practical in clinical practice; second, they focus mainly on multimodal images and ignore the valuable information provided by patients' electronic health records (EHRs), resulting in compromised results.
[0003] Currently, deep learning-based incomplete multimodal SPET image synthesis remains largely unexplored. On the other hand, several strategies have been developed to handle the case of incomplete modalities in other tasks, such as segmentation, diagnosis, and prognosis. Among them, a straightforward approach is to train separate models for all possible subsets of available modalities, but this is resource-intensive and time-consuming. Another intuitive approach is to synthesize the missing modalities and then perform image synthesis; however, its performance may be limited by the quality of the missing modality synthesis. Therefore, recent attempts have focused on learning a unified model that is applicable to all states of the missing modality; in this case, due to the different number of input modalities, how to effectively interact with multimodal features while addressing the missing states of all different modalities remains a challenge. Summary of the invention
[0004] In order to solve the above problems, the present invention proposes a medical image reconstruction method under missing modality, which effectively interacts with incomplete multi-modal data and synthesizes high-quality SPET images.
[0005] To achieve the above object, the technical solution adopted by the present invention is: a method for reconstructing medical images under missing modalities, comprising the steps of:
[0006] Acquire multimodal data from low-dose PET images, MRI images, and patient electronic health record data;
[0007] Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction result;
[0008] The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network is used as a generator and the adversarial neural network is used as a discriminator;
[0009] The encoder-decoder network includes a modality feature extraction encoder, a modality interaction encoder and a modality fusion decoder connected in sequence, and the processing process includes the steps of:
[0010] The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images and patient electronic health record data, and the extracted features are input into the modality interaction encoder;
[0011] Using the modality interaction encoder, a loss-aware transformer with loss-aware cues obtained from the loss-aware module is embedded. Multiple loss-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities.
[0012] The modality fusion decoder is used to integrate the features of each modality to generate high-quality PET images.
[0013] Furthermore, the modality feature extraction encoder aims to extract information in multiple modalities, including two image encoders and one text encoder;
[0014] The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data;
[0015] Use the image encoder to generate two compressed feature maps f LPET and f MRI ;
[0016] Generate text features f using text encoder EHR .
[0017] Furthermore, the modality interaction encoder includes a missing modality awareness module and a missing awareness transformer; before being fed to these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .
[0018] Furthermore, the missing modality perception module processing process includes the steps of:
[0019] The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ;
[0020] Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P;
[0021] Next, a pre-trained missing state classifier constructed by multi-layer perceptron (MLP) is introduced to predict the missing modality state of P, taking cue P as the missing-aware cue.
[0022] Furthermore, the core of the loss-aware transformer is a loss-aware attention module, and the processing process includes the steps of:
[0023] The image feature Z LPET , Z MRI or text feature Z EHR Project it into query Q, key K and value I of missing-aware transformer;
[0024] Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively;
[0025] Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET , Z * MRI or Z * EHR .
[0026] Furthermore, Z * LPET , Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.
[0027] A method for medical image reconstruction under missing modality according to claim 1, characterized in that the adversarial neural network takes a pair of images, namely a low-dose PET image and a corresponding real or estimated SPET image, as input to distinguish between real pairs and synthetic pairs.
[0028] The beneficial effects of adopting this technical solution are:
[0029] The present invention proposes a medical image reconstruction framework that utilizes multimodal images and patient electronic health record data for incomplete multimodal high-quality SPET synthesis. Specifically, the medical image reconstruction framework of the present invention uses a modality feature extraction encoder and a modality fusion decoder to extract modality-specific features respectively and integrate these multimodal features into SPET synthesis.
[0030] In order to promote the interaction between available modalities while being robust to missing modalities, we introduce Missing-Aware Transformers (MATs) in the modality fusion encoder to encourage effective interactions between available modalities. Considering the sensitivity of vanilla attention to missing tokens / inputs, unlike previous methods that directly fuse inter-modality features through cross-attention, we design missing-aware cues generated by the Missing Modality Perception (MMP) module, which contains the missing modality states as well as the available modality interactions. We first design missing-aware cues involving the missing modality states and the available modality interactions. Then, we inject these cues into MATs to establish stable dependencies between modalities and enhance the flexibility of learning from incomplete modalities. Finally, we also embed a modality fusion decoder to integrate multimodal features to generate high-quality SPET images.
[0031] In addition, the present invention also incorporates an adversarial training strategy to discern the authenticity of the estimated image, thereby encouraging the extraction of more discriminative features to further improve the synthesis quality and improve the fidelity of the synthesized SPET images.
[0032] Evaluations on a public dataset with 160 patients and an internal dataset with 16 patients demonstrated the superiority of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A schematic diagram of the principle of a method for medical image reconstruction under a missing modality of the present invention;
[0034] Figure 2 A result visualization diagram of a multimodal learning method in an embodiment of the present invention;
[0035] Figure 3 It is a qualitative comparison diagram of the PET synthesis method in the embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.
[0037] In this embodiment, see Figure 1 As shown, the present invention proposes a method for reconstructing medical images under missing modalities, comprising the steps of:
[0038] Acquire multimodal data including low-dose PET images (LPET), MRI images, and patient electronic health record data (EHR);
[0039] Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction result;
[0040] The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network serves as a generator and the adversarial neural network serves as a discriminator.
[0041] The encoder-decoder network includes a modality feature extraction encoder, a modality interaction encoder and a modality fusion decoder connected in sequence, and the processing process includes the steps of:
[0042] The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images and patient electronic health record data, and the extracted features are input into the modality interaction encoder;
[0043] Using the modality interaction encoder, a missing-aware transformer (MAT) is embedded with missing-aware cues obtained from the missing-aware (MMP) module. Multiple missing-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities.
[0044] The modality fusion decoder is used to integrate the features of each modality to generate high-quality SPET images.
[0045] As an optimization solution of the above embodiment, the modality feature extraction encoder aims to extract information from multiple modalities, including two image encoders and one text encoder;
[0046] The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data;
[0047] Use the image encoder to generate two compressed feature maps f LPET and f MRI ;
[0048] Generate text features f using text encoder EHR .
[0049] Specifically, the image encoder contains five stages, each stage has two convolution blocks, and the structure is GroupNorm-ReLU-3x3 convolution; a convolution layer with a stride of 2 is injected between every two stages for downsampling; after each stage, the feature channel will be doubled and the feature size will be halved. The BERT model is used as a text encoder to generate text features.
[0050] As an optimization solution of the above embodiment, the modality interaction encoder includes a missing modality perception module and a missing perception transformer; before being fed to these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .
[0051] The missing modality perception module processing process includes the following steps:
[0052] The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ;
[0053] Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P;
[0054] Next, a pre-trained missing state classifier built by a multi-layer perceptron (MLP) is introduced to predict the missing modality state of P; this process in turn injects knowledge about the missing state into P, ensuring a more flexible and robust synthesis; and the cue P is used as a missingness-aware cue.
[0055] Although the transformer effectively captures the long-range dependencies of cross-modal feature interactions, directly concatenating and projecting features from different modalities into queries, keys, and values for attention computation is very sensitive to missing modalities. Through cue learning with the intrinsic ability to guide model predictions, we propose to guide multimodal feature interactions based on missing-aware cues in different input situations of missing modalities.
[0056] The core of the missing-aware transformer is the missing-aware attention (MAA) module, which promotes effective interaction between available modalities and ensures robustness to the missing modality by leveraging the missing-aware cues P generated by the missing-modality awareness module. The processing includes the following steps:
[0057] The image feature Z LPET , Z MRI or text feature Z EHRProject it into query Q, key K and value I of missing-aware transformer;
[0058] Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively;
[0059] Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET , Z * MRI or Z * EHR .
[0060] Furthermore, following the standard transformation, a multi-head attention mechanism is adopted to mine more comprehensive long-term interactions.
[0061] Z * LPET , Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.
[0062] The decoder also contains five stages, each with two convolutional blocks, with GroupNorm, ReLU, and convolutional layers with kernel size 3; between the two stages, upsampling layers are used to enlarge the feature maps; skip connections are also added between the last three stages of the encoder and decoder to preserve more low-level details.
[0063] As an optimization scheme of the above embodiment, the adversarial neural network takes a pair of images, namely a low-dose PET image and a corresponding real or estimated SPET image, as input to distinguish between real pairs and synthetic pairs.
[0064] The adversarial neural network is based on real pairs of LPET images and real SPET images or fake pairs of LPET images and estimated SPET images. Specifically, the adversarial neural network can adopt AdvNet, which consists of five blocks; each with a 3x3x3 convolution layer, a batch normalization layer, and a LeakyReLU activation layer. The S-shaped activation function is followed to distinguish fake image pairs from real image pairs.
[0065] The advantages of the present invention are explained below by experimental analysis:
[0066] Datasets and Evaluation Metrics: In this study, two datasets were used for training and evaluation, including an in-house real human brain dataset and a publicly available UdpetX50 dataset. Specifically, the real human brain dataset scanned using the system has 16 samples, in which normal control (NC) samples and mild cognitive impairment (MCI) samples are equally divided. Each sample has one LPET image, one MRI image, and one SPET image. A standard dose of [18F]FDG was injected to obtain SPET images, while LPET images were obtained at one-fourth of the standard dose. A leave-one-out cross-validation strategy was used to enhance model stability. For the UdpetX50 dataset, 120 samples were selected for [18F]FDG PET imaging, and each sample had one SPET and one LPET image at 1 / 50 dose using visual scanning. These samples were divided into 90, 10, and 20 for training, validation, and testing. To expand the training samples, overlapping blocks of size 64X64X64 were extracted from each image. Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and normalized mean square error (NMSE) were used for model evaluation.
[0067] Table 1
[0068]
[0069] Comparison results: This scheme (M3PET) was compared with two representative incomplete multimodal learning methods, Former and A2FS. The comparison results are shown in Table 1, from which it can be seen that the M3PET of the present invention achieved the best results in all evaluation criteria. Compared with A2FS, the PSNR and SSIM of NC subjects were improved by 2.383dB and 0.003 respectively. The PSNR and SSIM of MCI subjects were improved by 2.023dB and 0.003 respectively.
[0070] In addition, the M3PET of the present invention was compared with two state-of-the-art PET synthesis methods, including LA-GANT (SPET synthesis using LPET and MRI) and PCC-GAN (SPET synthesis using LPET only). Figure 2 As shown. It can be observed that our method achieves the best PSNR in all cases of missing modalities. In addition, even compared with the original LA-GAN trained on all PET / MRI modalities (PSNR is 24.296dB and SSIM is 0.982), the NC method of the present invention still outperforms the former, which are 25.004dB and 0.987 respectively, proving that the present invention (M3PET) is effective even for PET synthesis based entirely on multi-modality.
[0071] The results of the proposed method and the comparison method are also visualized in Figure 3 As shown. It can be seen that compared with other comparison methods. The present invention M3PET produces the best visual effects in various available modalities, with the richest details and the smallest error. In addition, in order to verify the generalization of our method, we compared the performance on the UdpetX50 dataset. Since this dataset does not contain MRI data, we only use LPET and EHR data for evaluation. It can be observed that our method also produces the best performance in all indicators, further demonstrating its generalization ability.
[0072] Ablation study: We study the contribution of each key component, namely Missing Aware Transformer (MAT), Missing Modality Aware (MMP), Modality Fusion, and AdvNet. By gradually excluding one of them from our scheme M3PET. Note that excluding MAT means replacing MAT with a normal transformer, i.e., without loss-aware cues. In Table 2, we compare the performance of the four variants with M3PET by averaging all possible cases of the input modality. It can be seen that all key components contribute to the improvement in performance. In particular, it can be seen that removing MP and MAT leads to a significant drop in performance, validating their ability to effectively and flexibly capture multimodal interactions in the case of incomplete modalities.
[0073] Table 2
[0074]
[0075] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A method for medical image reconstruction under missing modality, It is characterized in that Includes steps: Acquire multimodal data from low-dose PET images, MRI images, and patient electronic health record data; Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction result; The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network is used as a generator and the adversarial neural network is used as a discriminator; The encoder-decoder network includes a modality feature extraction encoder, a modality interaction encoder and a modality fusion decoder connected in sequence, and the processing process includes the steps of: The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images and patient electronic health record data, and the extracted features are input into the modality interaction encoder; Using the modality interaction encoder, a loss-aware transformer with loss-aware cues obtained from the loss-aware module is embedded. Multiple loss-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities. The modality fusion decoder is used to integrate the features of each modality to generate high-quality SPET images.
2. The method for medical image reconstruction under missing modality according to claim 1, It is characterized in that The goal of the modality feature extraction encoder is to extract information from multiple modalities, including two image encoders and one text encoder; The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data; Use the image encoder to generate two compressed feature maps f LPET and f MRI ; Generate text features f using text encoder EHR .
3. The method for medical image reconstruction under missing modality according to claim 2, It is characterized in that The modality interaction encoder includes a missing modality awareness module and a missing awareness transformer; before feeding into these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .
4. The method for medical image reconstruction under missing modality according to claim 3, It is characterized in that The missing modality perception module processing process includes the steps of: The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ; Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P; Next, a pre-trained missing state classifier constructed by multi-layer perceptron (MLP) is introduced to predict the missing modality state of P, taking cue P as the missing-aware cue.
5. The method for medical image reconstruction under missing modality according to claim 4, It is characterized in that The core of the loss-aware transformer is the loss-aware attention module, and the processing process includes the following steps: The image feature Z LPET , Z MRI or text feature Z EHR Project it into query Q, key K and value I of missing-aware transformer; Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively; Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET , Z * MRI or Z * EHR .
6. The method for medical image reconstruction under missing modality according to claim 5, It is characterized in that Z * LPET , Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.
7. The method for medical image reconstruction under missing modality according to claim 1, It is characterized in that The adversarial neural network takes as input a pair of images, a low-dose PET image and a corresponding real or estimated SPET image, to distinguish between real pairs and synthetic pairs.
Citation Information
Patent Citations
Medical image segmentation model establishment method capable of missing input and segmentation method
CN113496495A
Multi-modal medical image coding and generating method based on generative adversarial network
CN114119788A
Systems and methods for machine learning assisted cognitive assessment and therapy
CN116829050A
Head and neck tumor radiotherapy prognosis prediction system based on adaptive multi-modal information fusion
CN118098588A
Multi-modal machine learning to determine risk stratification
WO2023201054A1
Cited By
Mild cognitive impairment conversion prediction method based on incomplete three-mode common-attention fusion
CN119905256A
Mild cognitive impairment conversion prediction method based on incomplete three-modal co-attention fusion
CN119905256B
