A medical image reconstruction method under missing modalities

Abstract

The present invention discloses a method for medical image reconstruction under missing modalities, comprising the steps of acquiring multimodal data comprising low-dose PET images, MRI images, and patient electronic health record data, inputting the data into a medical image reconstruction model, and obtaining a reconstruction result. The medical image reconstruction model comprises an encoder-decoder network and an adversarial neural network. The encoder-decoder utilizes a modality feature extraction encoder to extract features from the multimodal data and inputs the extracted features into a modality interaction encoder. The modality interaction encoder embeds a loss-aware transformer that receives loss-aware cues from a loss-aware module, and the multiple loss-aware transformers process the features of the multimodal data, interacting and integrating them. The modality fusion decoder integrates the features of each modality to synthesize a SPET image. The present invention effectively interacts incomplete multimodal data to synthesize high-quality SPET images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a medical image reconstruction method under missing modalities. Background Art

[0002] Positron emission tomography (PET) is a widely used nuclear imaging technique. However, radiation exposure during standard PET (SPET) imaging raises health concerns. To reduce radiation hazards, various methods have been proposed to synthesize SPET images from low-dose PET (LPET) images. Recent efforts aim to improve the quality of PET synthesis by using multimodal input. However, these multimodal methods are limited in two aspects. First, they require complete multimodal data for training and validation, which is not always practical in clinical practice. Second, they focus primarily on multimodal images, ignoring the valuable information provided by patients' electronic health records (EHRs), resulting in compromised results.

[0003] Currently, deep learning-based incomplete multimodal SPET image synthesis remains largely unexplored. On the other hand, several strategies have been developed to handle the case of incomplete modalities in other tasks, such as segmentation, diagnosis, and prognosis. Among them, a straightforward approach is to train separate models for all possible subsets of available modalities, but this is resource-intensive and time-consuming. Another intuitive approach is to synthesize the missing modalities and then perform image synthesis; however, its performance may be limited by the quality of the synthesis of the missing modalities. Therefore, recent attempts have focused on learning a unified model that is applicable to all states of the missing modalities; in this case, due to the different number of input modalities, how to effectively interact with multimodal features and simultaneously address the missing states of all different modalities remains a challenge. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a medical image reconstruction method under missing modalities, which effectively interacts incomplete multimodal data and synthesizes high-quality SPET images.

[0005] To achieve the above object, the present invention adopts a technical solution: a method for reconstructing medical images under missing modalities, comprising the steps of:

[0006] Acquire multimodal data from low-dose PET images, MRI images, and patient electronic health record data;

[0007] Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction results;

[0008] The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network serves as a generator and the adversarial neural network serves as a discriminator;

[0009] The encoder-decoder network includes a modal feature extraction encoder, a modal interaction encoder, and a modal fusion decoder connected in sequence, and the processing process includes the following steps:

[0010] The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images, and patient electronic health record data, and the extracted features are input into the modality interaction encoder;

[0011] A modality interaction encoder is used to embed a loss-aware transformer with loss-aware cues obtained from the loss-aware module. Multiple loss-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities.

[0012] The modality fusion decoder is used to integrate the features of each modality to generate high-quality PET images.

[0013] Furthermore, the goal of the modality feature extraction encoder is to extract information from multiple modalities, including two image encoders and one text encoder;

[0014] The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data;

[0015] Use the image encoder to generate two compressed feature maps f LPET and f MRI ;

[0016] Use the text encoder to generate text features f EHR .

[0017] Furthermore, the modality interaction encoder includes a missing modality awareness module and a missing awareness transformer; before being fed into these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .

[0018] Furthermore, the missing modality perception module processing process includes the steps of:

[0019] The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ;

[0020] Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P;

[0021] Next, a pre-trained missing state classifier constructed by multi-layer perceptron (MLP) is introduced to predict the missing modality state of P, taking the cue P as the missing-aware cue.

[0022] Furthermore, the core of the loss-aware transformer is the loss-aware attention module, and the processing process includes the following steps:

[0023] The image feature Z LPET 、Z MRI or text feature Z EHR Project it into query Q, key K and value I of the missing-aware transformer;

[0024] Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively;

[0025] Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET 、Z * MRI or Z * EHR .

[0026] Furthermore, Z * LPET 、Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.

[0027] Furthermore, the adversarial neural network takes a pair of images, namely a low-dose PET image and a corresponding real or estimated SPET image, as input to distinguish between real pairs and synthetic pairs.

[0028] The beneficial effects of adopting this technical solution are:

[0029] This paper proposes a medical image reconstruction framework that utilizes multimodal images and patient electronic health record data for high-quality SPET synthesis of incomplete multimodal data. Specifically, the framework uses a modality feature extraction encoder and a modality fusion decoder to extract modality-specific features and integrate these multimodal features into the SPET synthesis.

[0030] To promote interactions between available modalities while being robust to missing modalities, we introduce Missing-Aware Transformers (MATs) in the modality fusion encoder to encourage effective interactions between available modalities. Considering the sensitivity of vanilla attention to missing tokens / inputs, unlike previous methods that directly fuse inter-modality features via cross-attention, we design missing-aware cues generated by a Missing Modality Perception (MMP) module. This module incorporates missing modality states and available modality interactions. We first design missing-aware cues involving missing modality states and available modality interactions. These cues are then injected into MATs to establish stable dependencies between modalities and enhance flexibility in learning from incomplete modalities. Finally, a modality fusion decoder is embedded to integrate multimodal features to generate high-quality SPET images.

[0031] In addition, the present invention also incorporates an adversarial training strategy to discern the authenticity of the estimated image, thereby encouraging the extraction of more discriminative features to further improve the synthesis quality and enhance the fidelity of the synthesized SPET images.

[0032] Evaluations on a public dataset of 160 patients and an internal dataset of 16 patients demonstrate the superiority of the proposed method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the principle of a medical image reconstruction method under missing modalities of the present invention;

[0034] Figure 2 A visualization diagram of the results of the multimodal learning method in an embodiment of the present invention;

[0035] Figure 3 It is a qualitative comparison diagram of the PET synthesis method in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings.

[0037] In this embodiment, see Figure 1 As shown, the present invention proposes a medical image reconstruction method under missing modality, comprising the steps of:

[0038] Acquire multimodal data including low-dose PET images (LPET), MRI images, and patient electronic health record (EHR) data;

[0039] Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction results;

[0040] The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network serves as a generator and the adversarial neural network serves as a discriminator.

[0041] The encoder-decoder network includes a modal feature extraction encoder, a modal interaction encoder, and a modal fusion decoder connected in sequence, and the processing process includes the following steps:

[0042] The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images, and patient electronic health record data, and the extracted features are input into the modality interaction encoder;

[0043] A modality interaction encoder is used to embed the missing-aware transformer (MAT) with missing-aware cues obtained from the missing-aware (MMP) module. Multiple missing-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities.

[0044] The modality fusion decoder is used to integrate the features of each modality to generate high-quality SPET images.

[0045] As an optimization solution of the above embodiment, the goal of the modality feature extraction encoder is to extract information from multiple modalities, including two image encoders and one text encoder;

[0046] The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data;

[0047] Use the image encoder to generate two compressed feature maps f LPET and f MRI ;

[0048] Use the text encoder to generate text features f EHR .

[0049] Specifically, the image encoder consists of five stages, each with two convolutional blocks with a GroupNorm-ReLU-3x3 convolution structure. A convolutional layer with a stride of 2 is injected between each two stages for downsampling. After each stage, the feature channels are doubled and the feature size is halved. The BERT model is used as the text encoder to generate text features.

[0050] As an optimization solution of the above embodiment, the modality interaction encoder includes a missing modality perception module and a missing perception transformer; before feeding into these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .

[0051] The missing modality perception module processing process includes the following steps:

[0052] The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ;

[0053] Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P;

[0054] Next, a pre-trained missing state classifier constructed by a multi-layer perceptron (MLP) is introduced to predict the missing modality state of P; this process in turn injects knowledge about the missing state into P, ensuring a more flexible and robust synthesis; the cue P is used as a missingness-aware cue.

[0055] Although the transformer effectively captures the long-range dependencies of cross-modal feature interactions, directly concatenating and projecting features from different modalities into the query, key, and value for attention computation is very sensitive to missing modalities. Through cue learning with the inherent ability to guide model predictions, we propose to guide multimodal feature interactions based on missing-aware cues in different input cases of missing modalities.

[0056] The core of the missing-aware transformer is the missing-aware attention (MAA) module, which promotes effective interaction between available modalities and ensures robustness to missing modalities by leveraging missing-aware cues P generated by the missing-modality awareness module. The processing process includes the following steps:

[0057] The image feature Z LPET 、Z MRI or text feature Z EHRProject it into query Q, key K and value I of the missing-aware transformer;

[0058] Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively;

[0059] Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET 、Z * MRI or Z * EHR .

[0060] Furthermore, following the standard transformation, a multi-head attention mechanism is adopted to mine more comprehensive long-term interactions.

[0061] Z * LPET 、Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.

[0062] The decoder also contains five stages, each with two convolutional blocks, and convolutional layers with GroupNorm, ReLU, and kernel size 3; between the two stages, upsampling layers are used to enlarge the feature maps; skip connections are also added between the last three stages of the encoder and decoder to preserve more low-level details.

[0063] As an optimization solution of the above embodiment, the adversarial neural network takes a pair of images, namely a low-dose PET image and a corresponding real or estimated SPET image, as input to distinguish between real pairs and synthetic pairs.

[0064] The adversarial neural network is trained on pairs of real LPET images and real SPET images, or pairs of fake LPET images and estimated SPET images. Specifically, the adversarial neural network can be an AdvNet consisting of five blocks; each with a 3x3x3 convolutional layer, a batch normalization layer, and a LeakyReLU activation layer. A sigmoid activation function is used to distinguish fake from real image pairs.

[0065] The advantages of the present invention are described below through experimental analysis:

[0066] Datasets and Evaluation Metrics: In this study, two datasets were used for training and evaluation: an in-house real-world human brain dataset and the publicly available UdpetX50 dataset. Specifically, the real-world human brain dataset scanned using the system consists of 16 samples, evenly divided between normal controls (NC) and mild cognitive impairment (MCI). Each sample has one LPET image, one MRI image, and one SPET image. SPET images were acquired with a standard dose of [18F]FDG, while LPET images were acquired at a quarter of the standard dose. A leave-one-out cross-validation strategy was employed to enhance model stability. For the UdpetX50 dataset, 120 samples were selected for [18F]FDG PET imaging using visual scanning. Each sample had one SPET image and one LPET image at a 1 / 50 dose. These samples were divided into 90, 10, and 20 for training, validation, and testing, respectively. To expand the training sample, overlapping blocks of size 64x64x64 were extracted from each image. The peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and normalized mean square error (NMSE) were used for model evaluation.

[0067] Table 1

[0068]

[0069] Comparison Results: Our approach (M3PET) was compared with two representative incomplete multimodal learning methods, Former and A2FS. Table 1 shows the results, which show that M3PET achieved the best results across all evaluation criteria. Compared to A2FS, the PSNR and SSIM for NC subjects improved by 2.383 dB and 0.003, respectively. For MCI subjects, the PSNR and SSIM improved by 2.023 dB and 0.003, respectively.

[0070] In addition, the M3PET of the present invention was compared with two state-of-the-art PET synthesis methods, including LA-GANT (SPET synthesis using LPET and MRI) and PCC-GAN (SPET synthesis using only LPET). Figure 2 As shown in the figure, our method achieves the best PSNR for all missing modalities. Moreover, even compared to the original LA-GAN trained on all PET / MRI modalities (PSNR of 24.296 dB and SSIM of 0.982), our NC method still outperforms the former by 25.004 dB and 0.987, respectively, demonstrating that our method (M3PET) is effective even for PET synthesis based entirely on multi-modality.

[0071] The results of the proposed method and the comparison method are also visualized in Figure 3 As shown in the figure, compared with other comparison methods, our M3PET produces the best visual results across all available modalities, with the richest details and the smallest errors. In addition, to verify the generalization of our method, we compared the performance on the UdpetX50 dataset. Since this dataset does not contain MRI data, we only used LPET and EHR data for evaluation. It can be observed that our method also produces the best performance in all indicators, further demonstrating its generalization ability.

[0072] Ablation study: We study the contribution of each key component, namely Missing Aware Transformer (MAT), Missing Modality Perception (MMP), Modality Fusion, and AdvNet. This is done by gradually excluding one of them from our scheme M3PET. Note that excluding MAT means replacing MAT with a plain transformer, i.e., without loss-aware cues. In Table 2, we compare the performance of the four variants with M3PET by averaging over all possible cases of input modality. It can be seen that all key components contribute to the improvement in performance. In particular, it can be seen that removing MP and MAT leads to a significant drop in performance, validating their ability to effectively and flexibly capture multimodal interactions in the presence of incomplete modalities.

[0073] Table 2

[0074]

[0075] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for reconstructing medical images in a missing modality, characterized in that: Including steps: Acquire multimodal data from low-dose PET images, MRI images, and patient electronic health record data; Input the acquired multimodal data into the medical image reconstruction model to obtain the reconstruction results; The medical image reconstruction model includes: an encoder-decoder network and an adversarial neural network; after adversarial training, the encoder-decoder network serves as a generator and the adversarial neural network serves as a discriminator; The encoder-decoder network includes a modal feature extraction encoder, a modal interaction encoder, and a modal fusion decoder connected in sequence, and the processing process includes the following steps: The modality feature extraction encoder is used to extract features from low-dose PET images, MRI images, and patient electronic health record data, and the extracted features are input into the modality interaction encoder; A modality interaction encoder is used to embed a loss-aware transformer with loss-aware cues obtained from the loss-aware module. Multiple loss-aware transformers process the features of low-dose PET images, MRI images, and patient electronic health record data, respectively, to achieve feature interaction and integration between available modalities. The modality fusion decoder is used to integrate the features of each modality to generate high-quality SPET images.

2. The method for medical image reconstruction under missing modality according to claim 1, characterized in that: The goal of the modality feature extraction encoder is to extract information from multiple modalities, including two image encoders and one text encoder; The image encoder is used for inputting the image modalities of low-dose PET images and magnetic resonance MRI images, respectively, and the text encoder is used for inputting the text modality of patient electronic health record data; Use the image encoder to generate two compressed feature maps f LPET and f MRI ; Use the text encoder to generate text features f EHR .

3. The method for medical image reconstruction under missing modality according to claim 2, characterized in that: The modality interaction encoder includes a missing modality awareness module and a missing awareness transformer; before feeding into these two modules, the image input f LPET and f MRI The modal features are flattened to Z LPET and Z MRI ; Text feature f EHR Projected to Z EHR .

4. The method for medical image reconstruction under missing modality according to claim 3, characterized in that: The missing modality perception module processing process includes the following steps: The features from the three modalities of the modal interaction encoder are combined by cascading to produce a multimodal embedding f multi ; Then, f multi Elements are added with learnable position embeddings and processed by a transformer-based prompt generator to capture long-range dependencies between modalities, creating prompts P; Next, a pre-trained missing state classifier constructed by multi-layer perceptron (MLP) is introduced to predict the missing modality state of P, taking the cue P as the missing-aware cue.

5. The method for medical image reconstruction under missing modality according to claim 4, characterized in that: The core of the loss-aware transformer is the loss-aware attention module, and the processing process includes the following steps: The image feature Z LPET 、Z MRI or text feature Z EHR Project it into query Q, key K and value I of the missing-aware transformer; Then, the attention level cue is introduced by pre-adding the absence-aware cue P to the key K and value V of the absence-aware transformer, respectively; Finally, through layer normalization and feed-forward network, the output Z is obtained * LPET 、Z * MRI or Z * EHR .

6. The method for medical image reconstruction under missing modality according to claim 5, characterized in that: Z * LPET 、Z * MRI and Z * EHR The input is the modality fusion decoder, which gradually recovers the estimated SPET image.

7. The method for medical image reconstruction under missing modality according to claim 1, characterized in that: The adversarial neural network takes as input a pair of images, a low-dose PET image and a corresponding real or estimated SPET image, to distinguish between real and synthetic pairs.

Images