Multi-MRI (Magnetic Resonance Imaging) sequence deletion interpolation method based on variational auto-encoder

Abstract

The invention relates to the technical field of MRI image processing, and discloses a multi-MRI sequence deletion interpolation method based on a variational auto-encoder. According to the method, the FSL software is used for performing registration processing on the multi-mode MRI image, so that the consistency of the image in space, position and voxel pitch is ensured, and the precision of subsequent analysis is improved. In combination with the minimum and maximum value normalization and slice screening technology, the data preprocessing process is optimized, the data quality is improved, and redundancy is reduced. An end-to-end pre-trained variational auto-encoder model is adopted, an accurate potential space mapping relation is constructed, and the image reconstruction capability is enhanced. And fine training is carried out by using the SSIM loss after weight adjustment and the perception loss, so that high-quality modal missing interpolation is realized, and the integrity and availability of the medical image are further improved. According to the method, the multi-modal medical image processing capability can be effectively improved, and accurate technical support is provided for medical image analysis and computer-aided diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of MRI image processing, and particularly relates to a method for multi-MRI sequence missing imputation based on variational autoencoders. Background Art

[0002] In modern medical imaging, magnetic resonance imaging (MRI) has been widely used due to its excellent soft tissue imaging ability. For example, during the diagnosis of glioblastoma, T1 and FLAIR sequences can clearly define the tumor and its surrounding edema area, while the T1 contrast-enhanced (T1ce) sequence provides a clear boundary of the tumor enhancement area. This information is of great significance for evaluating the growth or shrinkage of the tumor. In addition, the FLAIR sequence also plays a key role in the diagnosis of vascular dementia (VD) by detecting white matter hyperintensities.

[0003] However, in practical applications, due to factors such as different scanning protocols and motion artifacts, some modalities may be missing or unavailable during image acquisition. This not only affects the image quality but may also lead to deviations in diagnostic results. At the same time, this phenomenon poses challenges to many downstream data analyses because they usually assume the existence of a complete set of pulse sequences to support their operations. Therefore, methods that can effectively compensate for missing modality information are of great significance for the accurate analysis of medical images and clinical decision-making.

[0004] To improve the consistency of medical image data, optimize the data preprocessing process, and enhance the image reconstruction and repair capabilities, the present invention proposes a method for multi-MRI sequence missing imputation based on variational autoencoders. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a method for multi-MRI sequence missing imputation based on variational autoencoders. This method breaks through the limitations of existing image imputation techniques and uses a multi-pair generation method to efficiently synthesize the imputation of different missing parts of the MRI sequence with one model. The specific steps are as follows:

[0006] Step 1: Register the multi-modal medical image data of T1, T2, Fliar, and T1ce sequences.

[0007] Step 2: Perform min-max normalization on the registered multi-MRI sequences and slice the sequences with obvious brain images.

[0008] Step 3: Use a variational autoencoder model to pre-train a variational autoencoder that can compress and restore the multi-MRI sequence data.

[0009] Step 4: Separately extract the decoder part of the variational autoencoder obtained in Step 3.

[0010] Step Five: Use an untrained encoder to align it with the decoder part of the pre-trained variational autoencoder obtained in Step Three in the latent space;

[0011] Step Six: Use the SSIM and perceptual loss adjusted according to medical image features for fine-tuning training;

[0012] Step Seven: Input the missing data of the random MRI sequence into the model to obtain the result of random imputation.

[0013] Preferably, in Step One, the registration of the multi-MRI image sequences of T1, T2, Fliar, and T1ce sequences is specifically as follows:

[0014] Use FSL software to take the T2 sequence as the standard for the multi-MRI image sequences of T1, Fliar, and T1ce to obtain medical image sequences with consistent orientations, positions, voxel spacings, and spatial distributions.

[0015] Preferably, in Step Two, the steps of normalizing and slicing the obvious brain image sequences are as follows:

[0016] Use min-max normalization to proportionally compress the voxel point values of the medical image sequences within the range of [0 - 1];

[0017] Write a Python program to slice each medical image sequence layer by layer, remove the slices with voxel points less than 0.2, and save the remaining slices in the.npy format.

[0018] Preferably, in Step Three, the pre-training method of the variational autoencoder model is specifically as follows:

[0019] Input the four medical image sequence data of T1, T2, Fliar, and T1ce, output the reconstructed four medical image sequence data of T1, T2, Fliar, and T1ce, perform complete end-to-end training, and establish an initial latent space mapping relationship based on the variational autoencoder model.

[0020] Preferably, in Step Four, the method of separately extracting the decoder part of the variational autoencoder obtained in Step Three is specifically as follows:

[0021] Construct a separate decoder model, screen the weights of the variational autoencoder model extracted in Step Four, retain the decoder part and import it into the separate decoder.

[0022] Preferably, in Step Five, the latent space alignment method is specifically as follows:

[0023] Construct a new encoder model with the same structure as that in the variational autoencoder;

[0024] Input the data of four medical image sequences of T1, T2, Fliar, and T1ce with missing values into the new encoder model, and set the missing medical image sequences to 0. For example, if the T1 and T2 modalities are missing, the input sequence is {0, 0, Fliar, T1ce};

[0025] Use the MSE loss to align the encoder and decoder in the latent space dimension.

[0026] The tabular form of MSE is:

[0027]

[0028] where N is the total number of pixels in the image, x i and are the values of the original image and the reconstructed image at the i-th pixel position, respectively.

[0029] Preferably, in the sixth step, the refined training with the weight-adjusted SSIM and perceptual loss is specifically as follows;

[0030] Use the SSIM loss adjusted according to the weights of multi-MRI sequence data for training, and its formula is:

[0031]

[0032] where x is the original image, is the reconstructed image, S represents the total number of independent scan sequences included in the multi-sequence image (such as the T1 and T2 weighted sequences of MRI), is the structural similarity measure between the original image and the reconstructed image of the s-th scan sequence, and after independently calculating the structural similarity of each sequence and weighted fusion, it effectively balances the contribution degree of cross-sequence features;

[0033] Use the perceptual loss adjusted according to the weights of multi-MRI sequence data for training, and its formula is:

[0034]

[0035] where φ j (x) is the feature map of the j-th layer extracted by the pre-trained VGG16 network, and M is the number of selected feature layers. This function introduces the principle of independent calculation for multi-MRI sequences, calculates the perceptual loss for multi-MRI sequences separately and then performs weighted fusion, avoiding cross-modal feature confusion.

[0036] Preferably, the method in the seventh step is specifically as follows: Input the data with randomly missing modalities into the model to obtain the result of random imputation.

[0037] Compared with related technologies, the method for multi-MRI sequence missing imputation based on variational autoencoder provided by the present invention has the following beneficial effects:

[0038] 1. Through end-to-end pre-training and latent space alignment, the model of the present invention can more accurately learn the feature distribution of multi-modal medical images, improving the reconstruction quality.

[0039] 2. By combining the SSIM loss with perceptual loss with weight adjustment for refined training, the present invention realizes high-quality imputation of modality missing images, improving the integrity and usability of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of a method for multi-MRI sequence missing imputation based on variational autoencoder proposed by the present invention;

[0041] Figure 2 is the model structure diagram of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0042] The present invention will be further described below with reference to the drawings and embodiments.

[0043] The present invention proposes a method for multi-MRI sequence missing imputation based on variational autoencoder (as Figure 1 shown), which adopts a phased training strategy to fully learn cross-modal deep representations, and combines a random masking mechanism to simulate the missing situation of multi-modal data, thereby improving the adaptability of the model in the case of missing inputs. Specifically, the method includes two core stages: in the first stage, cross-modal deep representation learning is performed using complete modality samples; in the second stage, the model is trained through a random masking mechanism so that it can still generate high-quality synthetic images in the case of missing different modalities. In addition, in order to improve the authenticity and structural integrity of the generated images, the present invention introduces improved perceptual loss and structural similarity (SSIM) loss into the loss function to enhance the model's ability to retain image details.

[0044] The specific implementation of this method is as follows:

[0045] Use FSL software to standardize the multi-MRI image sequences of T1, Fliar, and T1ce with the T2 sequence to obtain a medical image sequence with consistent orientation, position, voxel spacing, and spatial distribution.

[0046] Use min-max normalization to compress the voxel point values of the medical image sequence proportionally within the range of [0-1].

[0047] Write a Python program to slice each layer of a medical image sequence, remove slices with voxel points less than 0.2, and save the remaining slices in the.npy format.

[0048] Input the data of four medical image sequences of T1, T2, Fliar, and T1ce, output the reconstructed data of the four medical image sequences of T1, T2, Fliar, and T1ce, and perform complete end-to-end training to establish an initial latent space mapping relationship based on the variational autoencoder model.

[0049] Further, in step 4, the specific method for separately extracting the decoder part of the variational autoencoder obtained in step 3 is as follows:

[0050] Construct a separate decoder model, screen the weights of the variational autoencoder model extracted in the previous step, retain the decoder part and import it into the separate decoder.

[0051] Construct a new encoder model with the same structure as that in the variational autoencoder;

[0052] Input the data of the four medical image sequences of T1, T2, Fliar, and T1ce with missing parts into the new encoder model, set the missing medical image sequences to 0. For example, if the T1 and T2 modalities are missing, the input sequence is {0, 0, Fliar, T1ce}.

[0053] Use the MSE loss to align the encoder and decoder in the latent space dimension.

[0054] The tabular form of MSE is:

[0055]

[0056] where N is the total number of pixels in the image, x i and are the values of the original image and the reconstructed image at the i-th pixel position respectively.

[0057] Use the SSIM loss adjusted according to the weights of multi-MRI sequence data for training, and its formula is:

[0058]

[0059] where x is the original image, is the reconstructed image, S represents the total number of independent scan sequences included in the multi-sequence image (such as the T1 and T2 weighted sequences of MRI), is the structural similarity measure between the original image and the reconstructed image of the s-th scan sequence, and after independently calculating the structural similarity of each sequence and weighted fusion, it effectively balances the contribution degree of cross-sequence features.

[0060] Training is performed using a perceptual loss adjusted according to the weights of multi-MRI sequence data, and its formula is:

[0061]

[0062] where φ j (x) is the j-th layer feature map extracted by the pre-trained VGG16 network, and M is the number of selected feature layers. This function introduces the principle of independent calculation for multi-MRI sequences, calculates the perceptual loss for multi-MRI sequences separately and then performs weighted fusion, avoiding cross-modal feature confusion.

[0063] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for multi-MRI sequence missing imputation based on variational autoencoder, characterized in that Including: Step 1: Register the multimodal medical image T1, T2, Fliar, and T1ce sequence data; Step 2: Perform min-max normalization on the registered multi-MRI sequences and slice the sequences with obvious brain images; Step 3: Use a variational autoencoder model to pre-train a variational autoencoder that can compress and restore multi-MRI sequence data; Step 4: Separately extract the decoder part of the variational autoencoder obtained in Step 3; Step 5: Use an untrained encoder to perform latent space alignment with the decoder part of the pre-trained variational autoencoder obtained in Step 3; Step 6: Perform fine-tuning training using SSIM and perceptual loss adjusted according to medical image features; Step 7: Input the missing data of the random MRI sequence into the model to obtain the result of random imputation.

2. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, wherein In the said Step 1, the registration of the multi-MRI image sequences T1, T2, Fliar, and T1ce sequence data is specifically as follows: 2.

1. Use FSL software to take the T2 sequence as the standard for the multi-MRI image sequences of T1, Fliar, and T1ce to obtain medical image sequences with consistent orientations, positions, voxel spacings, and spatial distributions.

3. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, wherein In the said Step 2, the steps for normalizing and slicing the sequence of the obvious brain image are as follows: 3.

1. Use min-max normalization to proportionally compress the voxel point values of the medical image sequence within the range of [0 - 1]; 3.

2. Write a Python program to slice each medical image sequence layer by layer, remove the slices with voxel points less than 0.2, and save the remaining slices in the.npy format.

4. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, characterized in that, In the said Step 3, the specific method for pre-training the variational autoencoder model is as follows: 4.

1. Input the four medical image sequence data of T1, T2, Fliar, and T1ce, output the reconstructed four medical image sequence data of T1, T2, Fliar, and T1ce, perform complete end-to-end training, and establish an initial latent space mapping relationship based on the variational autoencoder model.

5. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, wherein In the said Step 4, the specific method for separately extracting the decoder part of the variational autoencoder obtained in Step 3 is as follows: 5.

1. Construct a separate decoder model, screen the weights of the variational autoencoder model extracted in Step 4, retain the decoder part and import it into the separate decoder.

6. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, wherein In the said Step 5, the latent space alignment method is specifically as follows: 6.

1. Construct a new encoder model with the same structure as that in the variational autoencoder; 6.

2. Input the four medical image sequence data of T1, T2, Fliar, and T1ce with missing values into the new encoder model, set the missing medical image sequences to 0. For example, if the T1 and T2 modalities are missing, the input sequence is {0, 0, Fliar, T1ce}; 6.

3. Use the MSE loss to align the encoder and decoder in the latent space dimension. The table format of MSE is: where N is the total number of pixels in the image, and x i and are the values of the original image and the reconstructed image at the i-th pixel position, respectively.

7. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, wherein In the said Step 6, the fine-tuning training using the SSIM and perceptual loss with adjusted weights is specifically as follows; 7.

1. Use the SSIM loss adjusted according to the weights of the multi-MRI sequence data for training. Its formula is: where x is the original image, is the reconstructed image, S represents the total number of independent scan sequences included in the multi-sequence image, is the structural similarity measure between the original image and the reconstructed image of the s-th scan sequence. By independently calculating the structural similarity of each sequence and then performing weighted fusion, it effectively balances the contribution degree of cross-sequence features; 7.

2. Training is performed using a perceptual loss adjusted according to the weights of multi-MRI sequence data, and its formula is: Among them, φ j (x) is the feature map of the j-th layer extracted by the pre-trained VGG16 network, and M is the number of selected feature layers; this function introduces the principle of independent calculation for multiple MRI sequences, calculates the perceptual loss for multiple MRI sequences separately and then performs weighted fusion, avoiding cross-modal feature confusion.

8. The method for multi-MRI sequence missing imputation based on variational autoencoder according to claim 1, characterized in that In the seventh step, the data with randomly missing modalities is input into the model to obtain the result of random imputation.

Images