A Deep Learning-Based Method for Synthesizing Multimodal Super-Resolution Images from Cross-Dataset Magnetic Resonance Imaging

Abstract

A deep learning-based method for synthesizing multimodal super-resolution magnetic resonance (MRI) images across datasets is presented, relating to the field of MRI. Utilizing unsupervised methods in deep learning and employing a generative adversarial network (GAN) architecture, multimodal super-resolution MRI images are synthesized. The images undergo data preprocessing, including registration, slicing, and normalization. A high-frequency template is then constructed, and the data is subjected to an FFT transform to obtain its K-space representation. The K-space image is multiplied by the high-frequency template to obtain its K-space high-frequency information, followed by an IFFT transform to obtain the high-frequency information in the image domain. The prepared images and their high-frequency information are then fed into the network. Through network training, multimodal data synthesis is achieved. An image degradation algorithm for the super-resolution reconstruction network is constructed. The super-resolution image reconstruction network architecture is constructed, and the degraded image and the original ground truth image are input into the network for training to obtain high-resolution multimodal MRI images.

Description

Technical Field

[0001] This invention relates to the field of magnetic resonance imaging, and in particular to the synthesis of multimodal super-resolution in magnetic resonance imaging based on deep learning. Background Technology

[0002] In the field of medical imaging, magnetic resonance imaging (MRI), as a non-invasive and non-radioactive diagnostic tool, is widely used in the diagnosis, treatment, and monitoring of various diseases. However, different MRI modalities provide complementary information, such as T1-weighted images, T2-weighted images, PD, T2flair, T2star, and DWI. These image modalities have unique advantages in capturing different tissue characteristics and different manifestations of lesions. For example, T1-weighted images have high contrast for structures such as medulla and fat, making them suitable for displaying the morphology and positional relationships of anatomical structures; while T2-weighted images provide better visualization of lesions such as fluid content and edema, helping to provide more information about the lesion.

[0003] However, acquiring multimodal MRI images typically requires separate scanning sequences and imaging parameter settings, which not only increases costs and time but may also introduce inconsistencies and image registration problems. Therefore, for the field of medical imaging, synthesized multimodal MRI images have different values ​​and significance. These multimodal MRI data can provide more comprehensive and accurate information, which is crucial for diagnosis, treatment, and disease monitoring.

[0004] Traditional image synthesis methods often rely on professional medical radiologists manually designing features and rules, which is not only inefficient but also subjective and limited. Many previous deep learning-based methods depend on a single dataset and require samples with paired modalities. Summary of the Invention

[0005] The main objective of this invention is to overcome the aforementioned deficiencies in the prior art and provide a deep learning-based synthesis of multimodal super-resolution magnetic resonance images. This invention designs an unsupervised adversarial network based on deep learning and achieves efficient synthesis of multimodal super-resolution magnetic resonance images across datasets trained on a few samples through two stages.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] A deep learning-based method for synthesizing multimodal super-resolution magnetic resonance images across datasets includes the following steps:

[0008] 1) Acquire raw medical data and analyze and preprocess it;

[0009] 2) Construct the high-frequency template HF_mask;

[0010] 3) High-frequency information data_HF is extracted using the high-frequency template HF_mask;

[0011] 4) Stack the data data and the high-frequency information data_HF into channels, perform random pruning on the stacked data, and then perform channel separation operation after pruning;

[0012] 5) Construct and train a multimodal synthesis network;

[0013] 6) Construct a multimodal super-resolution reconstruction network model;

[0014] 7) Input multi-database, multi-modal magnetic resonance images into a network built on SRGAN for training;

[0015] 8) Apply the trained weights to the multimodal data generated in stage one to obtain a high-resolution multimodal MR image with a resolution multiple of α.

[0016] 9) Apply multimodal super-resolution magnetic resonance imaging to a variety of downstream tasks.

[0017] In step 1), the specific steps for acquiring, analyzing, and preprocessing the raw medical data can be as follows:

[0018] 1.1) Obtain a public multimodal dataset;

[0019] 1.2) The images were observed and analyzed using the 3D medical image visualization software Slicer and ITK-SNAP to obtain the correspondence between the X, Y, and Z axes and the sagittal section, coronal section, and transverse section.

[0020] 1.3) Correct the spatial location of the multimodal data in nii.gz format and register the multimodal data:

[0021] 1.4) Perform cross-sectional slicing on the data in the dataset to obtain several layers of two-dimensional slices in .mat format;

[0022] 1.5) For slice preprocessing, the image is normalized to the range of intensity values ​​from 0 to 1 by using maximum value normalization;

[0023] 1.6) Standardize the size of the slices so that all slices are M x M, which facilitates better matching of the network's input applicability;

[0024] 1.7) Detect outliers for all slices, set INF values and NAN values in the two-dimensional slice data to 0, and eliminate damaged and abnormal data.

[0025] Preferably, steps 1.4) to 1.7) specifically include:

[0026] Perform Z-axis slicing on the data in the IXI and Simons datasets. For each sample, obtain approximately 50 two-dimensional slices in the.mat format with a size of M×M, located in the middle part of the brain rich in texture structure information. Normalize the image gray values to the range [0,1] using the maximum normalization method. For the HCP dataset, perform slicing on the data in a specified direction when the b value is 1000, obtain several slices with a size of W×H, and perform maximum normalization processing. Detect outliers and clean the slices of all datasets, set INF values and NAN values to 0, and eliminate damaged and abnormal data. Before feeding into the network, perform the resize operation in transform on the HCP data to adjust the size of all slices to M×M to meet the input requirements of the network.

[0027] In step 2), the specific steps for constructing the high-frequency template HF_mask can be:

[0028] 2.1) Set the scale factor and the center coordinates, and calculate the distance between any point in space and the center coordinates;

[0029] 2.2) Divide the matrix into three parts D1, D2, and D3 based on the distance;

[0030] 2.3) Assign values to the regions, set the value of D1 to 1, assign values to the D2 region through the Gaussian attenuation function. According to the distance from r1 to r2, set the value of D2 to decay from 1 to 0, and set the value of D3 to 0 to obtain the low-frequency template LF_mask, and calculate the high-frequency template HF_mask = 1 - LF_mask.

[0031] Preferably, step 2) specifically includes: Set the scale factors scale_factor1 and scale_factor2, set the parameter r1 = M / 2×scale_factor1, r2 = M / 2×scale_factor2 and r1 < r2, set the center coordinates as (centre_x, centre_y), and the coordinates of any point in space as (x, y). Set the distance between any point in space and the center coordinates as Divide the M x M two-dimensional matrix into three parts, namely D1 = Distance < r1, D2 = r1 < Distance < r2, and D3 = r2 < Distance < M / 2, and assign values to the regions: set the value of D1 to 1, assign values to D2 through a Gaussian decay function, set the value to decay from 1 to 0 according to the distance from r1 to r2, and set D3 to 0. Thus, the low-frequency template LF_mask is obtained.

[0032] The high-frequency template HF_mask = 1 - LF_mask;

[0033] In step 3), the specific steps for extracting the high-frequency information data_HF of the data can be as follows: Perform FFT transformation (i.e., the efficient and fast algorithm of DFT) on the data to obtain its K space, perform fast Fourier transform shift on the K space according to the fast Fourier transform shift function, and move the zero-frequency component of the fast Fourier transform (FFT) result to the center of the spectrum. Multiply the transformed K space with HF_mask to obtain the high-frequency information of the K space, and then perform fast Fourier transform shift on the high-frequency information of the K space and execute IFFT (i.e., the efficient and fast algorithm of IDFT) transformation to obtain the high-frequency information data_HF of the data.

[0034] The formula for DFT is:

[0035]

[0036] where N is the number of points of the time-domain discrete signal, n represents the number of the time-domain discrete signal, and k represents the number of the frequency-domain signal;

[0037] The formula for IDFT is:

[0038]

[0039] In step 5), the specific steps for constructing and training the multi-modal synthesis network can be as follows:

[0040] 5.1) Construct a deep neural network with two channels as network inputs (data and data_HF respectively) and one channel as the output for multi-modal synthesis, that is, the multi-modal synthesis network;

[0041] 5.2) Build a generator network based on SRGAN; for the encoder part in the generator, use two different network designs to extract the high-frequency image features and the features of the original image respectively; for the decoder part of the generator, use transposed convolution, and at the same time fuse some features of data_HF into the decoder part;

[0042] 5.3) Construct a discriminator to identify the rationality of the generated image;

[0043] 5.4) Construct a reasonable network loss function, including the cycle consistency loss function, the perceptual loss function, and the generative adversarial loss function.

[0044] 5.5) Train the multimodal synthesis network;

[0045] Preferably, the generator-decoder part of the multimodal synthesis network specifically includes: the encoder ultimately transforms the image with B×1×M / 2×M / 2 features extracted by feature downsampling into B×C1×H1×W1. The upsampling block includes a 2d transposed convolution with a kernel of 3, a BatchNorm2d, and an activation layer using the ReLU activation function. The feature is passed through an upsampling block once, resulting in B×C1 / 2×2H1×2W1. Then, B×C2 / 2×2H1×2W1 extracted based on ResNet34 is fused into this feature block. The feature is then passed through another upsampling block, resulting in B×(C1+C2 / 2) / 2×4H1×4W1. The high-frequency feature B×C2 / 2×4H1 x 4W1 is then fused into a single feature block, resulting in B×((C1+C2 / 2) / 2+C2 / 22)×4H1×4W1. This feature is then passed through another upsampling block, resulting in B×((C1+C2 / 2) / 2+C2 / 2) / 2×8H1×8W1. Finally, this feature is fused with the high-frequency feature B xC2 / 4×8H1×8W1 to achieve feature association. The reconstructed image is obtained after two upsampling operations.

[0046] In step 6), the specific steps for constructing the multimodal super-resolution reconstruction network model can be as follows:

[0047] 6.1) Construct a super-resolution network degradation model, including applying low-frequency templates for magnetic resonance image blurring, random image downsampling, and random image noise addition;

[0048] 6.2) Construct a training model with six input and output channels, including multiple magnetic resonance modes.

[0049] 6.3) Construct generator and discriminator networks based on SRGAN.

[0050] 6.4) Construct a multi-channel loss function, including a pixel loss function, a perceptual loss function, and a generative adversarial loss function.

[0051] Preferably, in step 6.1), the super-resolution network degradation model is constructed, which specifically includes three parts:

[0052] First, medical image sampling is employed, i.e., the image domain is converted to the K-space frequency domain. A low-frequency template, data_LF, is constructed according to step S23. This low-frequency template is applied to the K-space region, and the image is then converted back to the image domain to obtain a blurred magnetic resonance image. Second, three different sampling modes, mode1, mode2, and mode3, are used for random downsampling. Third, two noise reduction methods, Noise1 and Noise2, are used to randomly add noise to the image.

[0053] This invention also provides a deep learning-based synthesis system for multimodal super-resolution magnetic resonance images across datasets, comprising, in sequence: a data acquisition and analysis unit, a data preprocessing unit, a high-frequency template construction unit, a high-frequency information extraction unit, a multimodal synthesis network structure construction unit, a multimodal synthesis network loss function unit, a super-resolution degradation model construction unit, a network construction unit for the generator and discriminator of super-resolution reconstruction, a super-resolution reconstruction loss function unit, and a data training and application unit.

[0054] The data acquisition and analysis unit is used to collect and preprocess raw medical image data, including acquiring multimodal magnetic resonance images from multimodal public datasets and analyzing them using visualization software to determine the correspondence between the X, Y, and Z axes and the sagittal, coronal, and transverse planes; the multimodal public datasets include the IXI dataset, HCP dataset, Simons dataset, etc.; the visualization software can be Slicer or ITK-SNAP.

[0055] The data preprocessing unit is used to clean, normalize, and slice multimodal data to ensure the quality of input data and adapt to the input requirements of the network.

[0056] The high-frequency template construction unit calculates the high-frequency template HF_mask by setting a two-dimensional matrix partition and using a Gaussian decay function according to the set scaling factor and parameters, which is used to extract high-frequency information in the image.

[0057] The high-frequency information extraction unit obtains the high-frequency information data_HF from the data using a high-frequency template; it includes an FFT transformation module, a K-space shift module, a high-frequency information extraction module, an inverse K-space shift module, an IFFT transformation module, and a data output module; the FFT transformation module is used to transform the input time-domain data to the frequency domain using a Fast Fourier Transform (FFT) to obtain its K-space representation; the K-space shift module is used to perform a shift operation on the K-space obtained by the FFT transformation, moving the zero-frequency component to the center of the spectrum; the high-frequency information extraction module is used to utilize a pre-constructed high-frequency template HF_mas k is multiplied element-wise with the shifted K-space to extract the high-frequency information portion of the K-space; the inverse K-space shift module is used to perform an inverse shift operation on the K-space after the high-frequency information is extracted, restoring its original relative position for subsequent IFFT transformation; the IFFT transformation module is used to transform the processed K-space high-frequency information back to the time domain through inverse fast Fourier transform (IFFT) to obtain the final high-frequency information data data_HF; the data output module is used to format the high-frequency information data data_HF obtained by IFFT transformation and output it to the subsequent processing flow;

[0058] The multimodal synthesis network structure building unit is used to construct a complex neural network structure that can receive different types of input data and fuse information from different modalities through multi-step feature extraction to generate unified multimodal output data.

[0059] The multimodal synthesis network loss function unit is used to construct reasonable network loss functions, including cycle consistency loss, generative adversarial loss function, and perceptual loss function;

[0060] The super-resolution model building unit is used to construct a model that can convert the original magnetic resonance image into a low-resolution image (learning the mapping from low resolution to the original image); the super-resolution model building unit includes a K-space processing module, a downsampling module, and a noise addition module; the K-space processing module is used to convert the medical image from the image domain to the K-space frequency domain, and apply a low-frequency template in the K-space frequency domain to convert the image back to the image domain, obtaining a blurred magnetic resonance image; the noise addition module is used to randomly add noise to the image, finally obtaining a low-resolution image.

[0061] The network building unit of the super-resolution reconstruction generator and discriminator is used to create a generator to generate high-resolution images and a discriminator to evaluate the realism of the images, thereby optimizing the quality of the super-resolution images.

[0062] The super-resolution reconstruction loss function unit is used to create multiple loss functions to guide the model in learning how to improve image resolution; including pixel loss function, generative adversarial loss function, and perceptual loss function.

[0063] The data training and application unit is used to input the synthesized multimodal medical image data into the SRGAN-based network for training to improve the resolution of magnetic resonance images, and to apply the model to actual images to obtain high-resolution multimodal magnetic resonance image results.

[0064] Furthermore, the multimodal synthesis network structure building unit is used to design and construct a deep neural network structure for multimodal image synthesis, in order to generate high-quality multimodal super-resolution images by fusing information from different modalities. Specifically, this unit constructs a deep neural network with dual-channel input (data and data_HF) and single-channel output. This deep neural network is used for the synthesis of multimodal magnetic resonance images. The network adopts the framework of Generative Adversarial Network (GAN), combining a generator and a discriminator structure. The generator consists of an encoder, a decoder, and a feature fusion mechanism. The discriminator adopts PatchGAN. Specifically, the encoder uses the algorithm model Algorithm1 to extract data. The texture detail features of _HF are extracted using the Algorithm2 model to extract global features from data. The decoder uses an upsampling block containing transposed convolutions to expand and reshape the resolution of the feature map. The upsampling block includes a 2D transposed convolution with a kernel of 3, a BatchNorm2d layer, and an activation layer using the ReLU activation function. During the decoding process, the texture detail features extracted from data_HF and the global features extracted from data are fused using the Algorithm3 network structure. The final generated image is then fed into the PatchGAN discriminator for evaluation to determine the authenticity and reasonableness of the generated image.

[0065] Furthermore, the network building units for the generator and discriminator are constructed using SRGAN; the generator network uses RRDB blocks as the backbone structure of the network model, and the discriminator network model adopts a spectral normalized U-Net discriminator.

[0066] This invention proposes a deep learning-based unsupervised method that can utilize large amounts of medical imaging data, spanning datasets, and without requiring paired modal samples. Through a designed network, it automatically learns feature representations and complex nonlinear relationships from the data, exhibiting better generalization ability. By training a deep learning model, it can synthesize multimodal images from single-modal MRI images, such as T1-weighted, T2-weighted, PD-weighted, and T2*-weighted images, providing more comprehensive and accurate information support for medical imaging diagnosis. Furthermore, this invention focuses on the synthesis and construction of super-resolution images from multimodal MRI images. Synthesizing super-resolution medical images can effectively improve the spatial resolution of images, enabling medical radiologists to observe tissue structures and lesion features more clearly, bringing greater convenience and benefits to clinical practice. This deep learning-based multimodal super-resolution image synthesis method not only improves the accuracy and efficiency of diagnosis but also reduces the workload of medical imaging radiologists, bringing greater convenience and benefits to clinical practice. This invention combines K-space domain and image domain information with Swim... The combination of Transformer and CNN network structures enables the synthesis of multimodal magnetic resonance data across datasets without relying on paired modal samples; this invention can synthesize ultra-high resolution magnetic resonance images; this invention achieves an effective combination of multimodal synthesis and super-resolution reconstruction, and can synthesize high-resolution multimodal magnetic resonance data, which to some extent solves the problems of modality loss, modality damage, and unclear and low image resolution. Attached Figure Description

[0067] Figure 1 This is a flowchart of a magnetic resonance multimodal super-resolution image synthesis method according to an example of the present invention.

[0068] Figure 2 This is a network structure diagram of the magnetic resonance multimodal image synthesis method according to an example of the present invention.

[0069] Figure 3 This is a network structure diagram of the super-resolution network reconstruction of magnetic resonance multimodal data in an example of the present invention.

[0070] Figure 4 The sample in this invention is synthesized using the magnetic resonance multimodal synthesis method, which produces six corresponding modes. Among them, (a) is its T1-weighted image, (b) is its T2-weighted image, (c) is its PD-weighted image, (d) is its T2*-weighted image, (e) is its T2-flair-weighted image, and (f) is its DWI-weighted image.

[0071] Figure 5 This is a comparison image of an original magnetic resonance imaging (MRI) image before and after degradation using a super-resolution network algorithm, as an example of this invention. GT represents the original image, and LR represents the low-resolution image after degradation.

[0072] Figure 6 This is an example of an original T1-weighted image and its super-resolution image reconstructed using magnetic resonance multimodal super-resolution reconstruction. GT is the original weighted image with a resolution of 256×256, and SR is the image with a resolution of 1024×1024 after super-resolution rate reconstruction. Detailed Implementation

[0073] To more clearly describe the objectives, technical solutions, and advantages of this invention, the invention will be described in detail below with reference to the accompanying drawings. It should be noted that the described embodiments represent only some implementations of this invention, and not all of them. Within the framework of this invention, those skilled in the art can develop other implementations based on the provided embodiments without additional creative work. All such implementations should fall within the protection scope of this invention.

[0074] The present invention will be further described below through specific embodiments:

[0075] Reference Figure 1 A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning, characterized by the following steps:

[0076] S1 First, acquire raw medical data and analyze and preprocess it.

[0077] S11 Obtain multimodal public datasets; in this embodiment, the datasets include the IXI dataset (containing T1-w, T2-w, and PD modes), the HCP dataset (containing T1-w, T2-w, and DWI modes), and the Simons dataset (containing T1-w, T2-w, T2*, and T2-flair modes), etc.

[0078] S12 uses the 3D medical image visualization software Slicer and ITK-SNAP to observe and analyze images, and obtain the correspondence between the X, Y, and Z axes and the sagittal section, coronal section, and transverse section.

[0079] S13 performs spatial location correction on multimodal data in nii.gz format and uses the flirtlinear registration function of FSL software to register the multimodal data.

[0080] S14 performs cross-sectional slicing on the data in the dataset to obtain several layers of two-dimensional slices in .mat format;

[0081] S15 adopts the method of maximum normalization: (x - x_min) / (x_max - x_min), where x is the pixel intensity value; the image is normalized to the range of intensity values from 0 to 1;

[0082] S16 simultaneously performs unified size processing on the slices. Through the operation of padding 0 around, the size of all slices is made 256×256 to better match the input applicability of the network;

[0083] S17 performs outlier detection on all slices, sets the INF values and NAN values in the two-dimensional slice data to 0, and eliminates damaged and abnormal data.

[0084] S2 constructs the required high-frequency template HF_mask;

[0085] S21 sets the scale factor scale_factor1 to 0.10, scale_factor2 to 0.15, sets the parameter r1 = M / 2×scale_factor1 to 12.80, r2 = M / 2×scale_factor2 to 19.20, and r1 < r2. In this case, the distance of the transition band is 19.2 - 12.80 = 6.40. Set the center coordinates as (centre_x, centre_y), and the coordinates of any point in space as (x, y); set the distance between any point in space and the center coordinates as

[0086] S22 divides the M x M two-dimensional matrix into three parts, namely D1 = Distance < 12.80, D2 = 12.80 < Distance < r2, and D3 = 19.20 < Distance < 128;

[0087] S23 assigns values to the regions: sets the value of D1 to 1, and assigns values to D2 through the Gaussian decay function; the formula of the Gaussian decay function is:

[0088]

[0089] According to the distance from r1 to r2, set the value of D2 to decay from 1 to 0; set D' to < 0; thus obtain the low-frequency template LF_mask, and the high-frequency template HF_mask = 1 - LF_mask. Partitioning the image into three parts and performing Gaussian function decay is beneficial to avoid the sinc filtering effect; obtain the high-frequency information data_HF of the data through the template;

[0090] S3 obtains the high-frequency information data_HF through a high-frequency template; performs an FFT (i.e., an efficient and fast algorithm of DFT) on the data to obtain its K-space, and uses a window function to reduce the influence of boundary effects; performs a Fast Fourier Transform (FFT) shift on the K-space according to the Fast Fourier Transform (FFT) shift function, and moves the zero-frequency component of the FFT result to the center of the spectrum; multiplies the transformed K-space with HF_mask to obtain the high-frequency information of the K-space, then performs a Fast Fourier Transform (FFT) shift on the high-frequency information of the K-space and performs an IFFT (i.e., an efficient and fast algorithm of IDFT) to obtain data_HF;

[0091] The formula for DFT is:

[0092]

[0093] The formula for IDFT is:

[0094]

[0095] S4 performs channel stacking on data and data_HF. Channel stacking stacks the original image data data and the high-frequency information image data_HF in the channel dimension. If the original image is color and has three RGB channels, the high-frequency information image may need to be copied or converted to have the same number of channels as the original image in order to be able to stack. The stacked image has more channels, which contain information from the original image as well as enhanced high-frequency information.

[0096] Random cropping is performed on the stacked data to a size of 128×128; multiple different cropped versions are generated from the original images, thereby increasing the diversity of the training dataset.

[0097] After cropping, a channel separation operation is performed to process or analyze the original image and the high-frequency information image separately; a multimodal synthesis network structure is then constructed.

[0098] S5 reference Figure 2 Construct a multimodal synthesis network;

[0099] S51 constructs a deep neural network with dual channels as network input (data and data_HF respectively) and a single channel as output for multimodal synthesis.

[0100] The encoder part of the S52 generator uses two different network designs to extract high-frequency image features and original image features respectively. Specifically, ResNet34 with ResNet block as the backbone is used to extract texture detail features of data_HF, and Swim-T model with Swim Transformer block as the feature extractor to extract global features of data.

[0101] The decoder section of the S53 generator uses transposed convolution for resolution augmentation and reshaping. Based on the U-net network structure, it fuses the features extracted from data_HF into the decoder. Specifically, the encoder downsamples the features, transforming the 4×1×128×128 image into a 4×1280×4×4 image. The upsampling block includes a 2D transposed convolution with a kernel of 3, a BatchNorm2d, and an activation layer using the ReLU activation function. This feature is then processed through the upsampling block to obtain... The first feature block is 4×640×8×8. Then, a 4×256×8×8 feature extracted based on ResNet34 is fused into this feature block. This feature is then upsampled to 4×448×16×16. The high-frequency feature of 4×128×16×16 is then fused to obtain 4×576×16×16, which is then upsampled again to obtain 4×288×32×32. This is then fused with the high-frequency feature of 4×64×32×32 to achieve feature association. Finally, the reconstructed image is obtained after two upsampling operations.

[0102] The S54 constructs a discriminator to determine the legitimacy of the generated image;

[0103] S55 constructs a reasonable loss function for the multimodal synthesis network, including a cycle consistency loss function using L1 loss, a perceptual loss function using a VGG19 pre-trained network, and a generative adversarial loss function using BCEwithlogits loss. In the loss function, x represents the image in the source domain, y represents the image in the target domain, G and F are two generators, generator G maps the image in the x domain to the image in the y domain, and generator F maps the image mapped to the y domain back to the x domain, and D is the discriminator. The perceptual loss uses the pre-trained VGG19 network to extract feature layers and calculates the difference between the generated image and the target image in the feature space; φ l For the feature extractor in the l-th layer of the pre-trained network, λ l The weights for the feature loss of layer l are specifically chosen from the feature extractors of layers 3, 8, and 17 of VGG19; depending on the different synthesis tasks; λ cyc Set to 5.0~15.0, λ adv Set to 1.0~3.0 and λper Set to 0.3 to 0.7, with different supernatural parameter combinations depending on the task requirements;

[0104] The formula for the cycle consistency loss function is:

[0105]

[0106] The formula for the generator's adversarial loss function is:

[0107]

[0108] The adversarial loss function formula for the discriminator is:

[0109]

[0110] The formula for the total loss function of generating adversarial loss is:

[0111] L adv (G,D,x,y)=L advG (G,D,x)+L advD (D,G,x,y)

[0112] The formula for the perceptual loss function is:

[0113]

[0114] The total loss function formula for multimodal synthesis is:

[0115] L syn (G,D,F,x,y)=λ cyc L cycle (G,F,x,y)+λ adv L adv (G,D,x,y)+λ per L perceptual (G,x,y)

[0116] When training the multimodal synthesis network using S56, specifically: when generating T2* weighted sum and T2-flair modal data, 2000 sets of samples are used for training; when generating DWI modal data, 3000 sets of samples are used for training; the training batch size is set to 4, the initial learning rate is set to 0.0002, and the total training time is 100 epochs; at 50 epochs, the learning rate is reduced using Gaussian descent until the learning rate is set to 0 after 100 epochs.

[0117] S6 Figure 3 As shown, a multimodal super-resolution reconstruction network model is constructed;

[0118] S61 constructs a super-resolution degradation model, comprising three parts: blurring, downsampling, and noise addition. The blurring part employs a blurring method consistent with medical image sampling. Based on step S23, a low-frequency template `data_LF` is constructed and applied to the K-space region of the corresponding medical image. The image is then converted to the image domain to obtain a blurred low-resolution magnetic resonance image. The downsampling part uses three different sampling modes—mode1, mode2, and mode3—for random downsampling. In this embodiment, the mode can be randomly selected from area, bicubic, and bilinear, with a sampling factor of [missing value]. α is set to 4; the noise addition part uses two noise modes, Noise1 and Noise2, to randomly add noise to the image. In this embodiment, random Gaussian noise and Raleigh noise are used, and a random number z between 0 and 1 is used as the noise judgment standard. When z ≤ 0.3, Gaussian noise is added to the image, when z ≥ 0.7, Raleigh noise is added to the image, and when z takes other values, no additional noise processing is applied to the image. In addition, the noise added to the six channels is inconsistent and random. In the experimental part, it was found that if sinc filtering is added to the noise part, it will cause undesirable artifact effects after super-resolution. Therefore, sinc filtering is not added as part of the degradation model. Figure 5 This is a comparison result of a continuous layer image of a certain modality after being processed by a degradation algorithm. GT is the original image, and LR is the low-resolution image after the degradation model.

[0119] The S62 constructs a training model with six input channels and six output channels, including T1-w, T2-w, PD, T2flair, T2star, and DWI.

[0120] S63 builds generator and discriminator networks based on SRGAN;

[0121] S64 constructs a multi-channel, reasonable loss function for super-resolution reconstruction, including a pixel loss function applying L1 loss, a perceptual loss function applying a VGG19 pre-trained network, and a generative adversarial loss function applying BCEwithlogits loss. In the loss function, GT is the real image, x is the low-resolution, low-definition image processed by the degradation algorithm, G is the generator, and G reconstructs x into the corresponding real image; φ l_sr For the feature extractor in the l-th layer of the pre-trained network model, specifically the first 37 layers of VGG19 are used as the feature extractor, λ l_sr The loss weight for the l-th layer features is set to 1.0 by default. The formula for the pixel loss function is:

[0122]

[0123] The formula for the generator's adversarial loss function is:

[0124]

[0125] The adversarial loss function formula for the discriminator is:

[0126]

[0127] The formula for the total loss function of generating adversarial loss is:

[0128] L adv_sr (G,D,x,GT)=L advG_sr (G,D,x)+L advD_sr (D,G,x,GT)

[0129] The formula for the perceptual loss function is:

[0130]

[0131] The formula for the total loss function of super-resolution reconstruction is:

[0132] L sr (G,D,x,GT)=λ pixel L pixel (G,GT,x)+λ adv_sr L adv_sr (G,D,x,GT)+λ per_sr L perceptual (G,x,GT)

[0133] S7 Figure 4 The multimodal data shown can be used to synthesize magnetic resonance imaging (MRI) data for the corresponding six modes of a sample with missing data for one or more modes by using a synthesis network. After synthesizing these data, the MRI images of multiple databases and multiple modes are fed into a network built on SRGAN for training. Specifically, the training batch is set to 4, the initial learning rate is set to 0.0001, and the training is carried out for a total of 200 epochs. The learning rate is decayed by the gamma coefficient at 100 epochs, and the gamma is set to 0.2.

[0134] S8 applies the trained weights to the multimodal data generated in Stage 1, resulting in high-resolution multimodal magnetic resonance images with a multiplier of α of 4; for example... Figure 6 As shown, GT is the original weighted image with a resolution of 256×256, and SR is the image with a resolution of 1024×1024 after super-resolution variable reconstruction. It can be clearly seen that the high-resolution generated image has more obvious details and higher resolution.

[0135] S9 applies multimodal super-resolution medical magnetic resonance imaging to a variety of downstream tasks.

[0136] This invention utilizes unsupervised methods in deep learning, employing a generative adversarial network (GAN) architecture, to synthesize multimodal super-resolution images for magnetic resonance imaging (MRI). Different datasets often possess inconsistent modal information, potentially lacking certain modalities and exhibiting insufficient resolution. This method designs a process that constructs a multimodal synthesis network and an image super-resolution reconstruction network, achieving MRI multimodal super-resolution image synthesis through two stages. Stage 1: First, the image undergoes data preprocessing, including registration, slicing, and normalization. Then, a high-frequency template is constructed, and the data is subjected to FFT transformation to obtain its K-space. In the first stage, the K-space image is multiplied with a high-frequency template to obtain its K-space high-frequency information, and then an IFFT transformation is performed to obtain the high-frequency information in the image domain. Next, the prepared image and its high-frequency information are fed into the network as input. Through network training, multimodal data synthesis is achieved. In the second stage, an image degradation algorithm for the super-resolution reconstruction network is first constructed. Then, the network architecture for super-resolution image reconstruction is constructed. Next, the degraded image and the original real image are input into the network for training to obtain a high-resolution magnetic resonance image. Finally, high-resolution multimodal magnetic resonance images are obtained through stages one and two.

[0137] Phase 1 comprises the following units:

[0138] Data Acquisition and Analysis Unit: Specifically, this unit first acquires raw medical data and then analyzes and preprocesses it. The specific steps are as follows: Acquire multimodal public datasets, including the IXI dataset (containing T1-w, T2-w, and PD modes), the HCP dataset (containing T1-w, T2-w, and DWI modes), and the Simons dataset (containing T1-w, T2-w, T2*, and T2-flair modes); Use medical image visualization software for observation and analysis: Use Slicer and ITK-SNAP to observe and analyze images to determine the correspondence between the X, Y, and Z axes and the sagittal, coronal, and transverse planes.

[0139] Data preprocessing unit: This unit corrects the spatial location of multimodal data using FSL software for spatial registration to ensure data consistency in spatial location; it slices the data: Z-axis slicing is performed on the data from the IXI and Simons datasets, obtaining approximately 50 M×M 2D slices in .mat format for each sample, located in the brain region rich in texture and structural information; maximum value normalization is used to normalize the image grayscale values ​​to the range [0,1]; for the HCP dataset, slicing is performed in a specified direction with a b value of 1000, resulting in several W×H slices, which are then normalized to their maximum value; outlier detection and cleaning are performed on all dataset slices, setting INF and NAN values ​​to 0 and removing corrupt and outlier data; before inputting into the network, the HCP data undergoes a resize operation in the transform function to adjust the size of all slices to M×M to meet the network's input requirements.

[0140] High-frequency template construction unit: Setting scaling factors and parameters: Setting scaling factors scale_factor1 and scale_factor2, and parameters r1 and r2, and calculating the transition band distance based on these parameters; Setting two-dimensional matrix partitioning: Dividing the M x M two-dimensional matrix into three parts D1, D2, and D3; Attenuating the values ​​in region D2 from 1 to 0 according to the Gaussian attenuation function to obtain the low-frequency template LF_mask, and then calculating the high-frequency template HF_mask; The formula for the Gaussian attenuation function is:

[0141]

[0142] High-frequency information extraction unit: Obtains high-frequency information data_HF from the data through a template; the specific steps are as follows: Perform FFT transformation on the data (i.e., an efficient and fast algorithm of DFT) to obtain its K-space; perform Fast Fourier Transform shift on the K-space according to the Fast Fourier Transform shift function, and move the zero-frequency component of the Fast Fourier Transform (FFT) result to the center of the spectrum; multiply the K-space obtained after transformation with HF_mask to obtain the high-frequency information of the K-space; then perform Fast Fourier Transform shift on the high-frequency information of the K-space and perform IFFT transformation to obtain data_HF.

[0143] Multimodal synthesis network structure building units: Constructing a deep neural network with dual-channel input (data and data_HF) and single-channel output; The encoder uses Algorithm 1 to extract texture detail features from data_HF and Algorithm 2 to extract global features from data; The decoder uses an upsampling block containing transposed convolutions to expand and reshape the resolution, and simultaneously uses a network structure based on Algorithm 3 to fuse the two sets of features, specifically fusing the features extracted from data_HF into the decoder; The encoder ultimately... The image with B×1×M / 2×M / 2 features extracted by downsampling is transformed into B×C1×H1×W1. The upsampling block includes a 2d transposed convolution with a kernel of 3, a BatchNorm2d, and an activation layer using the ReLU activation function. This feature is then passed through the upsampling block to become B×C1 / 2×2H1×2W1. Then, B×C2 / 2×2H1×2W1 extracted based on ResNet34 is fused into this feature block. This feature is then passed through another upsampling block to become B×(C1+C2 / 2) / 2×4H1×4W1. Finally, the high-frequency features are transformed into B×C2 / 2×4H1. The features of x 4W1 are fused together to obtain B×((C1+C2 / 2) / 2+C2 / 22)×4H1×4W1, which is then entered into the upsampling block to obtain B×((C1+C2 / 2) / 2+C2 / 2) / 2×8H1×8W1. This is then fused with the high-frequency feature B xC2 / 4×8H1×8W1 to form a whole, realizing the correlation of features. Next, PatchGAN is used as a discriminator to identify the rationality of the generated image. A PatchGAN discriminator is constructed to identify the rationality of the generated image.

[0144] Multimodal synthetic network loss function unit: Constructs reasonable network loss functions, including cycle consistency loss, generative adversarial loss function, and perceptual loss function.

[0145] Phase Two comprises the following units:

[0146] Super-resolution model construction unit: This unit consists of three parts. First, the image domain is converted into the K-space frequency domain. According to step S23, a low-frequency template data_LF is constructed. The low-frequency template is applied in the K-space region, and then the image is converted into the image domain to obtain the blurred magnetic resonance image. Second, three different sampling methods, namely mode1, mode2 and mode3, are used to randomly downsample the image. Third, two noise methods, namely Noise1 and Noise2, are used to randomly add noise to the image.

[0147] Network building blocks for generators and discriminators: The generator network and discriminator network are constructed based on SRGAN; the generator network uses RRDB blocks as the backbone structure of the network model, and the discriminator network model adopts the spectral normalized U-Net discriminator.

[0148] Super-resolution reconstruction loss function unit: Constructs reasonable loss functions, including pixel loss function, generative adversarial loss function, and perceptual loss function.

[0149] Data Training and Application Unit: The synthesized multimodal medical image data is input into the network constructed based on SRGAN for training to obtain the network weights for multimodal super-resolution; the obtained single or modally missing magnetic resonance images are input into the synthetic network of multimodal super-resolution, and the trained weights are applied for inference to obtain a-fold resolution multimodal super-resolution magnetic resonance images.

[0150] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A method for synthesizing multimodal super-resolution images of magnetic resonance imaging across datasets based on deep learning, characterized in that... Includes the following steps: 1) Acquire raw multimodal magnetic resonance image data and analyze and preprocess the raw data; 2) Construct the high-frequency template HF_mask: Set the scaling factor and center coordinates, and calculate the distance between any point in space and the center coordinates; divide the two-dimensional matrix into three parts D1, D2, and D3 based on the distance; assign values ​​to the regions, set the value of D1 to 1, assign values ​​to the D2 region through the Gaussian decay function, set the value of D2 to decay from 1 to 0 according to the distance from r1 to r2, and set the value of D3 to 0 to obtain the low-frequency template LF_mask, and calculate the high-frequency template HF_mask = 1 - LF_mask; 3) Extract the high-frequency information data_HF from the high-frequency template HF_mask constructed in step 2); 4) Stack the data data and the high-frequency information data_HF into channels, perform random pruning on the stacked data, and then perform channel separation operation after pruning; 5) Construct and train a multimodal synthesis network; 5.1) Construct a deep neural network with two channels as network input, namely data and data_HF, and a single channel as output for multimodal synthesis. 5.2) Constructing a generator network based on SRGAN; The encoder part of the generator uses two different network designs to extract high-frequency image features and features of the original image, respectively. The decoder part of the generator uses transposed convolution, and also incorporates some features from data_HF into the decoder part; 5.3) Construct a discriminator to determine the validity of the generated image; 5.4) Construct a reasonable network loss function, including the cycle consistency loss function, the perceptual loss function, and the generative adversarial loss function; 5.5) Train a multimodal synthesis network to achieve the synthesis of multimodal data; 6) Construct a multimodal super-resolution reconstruction network model to build a model that can convert the original magnetic resonance image into a low-resolution image. It learns the mapping from the low-resolution image to the original image. 6.1) Construct a super-resolution network degradation model, including applying a low-frequency template to blur the magnetic resonance image, random downsampling of the image, and random noise addition to the image to obtain the degraded image; 6.2) Construct a six-channel input-output training model, including multiple magnetic resonance modes; 6.3) Construct generator and discriminator networks based on SRGAN; 6.4) Construct a multi-channel loss function, including a pixel loss function, a perceptual loss function, and a generative adversarial loss function; 7) Input the original multimodal magnetic resonance images and degraded images from multiple databases into a multimodal super-resolution reconstruction network based on SRGAN for training; 8) Apply the trained weights of the multimodal super-resolution reconstruction network to the multimodal data generated by the multimodal synthesis network, and use the trained weights for inference to obtain a high-resolution multimodal super-resolution magnetic resonance image with a resolution multiple of α. 9) Apply the multimodal super-resolution magnetic resonance images obtained in step 8) to a variety of downstream tasks.

2. The method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as described in claim 1, characterized in that... In step 1), the specific steps for acquiring the raw multimodal magnetic resonance image data and analyzing and preprocessing the raw data are as follows: 1.1) Obtain a public multimodal dataset; 1.2) Use 3D medical image visualization software Slicer and ITK-SNAP to observe and analyze the images to obtain the correspondence between the X, Y, and Z axes and the sagittal, coronal, and transverse planes; 1.3) Correct the spatial location of the multimodal data in nii.gz format and register the multimodal data: 1.4) Perform cross-sectional slicing on the data in the dataset to obtain several layers of two-dimensional slices in .mat format; 1.5) For slice preprocessing, the image is normalized to the range of intensity values ​​from 0 to 1 by using maximum value normalization; 1.6) Standardize the size of the slices so that all slices are M×M, which facilitates better matching of the network's input applicability; 1.7) Perform outlier detection on all slices, set the INF and NAN values ​​in the two-dimensional slice data to 0, and remove damaged and outlier data.

3. The method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as described in claim 2, characterized in that... Steps 1.4) to 1.7) specifically include: Z-axis slicing was performed on the data from the IXI and Simons datasets, resulting in 50 two-dimensional slices of size M×M in .mat format for each sample, located in the central part of the brain rich in texture and structural information. Maximum value normalization was used to normalize the image grayscale values ​​to the range [0,1]. For the HCP dataset, slicing was performed in a specified direction with a b value of 1000, resulting in several slices of size W×H, which were then normalized to maximum value. Outlier detection and cleaning were performed on all dataset slices, setting the INF and NAN values ​​to 0 and removing corrupted and outlier data. Before being fed into the network, the HCP data underwent a resize operation in the transform function to adjust the size of all slices to M×M to meet the network's input requirements.

4. The method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as described in claim 1, characterized in that... In step 2), the specific steps for constructing the high-frequency template HF_mask are as follows: Set the scaling factors scale_factor1 and scale_factor2, set the parameters r1 = M / 2 × scale_factor1, r2 = M / 2 × scale_factor2 and r1 < r2, set the center coordinates as (centre_x, centre_y), and the coordinates of any point in space as (x, y); set the distance between any point in space and the center coordinates as Divide the M×M two-dimensional matrix into three parts, namely D1 = Distance < r1, D2 = r1 < Distance < r2, and D3 = r2 < Distance < M / 2, and assign values to the regions: set the value of D1 to 1, assign values to D2 through a Gaussian decay function, according to the distance from r1 to r2, set the value to decay from 1 to 0, and set D3 to 0; thus obtain the low-frequency template LF_mask, and the high-frequency template HFmask = 1 - LF_mask.

5. The method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as described in claim 1, characterized in that... In step 3), the specific steps for extracting the high-frequency information data_HF are as follows: The data is subjected to an FFT transformation to obtain its K-space; the K-space is then shifted using a Fast Fourier Transform (FFT) shift function, moving the zero-frequency component of the FFT result to the center of the spectrum; the transformed K-space is multiplied by HF_mask to obtain the high-frequency information in the K-space; the high-frequency information in the K-space is then shifted using an FFT and subjected to an IFFT transformation to obtain the high-frequency information data_HF. The formula for DFT is: The formula for IDFT is:

6. The method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as described in claim 1, characterized in that... In step 6.1), a super-resolution network degradation model is constructed, which specifically includes three parts: First, medical image sampling is used, that is, the image domain is converted into the K-space frequency domain. According to the constructed low-frequency template data_LF, the low-frequency template is applied in the K-space region, and then the image is converted into the image domain to obtain the blurred magnetic resonance image; Second, three different sampling methods, namely mode1, mode2 and mode3, are used to randomly downsample; Third, two noise methods, namely Noise1 and Noise2, are used to randomly add noise to the image.

7. A deep learning-based system for synthesizing multimodal super-resolution images of magnetic resonance imaging across datasets, characterized in that... In order, they include: The system comprises: a data acquisition and analysis unit, a data preprocessing unit, a high-frequency template construction unit, a high-frequency information extraction unit, a multimodal synthesis network structure construction unit, a multimodal synthesis network loss function unit, a super-resolution degradation model construction unit, a network construction unit for the generator and discriminator of super-resolution reconstruction, a super-resolution reconstruction loss function unit, and a data training and application unit. The data acquisition and analysis unit is used to collect and preprocess raw medical image data, including acquiring multimodal magnetic resonance images from multimodal public datasets and analyzing them using visualization software to determine the correspondence between the X, Y, and Z axes and the sagittal, coronal, and transverse planes; the multimodal public datasets include the IXI dataset, the HCP dataset, and the Simons dataset; the visualization software used is Slicer and ITK-SNAP; The data preprocessing unit is used to clean, normalize, and slice multimodal data to ensure the quality of input data and adapt to the input requirements of the network. The high-frequency template construction unit calculates the high-frequency template HF_mask by setting a two-dimensional matrix partition and using a Gaussian decay function according to the set scaling factor and parameters, which is used to extract high-frequency information in the image. The high-frequency information extraction unit obtains the high-frequency information data_HF of the data through a high-frequency template; The system includes an FFT transform module, a K-space shift module, a high-frequency information extraction module, an inverse K-space shift module, an IFFT transform module, and a data output module. The FFT transform module converts the input time-domain data to the frequency domain using a Fast Fourier Transform (FFT) to obtain its K-space representation. The K-space shift module shifts the K-space obtained from the FFT transform, moving the zero-frequency component to the center of the spectrum. The high-frequency information extraction module extracts the high-frequency information by multiplying a pre-constructed high-frequency template (HF_mask) element-wise with the shifted K-space. The inverse K-space shift module performs a reverse shift operation on the K-space after high-frequency information extraction, restoring its original relative position for subsequent IFFT transforms. The IFFT transform module converts the processed K-space high-frequency information back to the time domain using an Inverse Fast Fourier Transform (IFFT) to obtain the final high-frequency information data (data_HF). The data output module formats the high-frequency information data (data_HF) obtained from the IFFT transform and outputs it to subsequent processing steps. The multimodal synthesis network structure building unit is used to construct a deep neural network structure with dual-channel input and single-channel output. This neural network structure can receive different types of input data data and data_HF, and fuse information from different modalities through multi-step feature extraction to generate unified multimodal output data. The multimodal synthesis network loss function unit is used to construct reasonable network loss functions, including cycle consistency loss, generative adversarial loss function, and perceptual loss function; The super-resolution degradation model construction unit is used to construct a model that can degrade the original magnetic resonance image into a low-resolution image. The super-resolution degradation model construction unit includes a K-space processing module, a downsampling module, and a noise addition module. The K-space processing module is used to convert the medical image from the image domain to the K-space frequency domain, and apply a low-frequency template in the K-space frequency domain to convert the image back to the image domain, obtaining a blurred magnetic resonance image. The noise addition module is used to randomly add noise to the image, ultimately obtaining a low-resolution image. The network building units for the generator and discriminator in the super-resolution reconstruction are constructed using SRGAN. The generator network uses RRDB blocks as the backbone structure of the network model, and the discriminator network model uses a spectral normalized U-Net discriminator. The super-resolution reconstruction loss function unit is used to create multiple loss functions to guide the model in learning how to improve image resolution; This includes pixel loss function, generative adversarial loss function, and perceptual loss function; The data training and application unit is used to input the synthesized multimodal medical image data into the network constructed based on SRGAN for training to obtain the network weights for multimodal super-resolution. The obtained single or modally missing magnetic resonance images are input into the multimodal super-resolution synthesis network, and the trained weights are applied to inference to obtain an α-fold resolution multimodal super-resolution magnetic resonance image.

8. The deep learning-based multimodal super-resolution image synthesis system for cross-dataset magnetic resonance imaging as described in claim 7, characterized in that... The multimodal synthesis network structure building unit constructs a deep neural network with dual-channel input (data and data_HF) and single-channel output. This deep neural network is used for the synthesis of multimodal magnetic resonance images. The network adopts the framework of Generative Adversarial Network (GAN), combining a generator and a discriminator. The generator consists of an encoder, a decoder, and a feature fusion mechanism. The discriminator uses PatchGAN. Specifically, the encoder uses Algorithm1 to extract texture detail features of data_HF and Algorithm2 to extract global features of data. The decoder uses an upsampling block containing transposed convolutions to expand and reshape the feature map resolution. The upsampling block includes a 2D transposed convolution with a kernel of 3, a BatchNorm2d layer, and an activation layer using the ReLU activation function. During the decoding process, the texture detail features extracted by data_HF and the global features extracted by data are fused using the Algorithm3 network structure. The final generated image is fed into the PatchGAN discriminator for evaluation to determine the authenticity and reasonableness of the generated image.

Images