Deep learning-based cross-dataset magnetic resonance multi-modal super-resolution image synthesis method

Abstract

The invention discloses a cross-dataset magnetic resonance multi-mode super-resolution image synthesis method based on deep learning, and relates to the field of magnetic resonance imaging. An unsupervised method in deep learning is utilized, and the architecture of a generative adversarial network is adopted, so that the synthesis of a magnetic resonance multi-mode super-resolution image is realized. Carrying out data preprocessing on the image, wherein the data preprocessing comprises registration, slicing and normalization; constructing a high-frequency template, performing FFT (Fast Fourier Transform) on the data to obtain a K space, multiplying a K space image by the high-frequency template to obtain high-frequency information of the K space, and performing IFFT to obtain high-frequency information of an image domain; the prepared image and the high-frequency information thereof are fed into the network as inputs. And through network training, synthesis of multi-modal data is realized. And constructing an image degradation algorithm of the super-resolution reconstruction network. And constructing a network architecture for super-resolution image reconstruction, and inputting the degraded image and the original real image into a network for training to obtain a high-resolution multi-modal magnetic resonance image.

Description

Technical Field

[0001] The present invention relates to the field of magnetic resonance imaging, and in particular to the synthesis of magnetic resonance multimodal super-resolution based on deep learning. Background Art

[0002] In the field of medical imaging, magnetic resonance imaging (MRI) is a non-invasive, non-radioactive diagnostic tool that is widely used in the diagnosis, treatment and monitoring of various diseases. However, MRI images under different modalities provide complementary information, such as T1-w, T2-w, PD, T2flair, T2star, DWI, etc. These image modalities have unique advantages in capturing different characteristics of tissues and different manifestations of lesions. For example, T1-weighted images have a higher contrast for structures such as medulla and fat, and are suitable for displaying the morphology and positional relationship of anatomical structures; while T2-weighted images have better display effects for lesions such as fluid content and edema, which helps to provide more information about lesions.

[0003] However, acquiring multimodal MRI images usually requires different scanning sequences and imaging parameter settings, which not only increases cost and time, but may also introduce problems such as inconsistency and image registration. Therefore, for the field of medical imaging, synthetic multimodal magnetic resonance images have different values ​​and significances. Through these multimodal magnetic resonance data, more comprehensive and accurate information can be obtained, which is crucial for diagnosis, treatment and disease monitoring.

[0004] Traditional image synthesis methods usually rely on professional medical radiologists to manually design features and rules, which is not only inefficient, but also subjective and limited. Many previous deep learning-based methods rely on a single dataset and require samples of paired modalities. Summary of the invention

[0005] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art, provide a synthesis of magnetic resonance multimodal super-resolution based on deep learning, design an unsupervised adversarial network based on deep learning, and realize the effective synthesis of cross-dataset magnetic resonance multimodal super-resolution images based on few-sample training through two stages.

[0006] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0007] A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning, comprising the following steps:

[0008] 1) Obtain raw medical data and analyze and preprocess the raw data;

[0009] 2) Construct high frequency template HF_mask;

[0010] 3) extracting the high-frequency information data_HF through the high-frequency template HF_mask;

[0011] 4) Channel stacking of the data data and the data high frequency information data_HF is performed, random cropping is performed on the stacked data, and a channel separation operation is performed after cropping;

[0012] 5) Build and train a multimodal synthesis network;

[0013] 6) Construct a multimodal super-resolution reconstruction network model;

[0014] 7) Transmitting multi-database multi-modal magnetic resonance images into the network built based on SRGAN for training;

[0015] 8) Applying the trained weights to the multimodal data generated in stage 1 to obtain a high-resolution multimodal MR image with a resolution multiple of ɑ;

[0016] 9) Apply multimodal super-resolution MRI to diverse downstream tasks.

[0017] In step 1), the specific steps of acquiring raw medical data and analyzing and preprocessing the raw data may be:

[0018] 1.1) Obtain multimodal public datasets;

[0019] 1.2) Use 3D medical image visualization software Slicer and ITK-SNAP to observe and analyze the images and obtain the correspondence between the X, Y, Z axes and the sagittal section, coronal section, and transverse section;

[0020] 1.3) Correct the spatial position of the multi-modal data in nii.gz format and align the multi-modal data:

[0021] 1.4) Perform cross-sectional slicing on the data in the dataset to obtain several layers of two-dimensional slices in .mat format;

[0022] 1.5) Preprocess the slices and normalize the image to an intensity value within the range of 0 to 1 using maximum normalization;

[0023] 1.6) The slices are uniformly sized so that the size of all slices is M x M, which is convenient for better matching the input applicability of the network;

[0024] 1.7) Perform outlier detection on all slices, set the INF values and NAN values in the two-dimensional slice data to 0, and剔除 damaged and abnormal data.

[0025] Preferably, steps 1.4) to 1.7) specifically include:

[0026] Perform Z-axis slicing on the data in the IXI and Simons datasets. For each sample, approximately 50 two-dimensional slices in the format of.mat with a size of M×M are obtained, located in the middle part of the brain rich in texture structure information. Normalize the image gray values to the range [0, 1] using the maximum normalization method. For the HCP dataset, perform slicing on the data in a specified direction when the b value is 1000, obtain several layers of data with a size of W×H and perform maximum normalization processing. Perform outlier detection and cleaning on the slices of all datasets, set the INF values and NAN values to 0, and剔除 damaged and abnormal data. Before feeding into the network, perform the resize operation in transform on the HCP data to adjust the size of all slices to M×M to meet the input requirements of the network.

[0027] In step 2), the specific steps for constructing the high-frequency template HF_mask can be:

[0028] 2.1) Set the scale factor and the center coordinates, and calculate the distance between any point in space and the center coordinates;

[0029] 2.2) Divide the matrix into three parts D1, D2, and D3 based on the distance;

[0030] 2.3) Assign values to the regions, set the value of D1 to 1, assign values to the D2 region through the Gaussian attenuation function. According to the distance from r1 to r2, set the value of D2 to decay from 1 to 0, and set the value of D3 to 0 to obtain the low-frequency template LF_mask, and calculate the high-frequency template HF_mask = 1 - LF_mask.

[0031] Preferably, step 2) specifically includes: Set the scale factors scale_factor1 and scale_factor2, set the parameters r1 = M / 2×scale_factor1, r2 = M / 2×scale_factor2 and r1 < r2, set the center coordinates as (centre_x, centre_y), and the coordinates of any point in space as (x, y). Set the distance between any point in space and the center coordinates as Divide the M x M two-dimensional matrix into three parts, namely D1 = Distance < r1, D2 = r1 < Distance < r2, and D3 = r2 < Distance < M / 2, and assign values to the regions: set the value of D1 to 1, assign values to D2 through a Gaussian decay function, with the value decaying from 1 to 0 according to the distance from r1 to r2, and set D3 to 0. Thus, the low-frequency template LF_mask is obtained.

[0032] The high-frequency template HF_mask = 1 - LF_mask;

[0033] In step 3), the specific steps for extracting the high-frequency information data_HF of the data can be as follows: perform FFT transformation (i.e., the efficient and fast algorithm of DFT) on the data to obtain its K-space, perform a fast Fourier transform shift on the K-space according to the fast Fourier transform shift function, and move the zero-frequency component of the fast Fourier transform (FFT) result to the center of the spectrum. Multiply the transformed K-space by HF_mask to obtain the high-frequency information of the K-space, and then perform a fast Fourier transform shift on the high-frequency information of the K-space and execute the IFFT (i.e., the efficient and fast algorithm of IDFT) transformation to obtain the high-frequency information data_HF of the data.

[0034] The formula for DFT is:

[0035]

[0036] where N is the number of points of the time-domain discrete signal, n represents the number of the time-domain discrete signal, and k represents the number of the frequency-domain signal;

[0037] The formula for IDFT is:

[0038]

[0039] In step 5), the specific steps for constructing and training the multi-modal synthesis network can be as follows:

[0040] 5.1) Construct a deep neural network with two channels as network inputs (data and data_HF respectively) and one channel as the output for multi-modal synthesis, that is, the multi-modal synthesis network;

[0041] 5.2) Based on SRGAN, construct a generator network; in the encoder part of the generator, use two different network designs to extract the high-frequency image features and the features of the original image respectively; in the decoder part of the generator, use transposed convolution, and at the same time fuse some features of data_HF into the decoder part;

[0042] 5.3) Construct a discriminator to identify the rationality of the generated image;

[0043] 5.4) Construct a reasonable network loss function, including cycle consistency loss function, perceptual loss function and generative adversarial loss function.

[0044] 5.5) Training multimodal synthesis network;

[0045] Preferably, the generator decoder part of the multimodal synthesis network specifically includes: the encoder finally converts the image with B×1×M / 2×M / 2 features extracted by feature downsampling into B×C 1 ×H 1 ×W 1 The upsampling block includes a 2d transposed convolution with a convolution kernel of 3, a BatchNorm2d, and an activation layer using the Relu activation function. The feature is passed through an upsampling block and becomes B×C 1 / 2×2H 1 ×2W 1 , and then extract B×C based on resnet34 2 / 2×2H 1 ×2W 1 Fusion to the feature block, and then upsample the feature to block B×(C 1 +C 2 / 2) / 2×4H 1 ×4W 1 , and then the high frequency features are B×C 2 / 2×4H 1 x 4W 1 The features of 1 +C 2 / 2) / 2+C 2 / 2 2 )×4H 1 ×4W 1 , enter the upsampling block again to get B×((C 1 +C 2 / 2) / 2+C 2 / 2) / 2×8H 1 ×8W 1 , and then with the high frequency feature B xC 2 / 4×8H 1 ×8W 1 The images are fused into a whole to achieve feature association; finally, the reconstructed image is obtained after two upsamplings.

[0046] In step 6), the specific steps of constructing the multimodal super-resolution reconstruction network model may be:

[0047] 6.1) Constructing a super-resolution network degradation model, including applying a low-frequency template to blur the magnetic resonance image, randomly downsampling the image, and randomly adding noise to the image;

[0048] 6.2) Construct a six-channel input and output training model that includes multiple magnetic resonance modes.

[0049] 6.3) Construct generator and discriminator networks based on SRGAN.

[0050] 6.4) Construct a multi-channel loss function, including pixel loss function, perceptual loss function and generative adversarial loss function.

[0051] Preferably, in step 6.1), a super-resolution network degradation model is constructed, which specifically includes three parts:

[0052] First, medical image sampling is used, that is, image domain is converted into K-space frequency domain, and low-frequency template data_LF is constructed according to step S23, and the low-frequency template is applied in the K-space region, and then the image is converted into image domain to obtain a blurred magnetic resonance image. Second, three different sampling modes, namely mode1, mode2 and mode3, are used to randomly downsample, and third, two noise modes, namely Noise1 and Noise2, are used to randomly add noise to the image.

[0053] The present invention also provides a synthesis system of cross-dataset magnetic resonance multimodal super-resolution images based on deep learning, which comprises in sequence: a data acquisition and analysis unit, a data preprocessing unit, a high-frequency template construction unit, a data high-frequency information extraction unit, a multimodal synthesis network structure construction unit, a multimodal synthesis network loss function unit, a super-resolution degradation model construction unit, a network construction unit of a super-resolution reconstruction generator and a discriminator, a super-resolution reconstruction loss function unit, and a data training and application unit;

[0054] The data acquisition and analysis unit is used to collect and preprocess original medical image data, including acquiring multimodal magnetic resonance images from multimodal public data sets, and using visualization software to analyze and determine the correspondence between the X, Y, and Z axes and the sagittal plane, coronal plane, and transverse plane; the multimodal public data sets include IXI data sets, HCP data sets, Simons data sets, etc.; the visualization software can be selected from Slicer and ITK-SNAP;

[0055] The data preprocessing unit is used to clean, normalize and slice the multimodal data to ensure the quality of the input data and adapt to the input requirements of the network;

[0056] The high-frequency template construction unit obtains a high-frequency template HF_mask by setting a two-dimensional matrix partition and using a Gaussian attenuation function to extract high-frequency information in the image according to the set scale factor and parameters;

[0057] The data high-frequency information extraction unit obtains the high-frequency information data_HF of the data through the high-frequency template; it includes an FFT transformation module, a K-space shift module, a high-frequency information extraction module, an inverse K-space shift module, an IFFT transformation module, and a data output module; the FFT transformation module is used to convert the input time domain data into the frequency domain through the fast Fourier transform FFT to obtain its K-space representation; the K-space shift module is used to perform a shift operation on the K-space obtained by the FFT transformation to move the zero-frequency component to the center of the spectrum; the high-frequency information extraction module is used to use the pre-constructed high-frequency template HF_mas k is element-by-element multiplied with the shifted K space to extract the high-frequency information part in the K space; the inverse K space shift module is used to perform a reverse shift operation on the K space after the high-frequency information is extracted to restore its original relative position so as to perform a subsequent IFFT transformation; the IFFT transformation module is used to convert the processed K space high-frequency information back to the time domain through the inverse fast Fourier transform IFFT to obtain the final high-frequency information data data_HF; the data output module is used to format the high-frequency information data data_HF obtained by the IFFT transformation and output it to the subsequent processing flow;

[0058] The multimodal synthesis network structure construction unit is used to construct a complex neural network structure, which can receive different types of input data and fuse information of different modes through multi-step feature extraction to generate unified multimodal output data;

[0059] The multimodal synthesis network loss function unit is used to construct a reasonable network loss function, including cycle consistency loss, generative adversarial loss function, and perceptual loss function;

[0060] The super-resolution model construction unit is used to construct a model that can convert an original magnetic resonance image into a low-resolution image (what is learned is the mapping from low resolution to original image); the super-resolution model construction unit includes a K-space processing module, a downsampling module and a noise addition module; the K-space processing module is used to convert the medical image from the image domain to the K-space frequency domain, and apply a low-frequency template in the K-space frequency domain to convert the image to the image domain to obtain a blurred magnetic resonance image; the noise addition module is used to randomly add noise to the image to finally obtain a low-resolution image.

[0061] The network construction unit of the generator and discriminator of the super-resolution reconstruction is used to create a generator to generate high-resolution images and a discriminator to evaluate the authenticity of the image, thereby optimizing the quality of the super-resolution image;

[0062] The super-resolution reconstruction loss function unit is used to create multiple loss functions to guide the model to learn how to improve image resolution; including pixel loss function, generation adversarial loss function, and perception loss function;

[0063] The data training and application unit is used to input the synthesized multimodal medical image data into the network constructed based on SRGAN for training to improve the resolution of magnetic resonance images, and apply the model to actual images to obtain high-resolution multimodal magnetic resonance image results.

[0064] Furthermore, a multimodal synthesis network structure construction unit is used to design and construct a deep neural network structure for multimodal image synthesis to generate high-quality multimodal super-resolution images by fusing information from different modalities; specifically, in this unit, a deep neural network with dual-channel input (data and data_HF) and single-channel output is constructed; the deep neural network is used for the synthesis of multimodal magnetic resonance images; the network adopts the framework of the generative adversarial network GAN, combining the structure of the generator and the discriminator; the generator is composed of an encoder, a decoder and a feature fusion mechanism; the discriminator adopts PatchGAN; specifically, the encoder uses the algorithm model Algorithm1 to extract data The texture detail feature information of data_HF is extracted, and the algorithm model Algorithm2 is used to extract the global features of data; the decoder uses an upsampling block containing transposed convolution to expand and reshape the resolution of the feature map. The upsampling block includes a 2D transposed convolution with a convolution kernel of 3, a BatchNorm2d layer, and an activation layer using a Relu activation function; in the decoding process, the network structure of Algorithm3 is used to fuse the texture detail features extracted by data_HF and the global features extracted by data; the final generated image is sent to the PatchGAN discriminator for evaluation to determine the authenticity and rationality of the generated image.

[0065] Furthermore, the network construction unit of the generator and the discriminator: the generator network and the discriminator network are constructed based on SRGAN; the RRDB block is used in the generator network as the backbone structure of the network model, and the discriminator network model adopts the spectral normalized U-Net discriminator.

[0066] The unsupervised method based on deep learning proposed in the present invention can utilize a large amount of medical imaging data, spanning data sets, and does not require paired modality samples. It automatically learns feature representation through the designed network, and learns complex nonlinear relationships from the data, and has better generalization ability; by training the deep learning model, it is possible to synthesize multiple modal images from a single modality MRI image, such as T1 weighted, T2 weighted, PD weighted, and T2* weighted, etc., to provide more comprehensive and accurate information support for medical imaging diagnosis; in addition, the present invention also focuses on the synthetic construction of super-resolution images of magnetic resonance multimodal images; synthetic super-resolution medical images can effectively improve the spatial resolution of images, so that medical radiologists can observe tissue structure and lesion characteristics more clearly, bringing greater convenience and benefits to clinical practice; this multimodal super-resolution image synthesis method based on deep learning can not only improve the accuracy and efficiency of diagnosis, but also reduce the workload of medical imaging diagnosticians, bringing greater convenience and benefits to clinical practice; the present invention combines K-space domain and image domain information and Swim The combination of Transformer and CNN network structures can realize the synthesis of multimodal magnetic resonance data across data sets and without relying on paired modal samples; the present invention can synthesize ultra-high resolution magnetic resonance images; the present invention realizes the effective combination of multimodal synthesis and super-resolution reconstruction, and can synthesize high-resolution multimodal magnetic resonance data, which to a certain extent solves the problems of modality missing, modality damage, unclear images, and low image resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 The present invention is a flowchart of a method for synthesizing magnetic resonance multimodal super-resolution images.

[0068] Figure 2 This is a network structure diagram of the magnetic resonance multimodal image synthesis method of an example of the present invention.

[0069] Figure 3 This is a network structure diagram of the magnetic resonance multi-modal super-resolution network reconstruction of an example of the present invention.

[0070] Figure 4 The following are the corresponding six modes synthesized by the magnetic resonance multi-modality synthesis method for a sample of the present invention. Among them, (a) is its T1-weighted image, (b) is its T2-weighted image, (c) is its PD-weighted image, (d) is its T2*-weighted image, (e) is its T2-flair-weighted image, and (f) is its DWI-weighted image.

[0071] Figure 5 This is a comparison diagram of an original magnetic resonance image before and after the super-resolution network degradation algorithm of an example of the present invention, where GT is the original image and LR is the low-resolution image after the degradation algorithm.

[0072] Figure 6 The figure shows an original T1 weighted image and its super-resolution image reconstructed by multi-modal magnetic resonance super-resolution in an example of the present invention, wherein GT is the original weighted image with a resolution of 256×256, and SR is the image with a resolution of 1024×1024 after super-resolution reconstruction and inference. DETAILED DESCRIPTION

[0073] In order to more clearly describe the purpose, technical solutions and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings. It should be pointed out that the described embodiments only represent some embodiments of the present invention, rather than all of them. Under the framework of the present invention, those skilled in the art can develop other embodiments based on the provided embodiments without performing additional creative work. All these embodiments should fall within the scope of protection of the present invention.

[0074] The present invention is further described below by specific embodiments:

[0075] Reference Figure 1 A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning, characterized in that it comprises the following steps:

[0076] S1 First, obtain the original medical data and analyze and preprocess the original data.

[0077] S11 obtains a multimodal public dataset; in this embodiment, it includes an IXI dataset (including T1-w, T2-w, and PD modalities), an HCP dataset (including T1-w, T2-w, and DWI modalities), a Simons dataset (including T1-w, T2-w, T2*, and T2-flair modalities), etc.;

[0078] S12 used 3D medical image visualization software Slicer and ITK-SNAP to observe and analyze the images and obtain the correspondence between the X, Y, Z axes and the sagittal section, coronal section, and transverse section;

[0079] S13 corrects the spatial position of the multimodal data in nii.gz format and uses the flirtlinear registration of FSL software to align the multimodal data;

[0080] S14 performs cross-sectional slicing processing on the data in the data set to obtain a plurality of layers of two-dimensional slices in a .mat format;

[0081] S15 adopts the method of maximum normalization: (x - x_min) / (x_max - x_min), where x is the pixel intensity value; the image is normalized to the range of intensity values from 0 to 1;

[0082] S16 simultaneously processes the slices to have a unified size. Through the operation of padding with 0 around, the size of all slices is made 256×256 to better match the input applicability of the network;

[0083] S17 performs outlier detection on all slices, sets the INF values and NAN values in the two-dimensional slice data to 0, and eliminates damaged and abnormal data.

[0084] S2 constructs the required high-frequency template HF_mask;

[0085] S21 sets the scale factor scale_factor1 to 0.10 and scale_factor2 to 0.15. Sets the parameter r1 = M / 2 × scale_factor1 to 12.80 and r2 = M / 2 × scale_factor2 to 19.20, and r1 < r2. In this case, the distance of the transition band is 19.2 - 12.80 = 6.40. Sets the center coordinates to (centre_x, centre_y), and the coordinates of any point in space to (x, y); sets the distance between any point in space and the center coordinates to

[0086] S22 divides the M x M two-dimensional matrix into three parts, namely D1 = Distance < 12.80, D2 = 12.80 < Distance < r2, and D3 = 19.20 < Distance < 128;

[0087] S23 assigns values to the regions: sets the value of D1 to 1, and assigns values to D2 through the Gaussian attenuation function; the formula of the Gaussian attenuation function is:

[0088]

[0089] According to the distance from r1 to r2, sets the value of D2 to decay from 1 to 0; sets D3 to 0; thus obtains the low-frequency template LF_mask, and the high-frequency template HF_mask = 1 - LF_mask. Partitioning the image into three parts and performing the Gaussian function attenuation is beneficial to avoiding the sinc filtering effect; obtains the high-frequency information data_HF of the data through the template;

[0090] S3 obtains the high-frequency information data_HF through the high-frequency template; performs FFT transformation (i.e., the efficient and fast algorithm of DFT) on the data to obtain its K space, and uses the window function to reduce the influence of the boundary effect; performs fast Fourier transform shift on the K space according to the fast Fourier transform shift function, and moves the zero-frequency component of the fast Fourier transform (FFT) (i.e., the efficient and fast algorithm of DFT) result to the center of the spectrum; after the transformation, the K space is obtained and multiplied by HF_mask to obtain the high-frequency information of the K space, and then performs fast Fourier transform shift on the high-frequency information of the K space and performs IFFT (i.e., the efficient and fast algorithm of IDFT) transformation to obtain data_HF;

[0091] The formula for DFT is:

[0092]

[0093] The formula for IDFT is:

[0094]

[0095] S4 performs channel stacking on data and data_HF. Channel stacking stacks the original image data data and the high-frequency information image data_HF in the channel dimension. If the original image is in color and has three RGB channels, the high-frequency information image may need to be copied or converted to the same number of channels as the original image so that it can be stacked. The stacked image has more channels, which contain the information of the original image and enhanced high-frequency information.

[0096] Random cropping is performed on the stacked data with a size of 128×128; multiple different cropped versions are generated from the original image to increase the diversity of the training dataset.

[0097] After cropping, a channel separation operation is performed to process or analyze the original image and the high-frequency information image separately; a multimodal synthesis network structure is constructed;

[0098] S5 reference Figure 2 , construct a multimodal synthesis network;

[0099] S51 constructs a deep neural network with dual channels as network input (data and data_HF respectively) and a single channel as output for multimodal synthesis;

[0100] The encoder part in the S52 generator uses two different network designs to extract high-frequency image features and original image features respectively; specifically, Resnet34 based on the resnet block as the backbone structure is used to extract the texture detail feature information of data_HF, and the Swim-T model with the Swim Transformer block as the backbone structure is used as the feature extractor to extract the global features of data;

[0101] The decoder part of the S53 generator uses transposed convolution to expand and reshape the resolution. Based on the U-net network structure, the features extracted by data_HF are integrated into the decoder. Specifically: the encoder finally extracts the features by downsampling, converting the 4×1×128×128 image into 4×1280×4×4; the upsampling block includes a 2d transposed convolution with a convolution kernel of 3, a BatchNorm2d, and an activation layer using the Relu activation function; the feature is passed through an upsampling block, 4×640×8×8, then extract 4×256×8×8 based on resnet34 and fuse it to the feature block, then upsample the feature to 4×448×16×16, then fuse the high-frequency feature of 4×128×16×16 to get 4×576×16×16, enter the upsampling block again to get 4×288×32×32, and then fuse it with the high-frequency feature of 4×64×32×32 to achieve feature association; finally, upsample twice to get the reconstructed image;

[0102] S54 constructs a discriminator to identify the rationality of the generated image;

[0103] S55 constructs a reasonable loss function of the multimodal synthesis network, including the cycle consistency loss function using L1 loss, the perceptual loss function using the VGG19 pre-trained network, and the generative adversarial loss function using BCEwithlogits loss; in the loss function, x represents the image in the source domain, y represents the image in the target domain, G and F are two generators, the generator G maps the image in the x domain to the image in the y domain, the generator F maps the image mapped to the y domain to the x domain, and D is the discriminator; the perceptual loss uses the pre-trained VGG19 network to extract the feature layer and calculates the difference between the generated image and the target image in the feature space; φ l is the feature extractor of the pre-trained network at layer l, l is the weight of the feature loss of the lth layer, and the feature extractors of the 3rd, 8th and 17th layers of VGG19 are selected; according to different synthesis tasks; λ cyc Set to 5.0~15.0, λ adv Set to 1.0~3.0 and λper Set it to 0.3~0.7, and different supernatural parameters are required according to different tasks;

[0104] The formula for the cycle consistency loss function is:

[0105]

[0106] The adversarial loss function formula of the generator is:

[0107]

[0108] The adversarial loss function formula of the discriminator is:

[0109]

[0110] The total loss function formula for generating adversarial loss is:

[0111] L adv (G,D,x,y)=L advG (G,D,x)+L advD (D,G,x,y)

[0112] The perceptual loss function formula is:

[0113]

[0114] The total loss function formula for multimodal synthesis is:

[0115] L syn (G,D,F,x,y)=λ cyc L cycle (G,F,x,y)+λ adv L adv (G,D,x,y)+λ per L perceptual (G,x,y)

[0116] When S56 trains the multimodal synthesis network, specifically: when generating T2* weighted and T2-flair modal data, 2000 sets of samples are used for training; when generating DWI modal data, 3000 sets of samples are used for training; the training batch is set to 4, the learning rate is initially set to 0.0002, and a total of 100 epochs are trained; at 50 epochs, the learning rate is reduced by Gaussian descent until the learning rate is set to 0 after 100 epochs.

[0117] S6 Figure 3 As shown, a multimodal super-resolution reconstruction network model is constructed;

[0118] S61 constructs a super-resolution degradation model, which includes three parts: blurring, downsampling, and noise addition; the blurring part adopts a blurring method that conforms to medical image sampling, and constructs a low-frequency template data_LF according to step S23, applies the low-frequency template to the K-space area corresponding to the medical image, and then converts the image into the image domain to obtain a blurred low-resolution magnetic resonance image; the downsampling part uses three different sampling modes, namely mode1, mode2 and mode3, to randomly downsample. In this embodiment, it can be randomly selected from the three modes of area, bicubic and bilinear, and the sampling multiple is α is set to 4; the noise adding part uses two modes of noise, namely Noise1 and Noise2, to randomly add noise to the image. In this embodiment, random Gaussion noise and Raleigh noise are used, and a random number z of 0 to 1 is taken as the noise judgment standard. When z≤0.3, Gaussian noise is added to the image, and when z≥0.7, Raleigh noise is added to the image. When z takes other values, no additional noise processing is performed on the image; in addition, the noise added to the six channels is inconsistent and random; in the experimental part, it is found that if sinc filtering is added to the noise part, it will cause adverse artifact effects after super-resolution, so sinc filtering is not added as part of the degradation model; Figure 5 It is the comparison result of a certain modality continuous layer image after the degradation algorithm, GT is the original image, LR is the low-resolution image after the degradation model;

[0119] S62 constructs a six-channel input and six-channel output training model including T1-w, T2-w, PD, T2flair, T2star, and DWI;

[0120] S63 builds the generator network and discriminator network based on SRGAN;

[0121] S64 constructs a multi-channel reasonable super-resolution reconstruction loss function, including a pixel loss function using L1loss, a perceptual loss function using the VGG19 pre-trained network, and a generative adversarial loss function using BCEwithlogits loss; in the loss function, GT is the real image, x is a low-resolution and low-definition image processed by the degradation algorithm; G is the generator, and G reconstructs x into the corresponding real image; φ l_sr is the feature extractor of the pre-trained network model at the lth layer. Specifically, the first 37 layers of VGG19 are used as feature extractors. l_sr The loss weight of the l-th layer feature is set to 1.0 by default. The formula of the pixel loss function is:

[0122]

[0123] The adversarial loss function formula of the generator is:

[0124]

[0125] The adversarial loss function formula of the discriminator is:

[0126]

[0127] The total loss function formula for generating adversarial loss is:

[0128] L adv_sr (G,D,x,GT)=L advG_sr (G,D,x)+L advD_sr (D,G,x,GT)

[0129] The perceptual loss function formula is:

[0130]

[0131] The total loss function formula for super-resolution reconstruction is:

[0132] L sr (G,D,x,GT)=λ pixel L pixel (G,GT,x)+λ adv_sr L adv_sr (G,D,x,GT)+λ per_sr L perceptual (G,x,GT)

[0133] S7 Figure 4 For the multimodal data shown in the figure, for the data of one modality or several modalities in the missing samples, the synthetic network can synthesize the corresponding 6 modal magnetic resonance data of the corresponding layer; after synthesizing these data, the multi-database multi-modal magnetic resonance images are transmitted to the network built based on SRGAN for training; specifically, the training batch is set to 4, the initial learning rate is set to 0.0001, a total of 200 epochs are trained, and the learning rate is attenuated by the gamma coefficient at 100 epochs, and gamma is set to 0.2.

[0134] S8 applies the trained weights to the multimodal data generated in stage 1 to obtain a high-resolution multimodal magnetic resonance image with a multiple of ɑ=4; Figure 6 As shown in the figure, GT is the original weighted image with a resolution of 256×256, and SR is the image with a resolution of 1024×1024 after super-resolution reconstruction and inference. It can be clearly seen that the image generated with high resolution has more obvious details and higher resolution.

[0135] S9 applies multimodal super-resolution of medical magnetic resonance images to diverse downstream tasks.

[0136] The present invention utilizes the unsupervised method in deep learning and adopts the architecture of generative adversarial network to realize the synthesis of magnetic resonance multimodal super-resolution images; different data sets have inconsistent modal information in many cases, some modalities may be missing, and the resolution is not high enough; this method designs a method to construct a multimodal synthesis network and an image super-resolution reconstruction network, and realizes the synthesis of magnetic resonance multimodal super-resolution images through two stages; stage one: first, the image is preprocessed, including registration, slicing, and normalization; then a high-frequency template is constructed, and the data is transformed by FFT to obtain its K-space Phase I: multiply the K-space image by the high-frequency template to obtain its K-space high-frequency information, and then perform IFFT transformation to obtain the high-frequency information in the image domain; secondly, send the prepared image and its high-frequency information as input to the network; through network training, the synthesis of multimodal data is realized; Phase II: first construct the image degradation algorithm of the super-resolution reconstruction network; then, construct the network architecture of super-resolution image reconstruction, and then, input the degraded image and the original real image into the network for training to obtain a high-resolution magnetic resonance image; finally, a high-resolution multimodal magnetic resonance image is obtained through phases I and II.

[0137] Phase 1 includes the following units:

[0138] Data acquisition and analysis unit: Specifically, in this unit, we first acquire the original medical data, analyze and preprocess them; the specific steps are as follows: acquire multimodal public data sets: including IXI data sets (including T1-w, T2-w, PD modalities), HCP data sets (including T1-w, T2-w, DWI modalities), Simons data sets (including T1-w, T2-w, T2*, T2-flair modalities), etc.; use medical image visualization software for observation and analysis: use Slicer and ITK-SNAP to observe and analyze the images to determine the correspondence between the X, Y, and Z axes and the sagittal, coronal, and transverse planes.

[0139] Data preprocessing unit: correct the spatial position of multimodal data, use FSL software for spatial registration to ensure the consistency of data in spatial position; slice the data: perform Z-axis slice processing on the data in the IXI and Simons datasets, and obtain approximately 50 layers of two-dimensional slices in .mat format with a size of M×M for each sample, located in the middle part of the brain rich in texture structure information; use maximum normalization to normalize the image grayscale value to the range [0,1]; for the HCP dataset, slice the data in a specified direction with a b value of 1000 to obtain several layers of data with a size of W×H and perform maximum normalization; perform outlier detection and cleaning on the slices of all datasets, set the INF value and NAN value to 0, and remove damaged and abnormal data; before feeding into the network, perform the resize operation in transform on the HCP data to adjust the size of all slices to M×M to meet the input requirements of the network.

[0140] High-frequency template construction unit: Set scale factors and parameters: Set scale factors scale_factor1 and scale_factor2, as well as parameters r1 and r2, and calculate the distance of the transition zone based on these parameters; Set two-dimensional matrix partitioning: Divide the M x M two-dimensional matrix into three parts D1, D2 and D3; According to the Gaussian attenuation function, the value of the D2 area is decayed from 1 to 0 to obtain the low-frequency template LF_mask, and then calculate the high-frequency template HF_mask; The formula of the Gaussian attenuation function is:

[0141]

[0142] Data high-frequency information extraction unit: obtain the high-frequency information data_HF of the data through the template; the specific steps are as follows: perform FFT transformation on the data (i.e., an efficient and fast algorithm of DFT) to obtain its K space, perform fast Fourier transform shift on the K space according to the fast Fourier transform shift function, and move the zero-frequency component of the fast Fourier transform (FFT) result to the center of the spectrum; after the transformation, the K space is obtained by multiplying it with HF_mask to obtain the high-frequency information of the K space, and then perform fast Fourier transform shift on the high-frequency information of the K space and perform IFFT transformation to obtain data_HF.

[0143] Multimodal synthesis network structure construction unit: construct a deep neural network with dual-channel input (data and data_HF) and single-channel output; the encoder uses the algorithm model Algorithm1 to extract the texture detail feature information of data_HF, and uses the algorithm model Algorithm2 to extract the global features of data; the decoder uses an upsampling block containing transposed convolution to expand and reshape the resolution, and uses the network structure of the algorithm model Algorithm3 to fuse the two parts of the features, which specifically fuses the features extracted by data_HF to the decoder; the encoder finally converts the feature downsampling extracted features of B×1×M / 2×M / 2 into B×C 1 ×H 1 ×W 1 ; The upsampling block includes a 2d transposed convolution with a convolution kernel of 3, a BatchNorm2d, and an activation layer using the Relu activation function; the feature is passed through an upsampling block, which is B×C 1 / 2×2H 1 ×2W 1 , and then extract B×C based on resnet34 2 / 2×2H 1 ×2W 1 Fusion to the feature block, and then upsample the feature to block B×(C 1 +C 2 / 2) / 2×4H 1 ×4W 1 , and then the high frequency features are B×C 2 / 2×4H 1 x 4W 1 The features of 1 +C 2 / 2) / 2+C 2 / 2 2 )×4H 1 ×4W 1 , enter the upsampling block again to get B×((C 1 +C 2 / 2) / 2+C 2 / 2) / 2×8H 1 ×8W 1 , and then with the high frequency feature B xC 2 / 4×8H 1 ×8W 1 The images are fused into a whole to achieve feature association; then PatchGAN is used as a discriminator to identify the rationality of the generated images; a PatchGAN discriminator is constructed to identify the rationality of the generated images.

[0144] Multimodal synthesis network loss function unit: construct reasonable network loss functions, including cycle consistency loss, generative adversarial loss function, and perceptual loss function.

[0145] Phase 2 includes the following units:

[0146] Super-resolution model construction unit: This unit consists of three parts. The first is to convert the image domain into the K-space frequency domain, construct a low-frequency template data_LF according to step S23, apply the low-frequency template in the K-space area, and then convert the image into the image domain to obtain a blurred magnetic resonance image; the second is to randomly downsample using three different sampling modes, namely mode1, mode2 and mode3; the third is to randomly add noise to the image using two noise modes, namely Noise1 and Noise2.

[0147] The network construction unit of the generator and the discriminator: The generator network and the discriminator network are constructed based on SRGAN; the RRDB block is used as the backbone structure of the network model in the generator network, and the discriminator network model adopts the spectrally normalized U-Net discriminator.

[0148] Super-resolution reconstruction loss function unit: construct reasonable loss functions, including pixel loss function, generative adversarial loss function, and perceptual loss function.

[0149] Data training and application unit: The synthesized multimodal medical image data is input into the network built based on SRGAN for training to obtain the network weights of multimodal super-resolution; the obtained single or modality-missing magnetic resonance image is input into the synthetic network of multimodal super-resolution, and the trained weights are used for inference to obtain a multimodal super-resolution magnetic resonance image with a resolution of ɑ times.

[0150] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning, characterized in that The steps include: 1) Obtain raw medical data and analyze and preprocess the raw data; 2) Construct high-frequency template HF_mask: set the scale factor and center coordinates, calculate the distance between any point in the space and the center coordinates; divide the two-dimensional matrix into three parts D1, D2, and D3 based on the distance; assign values ​​to the region, set the D1 value to 1, assign values ​​to the D2 region through the Gaussian attenuation function, set the D2 value to decay from 1 to 0 according to the distance from r1 to r2, set the D3 value to 0, obtain the low-frequency template LF_mask, and calculate the high-frequency template HF_mask = 1-LF_mask; 3) extracting the high-frequency information data_HF through the high-frequency template HF_mask constructed in step 2); 4) Channel stacking of the data data and the data high frequency information data_HF is performed, random cropping is performed on the stacked data, and a channel separation operation is performed after cropping; 5) Build and train a multimodal synthesis network; 5.1) Construct a deep neural network with dual channels as network input, namely data and data_HF, and a single channel as output for multimodal synthesis; 5.2) Constructing a generator network based on SRGAN; The encoder part in the generator uses two different network designs to extract high-frequency image features and original image features respectively; The decoder part of the generator uses transposed convolution and uses some features of data_HF to fuse into the decoder part; 5.3) Construct a discriminator to identify the rationality of the generated image; 5.4) Construct a reasonable network loss function, including cycle consistency loss function, perceptual loss function and generative adversarial loss function; 5.5) Training multimodal synthesis network; 6) Construct a multimodal super-resolution reconstruction network model; 6.1) Constructing a super-resolution network degradation model, including applying a low-frequency template to blur the magnetic resonance image, randomly downsampling the image, and randomly adding noise to the image; 6.2) Construct a six-channel input and output training model that includes multiple magnetic resonance modes; 6.3) Construct generator and discriminator networks based on SRGAN; 6.4) Construct multi-channel loss functions, including pixel loss function, perceptual loss function and generative adversarial loss function; 7) Transmitting multi-database multi-modal magnetic resonance images into the network built based on SRGAN for training; 8) Applying the trained weights to the multimodal data generated in stage 1 to obtain a high-resolution multimodal super-resolution magnetic resonance image with a resolution multiple of ɑ; 9) Apply the multimodal super-resolution magnetic resonance image obtained in step 8) to a variety of downstream tasks.

2. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 1, characterized in that In step 1), the specific steps of obtaining raw medical data and analyzing and preprocessing the raw data are: 1.1) Obtain multimodal public datasets; 1.2) Use 3D medical image visualization software Slicer and ITK-SNAP to observe and analyze the images and obtain the correspondence between the X, Y, Z axes and the sagittal plane, coronal plane, and transverse plane; 1.3) Correct the spatial position of the multi-modal data in nii.gz format and align the multi-modal data: 1.4) Perform cross-sectional slicing on the data in the dataset to obtain several layers of two-dimensional slices in .mat format; 1.5) Preprocess the slices and normalize the image to an intensity value within the range of 0 to 1 using maximum normalization; 1.6) The slices are uniformly sized so that the size of all slices is M×M, which is convenient for better matching the input applicability of the network; 1.7) Perform outlier detection on all slices, set the INF value and NAN value in the two-dimensional slice data to 0, and remove damaged and abnormal data.

3. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 2, characterized in that Steps 1.4) to 1.7) specifically include: The data in the IXI and Simons datasets were sliced ​​along the Z axis, and each sample obtained approximately 50 layers of two-dimensional slices in the .mat format with a size of M×M, located in the middle part of the brain rich in texture structure information; the image grayscale value was normalized to the range [0,1] using the maximum value normalization method; for the HCP dataset, the data was sliced ​​in the specified direction with a b value of 1000 to obtain several layers of data with a size of W×H and then subjected to maximum value normalization; outlier detection and cleaning were performed on the slices of all datasets, the INF value and NAN value were set to 0, and damaged and abnormal data were removed; before being fed into the network, the resize operation in the transform was performed on the HCP data to adjust the size of all slices to M×M to meet the input requirements of the network.

4. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 1, characterized in that In step 2), the specific steps of constructing the high-frequency template HF_mask are: Set the scale factors scale_factor1 and scale_factor2, set the parameters r1 = M / 2 × scale_factor1, r2 = M / 2 × scale_factor2 and r1 < r2, set the center coordinates as (centre_x, centre_y), and the coordinates of any point in space as (x, y); set the distance between any point in space and the center coordinates as Divide the M×M two-dimensional matrix into three parts, namely D1 = Distance < r1, D2 = r1 < Distance < r2, and D3 = r2 < Distance < M / 2, and assign values to the regions: set the value of D1 to 1, assign values to D2 through the Gaussian decay function, according to the distance from r1 to r2, set the value to decay from 1 to 0, and set D3 to 0; thus obtain the low-frequency template LF_mask, and the high-frequency template HF_mask = 1 - LF_mask.

5. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 1, characterized in that In step 3), the specific steps of extracting the data high-frequency information data_HF are: performing FFT transformation on the data to obtain its K space, performing fast Fourier transform shift on the K space according to the fast Fourier transform shift function, and moving the zero-frequency component of the fast Fourier transform FFT result to the center of the spectrum; performing the product of the K space obtained after the transformation and HF_mask to obtain the K space high-frequency information, and then performing fast Fourier transform shift on the K space high-frequency information and performing IFFT transformation to obtain the data high-frequency information data_HF; The formula for DFT is: The formula for IDFT is:

6. A method for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 1, characterized in that In step 6.1), a super-resolution network degradation model is constructed, which specifically includes three parts: first, medical image sampling is used, that is, the image domain is converted into the K-space frequency domain. According to the constructed low-frequency template data_LF, the low-frequency template is applied in the K-space area, and then the image is converted into the image domain to obtain a blurred magnetic resonance image; second, three different sampling modes, namely mode1, mode2 and mode3, are used for random down-sampling; third, two noise modes, namely Noise1 and Noise2, are used to randomly add noise to the image.

7. A deep learning-based cross-dataset magnetic resonance multimodal super-resolution image synthesis system, characterized in that Including in order: Data acquisition and analysis unit, data preprocessing unit, high-frequency template construction unit, data high-frequency information extraction unit, multimodal synthesis network structure construction unit, multimodal synthesis network loss function unit, super-resolution degradation model construction unit, super-resolution reconstruction generator and discriminator network construction unit, super-resolution reconstruction loss function unit, data training and application unit; The data acquisition and analysis unit is used to collect and preprocess original medical image data, including acquiring multimodal magnetic resonance images from multimodal public data sets, and using visualization software to analyze and determine the correspondence between the X, Y, and Z axes and the sagittal plane, coronal plane, and transverse plane; the multimodal public data sets include the IXI data set, the HCP data set, and the Simons data set; the visualization software uses Slicer and ITK-SNAP; The data preprocessing unit is used to clean, normalize and slice the multimodal data to ensure the quality of the input data and adapt to the input requirements of the network; The high-frequency template construction unit obtains a high-frequency template HF_mask by setting a two-dimensional matrix partition and using a Gaussian attenuation function to extract high-frequency information in the image according to the set scale factor and parameters; The data high-frequency information extraction unit obtains the high-frequency information data_HF of the data through the high-frequency template; It includes an FFT transformation module, a K space shift module, a high-frequency information extraction module, an inverse K space shift module, an IFFT transformation module, and a data output module; the FFT transformation module is used to transform the input time domain data into the frequency domain through the fast Fourier transform FFT to obtain its K space representation; the K space shift module is used to perform a shift operation on the K space obtained by the FFT transformation to move the zero-frequency component to the center of the spectrum; the high-frequency information extraction module is used to perform element-by-element multiplication of the shifted K space using a pre-built high-frequency template HF_mask to extract the high-frequency information part in the K space; the inverse K space shift module is used to perform a reverse shift operation on the K space after the high-frequency information is extracted to restore its original relative position so as to perform subsequent IFFT transformation; the IFFT transformation module is used to transform the processed K space high-frequency information back to the time domain through the inverse fast Fourier transform IFFT to obtain the final high-frequency information data data_HF; the data output module is used to format the high-frequency information data data_HF obtained by the IFFT transformation and output it to the subsequent processing flow; The multimodal synthesis network structure construction unit is used to construct a complex neural network structure, which can receive different types of input data and fuse information of different modes through multi-step feature extraction to generate unified multimodal output data; The multimodal synthesis network loss function unit is used to construct a reasonable network loss function, including cycle consistency loss, generative adversarial loss function, and perceptual loss function; The super-resolution degradation model construction unit is used to construct a model that can degenerate the original magnetic resonance image into a low-resolution image. The super-resolution degradation model construction unit includes a K-space processing module, a downsampling module and a noise adding module; the K-space processing module is used to convert the medical image from the image domain to the K-space frequency domain, and apply a low-frequency template in the K-space frequency domain to convert the image into the image domain to obtain a blurred magnetic resonance image; the noise adding module is used to randomly add noise to the image to finally obtain a low-resolution image; The network construction unit of the generator and discriminator of the super-resolution reconstruction is used to create a generator to generate high-resolution images and a discriminator to evaluate the authenticity of the image, thereby optimizing the quality of the super-resolution image; The super-resolution reconstruction loss function unit is used to create multiple loss functions to guide the model to learn how to improve image resolution; Including pixel loss function, generative adversarial loss function, and perceptual loss function; The data training and application unit is used to input the synthesized multimodal medical image data into the network constructed based on SRGAN for training to improve the resolution of magnetic resonance images, and apply the model to actual images to obtain high-resolution multimodal magnetic resonance image results.

8. A system for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 7, characterized in that The multimodal synthesis network structure construction unit: constructs a deep neural network with dual-channel input of data and data_HF and single-channel output; the deep neural network is used for the synthesis of multimodal magnetic resonance images; the network adopts the framework of generative adversarial network GAN, combining the structure of generator and discriminator; the generator is composed of an encoder, a decoder and a feature fusion mechanism; the discriminator adopts PatchGAN; specifically, the encoder uses algorithm model Algorithm1 to extract texture detail feature information of data_HF, and uses algorithm model Algorithm2 to extract global features of data; The decoder uses an upsampling block containing transposed convolution to expand and reshape the resolution of the feature map. The upsampling block includes a 2D transposed convolution with a convolution kernel of 3, a BatchNorm2d layer, and an activation layer using the Relu activation function. During the decoding process, the texture detail features extracted by data_HF and the global features extracted by data are fused using the network structure of Algorithm3. The final generated image is sent to the PatchGAN discriminator for evaluation to determine the authenticity and rationality of the generated image.

9. A system for synthesizing cross-dataset magnetic resonance multimodal super-resolution images based on deep learning as claimed in claim 7, characterized in that The network construction unit of the generator and the discriminator: the generator network and the discriminator network are constructed based on SRGAN; the RRDB block is used as the backbone structure of the network model in the generator network, and the discriminator network model adopts the spectrally normalized U-Net discriminator.

Images