Multi-modal MRI-assisted Reconstruction Method Based on SPRNN and Multi-scale Reference Modes

Abstract

A fast MRI assisted reconstruction method based on SPRNN and multi-scale reference modalities, comprising the following steps: Step 1: Input the u-fold undersampled reference modality image #imgabs0# and the fully sampled reference modality image #imgabs1#; Step 2: Obtain the reconstructed pending modality image #imgabs3# from the deep sub-pixel recursive neural network #imgabs2#. Figure 1 shows the design scheme of the deep sub-pixel recursive neural network f SPRNN (y u , y′; Θ). The present invention combines sub-pixel convolution, data consistency operator, multi-scale reference modalities with multi-modal MRI assisted reconstruction technology, and gradually reconstructs the highly undersampled MR images based on the recursive neural network, which can effectively reduce the influence of a large number of artifacts in the input data on the reconstruction effect, and can effectively improve the reconstruction quality of the MRI images of the pending modality.

Description

Technical Field

[0001] The present invention relates to the field of magnetic resonance image processing, and mainly provides technical support for the accelerated scanning of multi-modal magnetic resonance images: in the post-processing stage, multi-scale reference modalities are used to assist in reconstructing highly undersampled MRI images. Background Art

[0002] Magnetic Resonance Imaging (MRI) is one of the most important non-invasive examination methods in current clinical medical imaging. It is one of the few safe clinical diagnostic methods that cause no harm to the human body and has been widely used in various clinical applications such as angiography, neurology, and cardiology examinations. In order to comprehensively analyze diseases, multi-modal MRI is often required in clinical practice (this is also a major advantage of magnetic resonance imaging compared to other imaging methods). For example, T1-weighted images (T1WI) and T2-weighted images (T2WI) are two of the most commonly used MRI sequences in clinical evaluation of anatomical structures and pathological analysis, and their functions are different: T1WI is often used to detect adipose tissue, locate focal liver lesions, describe lesion morphological information, etc., while T2WI is often used to detect edema and inflammation, discover white matter lesions in the brain, evaluate zonal structures in the prostate and uterus, etc. Obviously, the combination of multi-modal magnetic resonance images is more helpful for doctors to make accurate disease diagnoses.

[0003] However, due to the limitations of magnetic resonance imaging principles, compared with imaging methods such as CT, the imaging speed of MRI is too slow and the cost is also higher. This problem is particularly significant for multi-modal magnetic resonance imaging: different MRI modalities need to be scanned sequentially, and each modality scan takes a relatively long time. In order to accelerate the speed of multi-modal magnetic resonance imaging, in recent years, researchers have proposed auxiliary reconstruction methods for multi-modal magnetic resonance imaging. Such methods suggest that during the multi-modal magnetic resonance imaging stage, full sampling is performed on one modality with a relatively fast imaging speed (usually T1WI), and this modality is called the "reference modality"; while for other modalities with slower imaging speeds (such as T2WI, PD, FLAIR, etc.), high-fold undersampling scans are performed, and these modalities are called "pending modalities". Since only the reference modality is fully sampled while multiple other modalities are highly undersampled, the MRI scanning speed can be greatly improved. The feasibility of such methods is based on the fact that multiple magnetic resonance imaging modalities of the same tissue are highly correlated, and using this high correlation, the fully sampled "reference modality" can be used to assist in reconstructing the "pending modalities" during the post-processing stage. Multi-modal assisted reconstruction requires the MRI images between the reference modality and the pending modalities to be aligned with each other, and this requirement results in usually less data available for training the multi-modal assisted reconstruction model.

[0004] Existing multi-modal assisted reconstruction methods are mainly based on U-Net. The fully sampled "reference modality" and a highly undersampled "pending modality" are used as two channels and input into the network simultaneously. Utilizing the information fusion ability of the convolutional neural network, the detailed information in the reference modality is "transferred" to the pending modality to achieve good reconstruction of the pending modality. The main deficiencies of such methods are: (1) The parameter scale of the U-Net network adopted is usually large and it is difficult to be applicable to small-scale multi-modal training data; (2) A large number of zero-fillings are performed on the MRI images of the highly undersampled pending modality at the beginning (network input stage). A large number of zero-fillings will not only increase the additional computational overhead, but also introduce obvious artifacts in the reconstructed MRI images, resulting in a relatively low signal-to-noise ratio of the reconstructed MRI images; (3) The participation degrees of both the fully sampled "reference modality" and the highly undersampled "pending modality" in the forward propagation of the network are very low, which will lead to the detailed information contained in the "reference modality" not being fully utilized, and the real sampling information contained in the "pending modality" being destroyed during the forward propagation process. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, eliminate a large number of artifacts existing in high-fold undersampled MRI images, and efficiently utilize the limited network capacity to learn complex end-to-end mappings, the present invention combines sub-pixel convolution, deep recurrent networks, multi-scale reference modalities, and a multi-scale supervised training method to propose a multi-modal MRI reconstruction method based on a deep sub-pixel recurrent neural network (SPRNN). Among them, the sub-pixel convolution technology adopted can effectively reduce the influence of a large number of artifacts existing in the input data on the reconstruction effect. The recurrent network structure can not only improve the participation of the reference modality and the to-be-determined modality in the forward propagation process of the network, but also effectively compress the parameter scale of the neural network. The multi-scale supervision of the recurrent network can gradually reconstruct the high-frequency information in the high-fold undersampled MRI image.

[0006] The specific technical solution provided by the present invention is as follows:

[0007] A multi-modal MRI assisted reconstruction method based on a deep sub-pixel recurrent neural network (Sub-Pixel Recursive Neural Network, SPRNN) and multi-scale reference modalities, comprising the following steps:

[0008] Step 1: Input the u-fold undersampled reference modality image and the fully sampled reference modality image Denote the MRI image of the u-fold undersampled to-be-determined modality in the k-space, Denote the k-space where the MRI image is located, Denote the dimension of y u and N is the dimension of the fully sampled MRI image;

[0009] Step 2: Obtain the reconstructed to-be-determined modality image by the deep sub-pixel recurrent neural network f SPRNN ((·,·); Θ):

[0010] Furthermore, in the above Step 1, the undersampled MRI image y u has the following relationship with the fully sampled MRI image:

[0011] y u = M u y = M u Fx,

[0012] where denotes the fully sampled image of y u in the k-space, denotes the fully sampled image of y u in the pixel space, denotes the pixel space where the MRI image is located, is a Fourier transform matrix used to transform the MRI image from the pixel space to the k-space; M u ∈ {0, 1} M×N represents a masking matrix for undersampling y by u times.

[0013] Furthermore, the deep sub-pixel recursive neural network f in step 2 SPRNN ((·,·); Θ) is defined as follows:

[0014] f SPRNN (y u , y′; Θ) = F H f D (f D-1 ((…f 2 (f 1 (y u , y′; Θ 1 ), y′; Θ 2 ), y′; Θ D-1 ), y′; Θ D )

[0015] where Θ = {Θ 1 , …, Θ D} is the parameter set of the deep sub-pixel recursive neural network f SPRNN , H represents the Hermitian transpose operation, D is the number of subnets of f SPRNN , f d ((·,·); Θ d ) is the d-th subnet of f SPRNN and is defined as follows:

[0016]

[0017] Here, d ∈ {1, 2, …, D},

[0018] Furthermore, in equation (1), is the sub-pixel upsampling operator, which means mapping the MRI image with u-fold undersampling in the k-space to the MRI image with u d-1 ((u d (u d < u d-1 ))-fold undersampling, is its parameter. For the input MRI image it is defined as follows:

[0019]

[0020] where is a partial zero-padding operator, and SPC(·) represents a sub-pixel convolution operator.

[0021] Furthermore, in equation (1), represents central downsampling of the fully undersampled MRI image in the k-space, such that the dimension of the MRI image after sampling is M d .

[0022] In equation (1), is a reconstruction module, which represents a network module for reconstructing and calculating the MRI image in the pixel domain. θ is its parameter. For the input k-space data of the pending modality and the k-space data of the reference modality R((·,·);θ) is composed of B sub-modules cascaded. θ = {θ 1 ,θ 2 ,…,θ B}, and θ b is 's parameter. The sub-module is defined as follows:

[0023]

[0024] Among them, x 0 =F H k, Cat(·,·) represents concatenation of the input image in the channel direction, is an activation function.

[0025] Furthermore, in equation (1), represents a data consistency operator. Let The elements in are calculated in the following way:

[0026]

[0027] Among them, Ω represents the index subset of the elements in y u used to construct y.

[0028] Furthermore, the multi-scale depth supervision method is used for training the deep sub-pixel recursive neural network f SPRNN (·;Θ) in step 2, and the steps are as follows:

[0029] Step 2.1 Construct a multi-scale MRI image set for training the deep sub-pixel recursive neural network f SPRNN (·;Θ)

[0030]

[0031] Here, Γ = {1, 2, …, L} is the index set of training samples, L is the number of training samples, and u = u 0 > u 1 > u 2 > … > u D are different undersampling multiples of the MRI image, is the reference modal data, is the MRI image of the γ-th full-sampled pending modal, is of dimension the MRI image of the pending modal, is obtained by the following method:

[0032]

[0033] where, represents the mask matrix for undersampling the MRI image in the k-space by u d times; since each training sample in is composed of subsamples of different dimensions, is called a multi-scale training set;

[0034] Step 2.2 Let the iteration number t = 0, and randomly initialize the parameter set Θ of f SPRNN (·; Θ) as Θ (0) :

[0035]

[0036] Step 2.3 Let t = t + 1, and minimize the following loss function by the gradient descent method:

[0037]

[0038] where, l(·, ·) represents the distance metric operation between two images;

[0039] Step 2.4 Iteratively execute the above Step 2.3 until t = T, where T is the preset maximum number of iterations;

[0040] Step 2.5 Select Θ = Θ (T) as the optimal parameter set of f SPRNN (·; Θ).

[0041] The technical concept of the present invention is as follows: There are mainly three deficiencies in the existing multi-modal assisted reconstruction methods: (1) The parameter scale of the network is relatively large and it is difficult to be applied to small training data; (2) Zero-padding is directly performed on the highly undersampled MRI images, resulting in additional computational overhead and low signal-to-noise ratio; (3) The low participation of the fully sampled "reference modality" and the highly undersampled "to-be-determined modality" in the forward propagation of the network leads to the fact that the detailed information in the "reference modality" and the real sampling information contained in the "to-be-determined modality" cannot be fully utilized. To overcome these deficiencies, the present invention proposes a new multi-modal MRI assisted reconstruction method based on the Sub-Pixel Recursive Neural Network (SPRNN) and multi-scale reference modality. SPRNN mainly includes two parts: sub-pixel convolution and recursive neural network. Here, the role of using the recursive neural network mainly has two aspects: on the one hand, it can effectively save the parameter scale of the network, so that it can effectively adapt to small-scale multi-modal training data; on the other hand, it can improve the participation of the reference modality and the to-be-determined modality in the forward propagation process of the network, so that the network can form a long-term memory of the data in the reference modality and the to-be-determined modality. The use of sub-pixel convolution is to avoid a large amount of zero-padding for the MRI data of the to-be-determined modality and avoid the noise information and aliasing artifacts introduced by a large amount of zero-padding. In addition, since the undersampling rate of the to-be-determined modality is very high, in order to avoid using sub-pixel convolution to estimate all the k-space data that have not been acquired in the to-be-determined modality at one time (the computational cost and error estimation rate of doing so are very high), the SPRNN proposed by the present invention adopts a strategy of gradually upsampling, so that each recursive subnet only restores a part of the undersampled data. This strategy requires the reference modality data used to be decomposed at multiple scales in the k-space to specifically assist in reconstructing the unsampled data in the corresponding interval of the to-be-determined modality.

[0042] Compared with the existing deep learning-based multi-modal MRI assisted reconstruction methods, the beneficial effects of the present invention are mainly reflected in: (1) By adopting the sub-pixel convolution module, the artifacts in the input data are reduced, enabling the network to fully extract features from less interfered data and enhancing the reconstruction effect; (2) Fully drawing on the idea of weight sharing of the recursive neural network, it can greatly save the storage space of the network model and the scale of training data; (3) At the same time, it improves the participation of the reference modality and the to-be-determined modality in the forward propagation of the network; (4) The network training uses a multi-scale deep supervision method, which can guide the network to gradually reconstruct high-precision MRI images from low-frequency information to high-frequency information, thereby improving the training efficiency and reconstruction effect of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is the deep sub-pixel recursive neural network f designed by the present invention SPRNNThe overall structure diagram, where y u is the u-fold undersampled MRI image in the input k-space, and are the reconstructed images of y in the output k-space and pixel space respectively u ; is the intermediate output result of the d-th subnet and f DC is the data consistency operator (for its operation examples, see Figure 8 ); F H represents the inverse Fourier transform; U d is the upsampling operator of the d-th subnet of f SPRNN , and its specific definition can be found in Figure 2 (a); R is the reconstruction module shared by each subnet of f SPRNN , and its specific definition can be found in Figure 2 (b).

[0044] Figure 2 (a) is the structure diagram of the upsampling operator U SPRNN of the d-th subnet of f d , where Conv d,j represents the j-th convolutional layer of U d , SPC represents the sub-pixel convolutional operator (for its operation examples, see Figure 6 ), and PZF represents the partial zero-padding (for its operation examples, see Figure 5 ); Figure 2 (b) is the structure diagram of the reconstruction module R shared by each subnet of f SPRNN , which is a recursive residual network. The definition of the convolutional module C b (b ∈ {1, 2, …, B}) in R can be found in the sub-figure (c), where Conv b-i represents the i-th convolutional layer of C b .

[0045] Figure 3 (a) and (b) are examples of the 8-fold undersampled MRI image and the fully sampled MRI image in the k-space respectively, and (c) and (d) are examples of the reconstructed MRI image and the fully sampled MRI image in the pixel space respectively.

[0046] Figure 4 (a) is an example of the central undersampling mask matrix, Figure 4 (b) is an example of the Gaussian undersampling mask matrix.

[0047] Figure 5Description of the specific operations of the partial zero-padding operator: (a), (b), and (c) are the results of partial zero-padding of MRI images undersampled by 8 times, 2.67 times, and 1.33 times in the k-space respectively. The dimensions of the padded MRI images are and N, where N = 336.

[0048] Figure 6 Description of the specific operations of the sub-pixel convolution operator. Here, the sub-pixel convolution operator combines three feature maps in the channel direction in a certain order, and triples the dimension of the input undersampled MRI image.

[0049] Figure 7 Comparison of the curves of two activation functions ReLU and LReLU. Subfigure (a) shows the curve of ReLU, and subfigure (b) shows the curve of LReLU.

[0050] Figure 8 Explanation of the data consistency operator f DC where subfigure (a) is the input undersampled MRI image, subfigure (b) is the reconstructed MRI image, and subfigure (c) is the MRI image after performing data consistency calculation on subfigure (b) using the information in subfigure (a). Essentially, the data consistency operator replaces the information in the corresponding frequency band of the reconstructed MRI image with the frequency information (mainly low-frequency information) of the input undersampled MRI image.

[0051] Figure 9 Explanation of the data consistency operator f DC Effect demonstration of the data consistency operator f: The first two columns compare the changes in the MRI images obtained by three subnets of the deep sub-pixel recursive neural network (the subfigures in the first two columns of each row are the outputs of one subnet) before and after processing with f DC It can be seen that the MRI images obtained before data consistency processing (the first column) mainly show their details, while the MRI images obtained after data consistency processing (the second column) are closer to the real images (the third column).

[0052] Figure 10 Examples of 5 MR training samples for training the deep sub-pixel recursive neural network. Each row is a training sample, and each training sample consists of 5 MRI images: the MRI image undersampled by u times in the k-space (the first column), the undersampled MRI images with u 1 = 6 times, u 2 = 4 times, u 3 = 2 times (the second to fourth columns), and the fully sampled MRI image (the last column).

[0053] Figure 11 Quantitative comparison of the PSNR / SSIM values of the reconstruction results of the deep sub-pixel recursive neural network (SPRNN) proposed in the present invention, the traditional zero filling method (Zero Filling, ZF), and five cutting-edge methods UNet, DenseUNet, RefineGAN, D5C5, and D5C15, with T1WI as the reference modality, for two undetermined modalities T2WI and DP at downsampling ratios of 1 / 4, 1 / 8, and 1 / 12 respectively. The MRI images of 8 subjects were used as training samples, and the MRI images of the remaining 2 subjects were used as test samples. RM and QM represent the reference modality and the undetermined modality respectively.

[0054] Figure 12 It is the auxiliary reconstruction process of the deep sub-pixel recursive neural network (SPRNN) proposed in the present invention for the 8-fold undersampled undetermined modality based on the multi-scale reference modality, where represents the result of reconstructing the input undersampled undetermined modality image to scale u using the sub-pixel upsampling subnet d after and represent the reference modality image and the real image of the undetermined modality at scale u d respectively, and is the image of the undetermined modality after d recursive reconstructions by SPRNN.

[0055] Figure 13 It is a visual comparison of the reconstruction effects of the deep sub-pixel recursive neural network (SPRNN) proposed in the present invention and the comparison methods (ZF, UNet, DenseUNet, RefineGAN, D5C5, D5C15). Here, the reference modality is T1WI, and the undetermined modality is 12-fold undersampled T2WI. Three regions of interest were selected from each reconstructed image and placed in the subgraphs corresponding to the respective methods below in the order of "from left to right, from top to bottom" after magnification. Detailed implementation manner

[0056] The implementation steps of the present invention are further described below with reference to the accompanying drawings.

[0057] Referring to Figures 1 to 13 , a multi-modal MRI reconstruction method based on a deep sub-pixel recursive neural network (Sub-Pixel Recursive Neural Network, SPRNN) and a multi-scale reference modality includes the following steps:

[0058] Step 1: Input the u-fold undersampled reference modality image and the full-sampled reference modality image

[0059] Step 2 is performed by the deep sub-pixel recursive neural network f SPRNN ((·,·); Θ): to obtain the reconstructed modal image to be determined Figure 1 Figure gives f SPRNN (·; Θ)'s specific network structure diagram.

[0060] Furthermore, in the said Step 1, represents the undersampled MRI image that is u times in the k-space, represents the k-space where the MRI image is located, represents y u 's dimension, and N is the dimension of the fully sampled MRI image; the undersampled MRI image y u and the fully sampled MRI image have the following relationship:

[0061] y u = M u y = M u Fx,

[0062] wherein, represents the fully sampled MRI image of y u in the k-space, represents the fully sampled MRI image of y u in the pixel space. Here, represents the pixel space where the MRI image is located, is the Fourier transform matrix for transforming the MRI image from the pixel space to the k-space; M u ∈ {0, 1} M×N represents the mask matrix for undersampling y by u times. Figure 3 (a) gives an example of y u with u = 8 times undersampling, Figure 3 (b) and (d) respectively give examples of the fully sampled (N = 336) MRI images y and x in the k-space and the pixel space; the mask matrix M u can use the central undersampling matrix or the Gaussian undersampling matrix, as Figure 4 shown, Figure 3 (a) is the MRI image in the k-space obtained by central undersampling of Figure 3 (b).

[0063] Furthermore, in the said Step 2, the deep sub-pixel recursive neural network f SPRNN ((·,·); Θ) is defined as follows:

[0064] f SPRNN (y u , y′; Θ) = F H fD (f D-1 ((…f 2 (f 1 (y u ,y′; Θ 1 ),y′; Θ 2 )),y′; Θ D-1 ),y′; Θ D )

[0065] where Θ = {Θ 1 ,…, Θ D} is the parameter set of the deep sub - pixel recursive neural network f SPRNN , H represents the Hermitian transpose operation, D is the number of sub - networks of f SPRNN , and f d ((·,·); Θ d ) is the d - th sub - network of f SPRNN , which is defined as follows:

[0066]

[0067] where d ∈ {1, 2,…, D}, Here, the number of sub - networks has a great influence on the reconstruction effect of the network. Usually, D can be set to 3 or 5.

[0068] Furthermore, in equation (2), is the sub - pixel up - sampling operator, which maps the k - space u d-1 times undersampled MRI image to the pixel - space u d (u d < u d-1 ) times undersampled MRI image, is its parameter. For the input MRI image , it is defined as follows:

[0069]

[0070] where is the partial zero - padding operator, and SPC(·) represents the sub - pixel convolution operator. Figure 2 (a) shows the network structure of U d . Figure 5 The specific operation of the partial zero - padding operator is illustrated, Figure 5 (a), (b), and (c) are the results of partial zero - padding for 8 - fold, 2.67 - fold, and 1.33 - fold undersampled MRI images respectively. The dimensions of the padded MRI images are and N, where N = 336; Figure 6The specific operation of the sub-pixel convolution operator is described. Here, the sub-pixel convolution operator combines three feature maps in the channel direction in a certain order, increasing the dimension of the input undersampled MRI image by three times.

[0071] Furthermore, in Equation (2), denotes central undersampling of the fully undersampled MRI image in the k-space, such that the dimension of the MRI image after sampling is M d .

[0072] In Equation (2), is the reconstruction module, representing the network module for reconstructing the MRI image in the pixel domain, and θ is its parameter. Specifically, for the input k-space data of the to-be-determined modality and the k-space data of the reference modality R((·,·); θ) is composed of B sub-modules cascaded, θ = {θ 1 , θ 2 , …, θ B}, and θ b is the parameter of , The sub-module is defined as follows:

[0073]

[0074] Among them, x 0 = F H k, Cat(·,·) represents concatenation of the input image in the channel direction, is the activation function. Commonly used activation functions include ReLU and LReLU. Here, the pre-activation adopted is LReLU, which is defined as follows:

[0075]

[0076] Among them, α can take 0.1. Figure 2 (b) and (c) give the network structures of R and C b , Figure 7 comparing the differences between ReLU and LReLU.

[0077] Furthermore, in Equation (2), denotes the data consistency operator. Let The element in is calculated in the following way:

[0078]

[0079] Among them, Ω represents the index subset of the elements in y used to construct y u . The data consistency operator essentially replaces the information in the corresponding frequency band of the reconstructed MRI image with the frequency information (mainly low-frequency information) of the input MRI image. Figure 8 Figure 1 gives a graphical explanation of the data consistency operator, where subfigure (a) is the input undersampled MRI image, subfigure (b) is the reconstructed MRI image, and subfigure (c) is the MRI image after performing data consistency calculation on subfigure (b) using the information in subfigure (a). Since the data consistency operator constrains the low-frequency information in the reconstructed image, from the pixel domain perspective, its role in the network learning process is similar to residual learning, that is: forcing the network to learn the high-frequency information or detailed information in the MRI image before performing data consistency processing. Figure 9 The first two columns of Figure 2 compare the changes in the MRI images obtained by three subnets of the deep sub-pixel recurrent neural network before and after using data consistency processing. It can be seen that: the MRI image obtained before using data consistency processing (the first column) is mainly its details, while the MRI image obtained after using data consistency processing (the second column) is closer to the real image (the third column).

[0080] Furthermore, the training of the deep sub-pixel recurrent neural network f SPRNN (·; Θ) in step 2 uses a multi-scale deep supervision method, and the steps are as follows:

[0081] Step 2.1 Construct a multi-scale MRI image set for training the deep sub-pixel recurrent neural network f SPRNN (·; Θ)

[0082]

[0083] Here, Γ = {1, 2, …, L} is the index set of training samples, L is the number of training samples, u = u 0 > u 1 > u 2 > … > u D are different undersampling multiples of the MRI image. is the data of the to-be-determined modality. is the reference modality data. is the γ-th full-sampled MRI image of the to-be-determined modality. is the MRI image of the to-be-determined modality with a dimension of . It is obtained through the following method:

[0084]

[0085] Among them, represents the mask matrix for undersampling the MRI image in k-space by u d times; Since each training sample in is composed of subsamples of different dimensions, the present invention refers to as a multi-scale training set; Usually, the number of training samples L ≥ 200; Figure 10 gives an example of 5 training samples for constructing .

[0086] Step 2.2 Let the iteration number t = 0, and initialize the parameter set Θ of f SPRNN (·; Θ) as Θ (0) :

[0087]

[0088] Common initialization methods for neural networks include: random initialization, Xavier initialization, and He initialization; For subnets containing ReLU-type activation functions, He initialization is recommended; For subnets containing tanh-type activation functions, Xavier initialization is recommended;

[0089] Step 2.3 Let t = t + 1, and minimize the following loss function by gradient descent:

[0090]

[0091] Among them, l(·,·) represents the distance metric operation between two images, and can use l 1 norm, l 2 norm or Charbonnier penalty function. Among them, the Charbonnier penalty function is defined as: For noisy MRI images, it is recommended to use l 1 norm or Charbonnier penalty function;

[0092] Step 2.4 Iteratively execute the above Step 2.3 until t = T. Here, T is the preset maximum number of iterations; T is usually taken as 10 4 .

[0093] Step 2.5 Select Θ = Θ (T) as the optimal parameter set of f SPRNN (·; Θ).

[0094] Experiments were conducted to verify the beneficial effects of the deep sub-pixel recursive neural network proposed in the present invention. Ten subjects were scanned using a Philips Ingenia 3T scanner to obtain MRI images in the T2WI modality, and each MR slice was cropped to a size of 336×336×261. The U-Net network was trained using the Adam optimization algorithm, with the momentum set to 0.9 and the number of iterations set to 10 4 , the initial learning rate was set to 0.001, and it decreased by a factor of 10 every 5500 iterations. To quantitatively evaluate the reconstruction performance of the network, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to evaluate the quality of the reconstructed MRI images.

[0095] To quantitatively evaluate the reconstruction performance of the network, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to evaluate the quality of the reconstructed MRI images. The deep sub-pixel recursive neural network (SPRNN) proposed in the present invention was compared with the filling method (Zero Filling, ZF) and four state-of-the-art methods, DeepCascade, RefineGAN, UNet, and DenseUNet. The MRI images of 8 subjects were used as training samples, and the MRI images of the remaining 2 subjects were used as test samples. The cross-validation method was used to test the stability of the compared methods, and the MRI images of 4-fold, 8-fold, and 12-fold undersampled subjects numbered 2 and 3, 0 and 1, 4 and 5, 6 and 7, 8 and 9 were reconstructed respectively. The reconstructed results were compared with the fully sampled images, and the PSNR / SSIM values are as Figure 11 shown, and it can be seen that the method proposed in the present invention outperforms other methods.

[0096] Figure 12 Furthermore, the auxiliary reconstruction process of the SPRNN proposed in the present invention using multi-scale reference modality data for 8-fold undersampled pending modality is demonstrated, where represents the result of reconstructing the input undersampled pending modality image to scale u using the sub-pixel upsampling subnet d afterwards, and represent the reference modality image and the real image of the pending modality at scale u d respectively, and is the image of the pending modality after d recursive reconstructions by the SPRNN.

[0097] Figure 13Visual comparison of the reconstruction effects of the proposed SPRNN of the present invention and its comparison methods (ZF, UNet, DenseUNet, RefineGAN, D5C5, D5C15) is shown. Here, the reference modality is T1WI, and the modality to be determined is 12-fold undersampled T2WI. Three regions of interest are selected from each reconstructed image and placed in the subgraphs corresponding to the respective methods below in the order of "from left to right, from top to bottom" after magnification. It can be seen that the proposed SPRNN method of the present invention can better reconstruct the detailed information of highly undersampled MRI images.

Claims

1. A multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes, characterized in that, the method comprises the following steps: Step 1 Input undersampled reference modal image and fully sampled reference modal image represent in k-space undersampled MRI image of the to-be-determined modal, represent the k-space where the MRI image is located, represent the dimension of y u , where N is the dimension of the fully sampled MRI image; Step 2 is performed by a deep sub-pixel recursive neural network to obtain the reconstructed undetermined modal image wherein represents the pixel space where the MRI image is located; The said step 2, the deep sub-pixel recurrent neural network f SPRNN ((·,·); Θ) is defined as follows: where, Θ = {Θ 1 , …, Θ D} is the parameter set of the deep sub-pixel recursive neural network f SPRNN , H represents the Hermitian transpose operation, D is the number of sub-networks of f SPRNN , and f d ((·, ·); Θ d ) is the d-th sub-network of f SPRNN , which is defined as follows: Here, is a sub-pixel upsampling operator, is a reconstruction module, represents central downsampling of a fully undersampled MRI image in k-space, represents a data consistency operator, d ∈ {1, 2, …, D}, 2. The multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes according to claim 1, characterized in that, In the said step 1, the undersampled MRI image has the following relationship with the fully sampled MRI image: Among them, represents the fully sampled image in k-space, represents the fully sampled image in pixel space, is the Fourier transform matrix for transforming the MRI image from pixel space to k-space; represents the mask matrix for undersampling y by u times.

3. The multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes according to claim 1, characterized in that, (1) In the formula represents mapping an MRI image with times undersampled in k-space to times undersampled MRI image, is its parameter. For the input MRI image is defined as follows: Among them, PZF(·) is a partial zero-padding operator, and SPC(·) represents a sub-pixel convolution operator.

4. The multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes according to claim 1, characterized in that, (1) In the formula makes the dimension of the MRI image after sampling be M d .

5. The multi-modal MRI assisted reconstruction method as claimed in claim 1, which is based on SPRNN and multi-scale reference represents a multi-modal MRI assisted reconstruction method for the central undersampling modality of fully undersampled MRI images in k-space characterized in that, (1) In the formula represents a network module for reconstructing and calculating the MRI image in the pixel domain, θ is its parameter, and for the input k-space data of the to-be-determined modality and the k-space data of the reference modality R((·,·); θ) is composed of B sub-modules cascaded, θ = {θ 1 , θ 2 , …, θ B}, b is 's parameter, The sub-module is defined as follows: Among them, x 0 = F H k, Cat(·, ·) represents the concatenation of the input images in the channel direction, is the activation function.

6. The multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes according to claim 3, characterized in that, In formula (1), Let The elements in are calculated as follows: Among them, Ω represents the index subset of the elements in y used to construct ​ 7. The multi-modal MRI assisted reconstruction method based on SPRNN and multi-scale reference modes according to claim 1 or 2, characterized in that, The deep sub-pixel recurrent neural network f SPRNN (·; Θ) in step 2 is trained using a multi-scale deep supervision method, and the steps are as follows: Step 2.1 Construct a multi-scale MRI image set for training the deep sub-pixel recurrent neural network f SPRNN (·; Θ) Here, Γ = {1, 2, …, L} is the index set of training samples, and L is the number of training samples. are different undersampling multiples of the MRI image. is the reference modal data. is the γ-th full-sampled MRI image of the to-be-determined modality. is of dimension of the MRI image of the to-be-determined modality and is obtained in the following way: ​ Among them, represents a mask matrix for times undersampled MRI images in k-space; Since each training sample in is composed of subsamples of different dimensions, is called a multi-scale training set; Step 2.2 Let the number of iterations \(t = 0\), and randomly initialize the parameter set \(\Theta\) of \(f \) SPRNN (·;\(\Theta\)) as \(\Theta( \) 0 ): Step 2.3 Let t = t + 1, and minimize the following loss function by the gradient descent method: Among them, represents the distance metric operation between two images; Step 2.4 Iteratively execute the above Step 2.3 until t = T, where T is a preset maximum number of iterations; Step 2.5 Select Θ = Θ (T) as the optimal parameter set of f SPRNN (·; Θ).

Images