Magnetic Resonance Image Reconstruction and Super-Resolution Method Based on Multi-Stage and Multi-Module Neural Network

Abstract

A method for magnetic resonance image reconstruction and super-resolution based on a multi-stage and multi-module neural network, comprising: S1. Reading an undersampling mask and an original high-resolution image; S2. Performing normalization processing on the read images; S3. Generating 4 types of patches from the input images according to different cutting methods; S4: Using an encoder-decoder to extract features within different patches; S5. Concatenating each group of feature maps corresponding to each branch, and then adding the feature maps of the two branches; S6. Using a self-attention module to extract enhanced features of the complete image; S7. Using an encoder-decoder architecture and a self-attention module to extract features in the second stage, where the encoder-decoder architecture and the self-attention module in the second stage are consistent with those in the first stage; S8. Using a deep residual convolution to extract spatial information in the third stage; S9. Performing upsampling and convolution operations on the output result of the second stage to obtain the target image. The present invention solves the problems of super-resolution and reconstruction of low-resolution undersampled MRI.

Description

Technical Field

[0001] The present invention relates to a method for magnetic resonance image reconstruction and super-resolution. Background Art

[0002] Magnetic resonance images (MRI) are special images obtained by nuclear magnetic resonance imagers and are widely used due to their high resolution and low radiation. By generating images in different modalities, MRI can provide functional and anatomical information for clinical diagnosis, including neurological, musculoskeletal, and tumor diseases. Due to its excellent performance, MRI plays an extremely important role in current medical diagnosis. However, due to complex physical principles, the data acquisition process takes a long time, causing great pain to patients. At the same time, the respiratory movement of patients, etc., will cause the generation of image artifacts, inhibiting the acquisition of high-resolution MRI images. Therefore, improving the data acquisition speed of MRI has been an important research field in recent decades.

[0003] Traditional MRI image restoration techniques include compressive sensing, low-rank matrix, dictionary learning, parallel imaging, etc. These techniques all use the prior knowledge of images to solve image artifacts generated due to violation of the Shannon-Nyquist sampling theorem. However, at very high acceleration rates, parallel imaging will amplify the noise of the picture and generate low-quality pictures. Compressive sensing will also result in the residue of artifacts and consume a long reconstruction time.

[0004] With the development of computer vision technology, image reconstruction and super-resolution are important means to solve the above problems. In image reconstruction, by using a sampling mask to obtain less MRI data, the scanning time of MRI can be reduced. Image super-resolution technology can restore low-resolution MRI to high-resolution MRI to display more image details and provide more reliable guarantees for clinical medical diagnosis. Summary of the Invention

[0005] The present invention overcomes the above-mentioned disadvantages of the prior art and proposes a method for magnetic resonance image reconstruction and super-resolution based on a multi-stage and multi-module neural network.

[0006] A method for magnetic resonance image reconstruction and super-resolution based on a multi-stage and multi-module neural network of the present invention includes the following steps:

[0007] S1. Read the undersampling mask I mask , the original high-resolution image I HR (height×width). Wherein, height represents the vertical resolution of the image, and width represents the horizontal resolution of the image.

[0008] S2. For the read picture I HRPerform normalization to normalize the pixel values of the image to 0-1. Use image truncation and undersampling mask I mask Construct grayscale images that need to be reconstructed and super-resolved

[0009] S3. Take the input image I LR Generate 4 types of patches according to different cutting methods. Among them, the corresponding ones are grouped in pairs to construct two sets of input data, namely (I LR，top , I LR，bottom ) and (I LR，left , I LR，right ). Use the multi-channel feature enhancement module to obtain multi-channel feature maps. The formula is as follows:

[0010]

[0011]

[0012] Among them, MFE 1 (), MFE 2 () represent the multi-channel feature enhancement modules of the upper and lower branches respectively.

[0013] S4: Use the encoder-decoder to extract features within different patches. Among them, channel attention modules are used in both the encoder-decoder module and the skip connection to enhance the expression of effective information and suppress the expression of invalid information. The formula is as follows:

[0014]

[0015]

[0016]

[0017] Among them represents the outputs of the 1st, 2nd, and 3rd layer encoders, where i ∈ {top, bottom, left, right}. enc CA represents the output of the skip connection, and dec i represents the output of the last decoder. Encoder j () and Decoder j () represent the encoder module and the decoder module respectively, where j ∈ {1, 2}, 1 represents the upper branch, and 2 represents the lower branch, as Figure 2 shown.

[0018] S5. Concatenate the sets of feature maps corresponding to each branch, and then add the feature maps of the two branches. The formula is as follows:

[0019] F = cat(dectop , dec bottom ) + cat(dec left , dec right ) (6)

[0020] Among them, cat() represents the concatenation operation, and F represents the mixed enhanced features extracted from different patches.

[0021] S6. Use the self-attention module to extract the enhanced features of the complete image. The formula is as follows:

[0022] F 0 = SAB(F) (7)

[0023] Among them, SAB() represents the self-attention module. F 0 represents the first-stage features obtained by performing feature extraction on F, as Figure 3 shown.

[0024] S7. Use the encoder-decoder architecture and the self-attention module to extract the features of the second stage, where the encoder-decoder architecture and the self-attention module in the second stage are the same as those in the first stage. The formula is as follows:

[0025] F i = UNet i (F i-1 ), i ∈ 1, 2, 3...N (8)

[0026] F i = SAB i (F i ), i ∈ 1, 2, 3...N (9)

[0027] Among them, UNet i () represents the abbreviation of the encoder-decoder architecture, SAB i () represents the self-attention module, and i represents the i-th module in the second stage.

[0028] S8. Use the deep residual convolution to extract the spatial information in the third stage. The formula is as follows:

[0029]

[0030] Among them, S() represents the deep residual convolution module, which is used to extract the features rich in spatial accuracy information. F N represents the output result of the last module in the second stage, G i represents the output result of the i-th deep residual convolution module, as Figure 4 shown.

[0031] S9. For the output result (G NThe target image I can be obtained by adopting the sum and convolution operations. SR . The formula is as follows:

[0032] I SR = Conv(Upsample(G N )) (11)

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) It solves the super-resolution and reconstruction problems of low-resolution undersampled MRI. By modifying the upsampling module, the network can separately solve the super-resolution or reconstruction problems, improving the generalization ability of the model.

[0035] (2) The image peak signal-to-noise ratio and structural similarity index are improved. The multi-stage multi-module network proposed by the present invention uses the different functions of different modules to decompose the complex super-resolution reconstruction problem into smaller problems that are easier to solve, reducing the difficulty of network learning.

[0036] (3) The model has fewer parameters and a fast inference speed. Compared with the previous methods, the network proposed by the present invention has better model parameters and a faster inference speed on the premise of achieving better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is the overall flow chart of the method of the present invention.

[0038] Figure 2 is the structural diagram of the encoder-decoder network in the method of the present invention.

[0039] Figure 3 is the structural diagram of the self-attention module in the method of the present invention.

[0040] Figure 4 is the structural diagram of the depth residual convolution module in the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0041] In order to more easily understand the process of the present invention, the present invention will be described in detail with reference to examples.

[0042] A method for magnetic resonance image reconstruction and super-resolution based on a multi-stage multi-module neural network includes:

[0043] Classify the collected MRI data set (the size of a single data is 320×256×256) into slices, and the size of the sliced data is 256×256, denoted as I HR . Use an undersampling mask generator to generate an undersampling mask with corresponding acceleration (the acceleration includes ×4, ×6, ×8), denoted as I mask . I HRAfter normalization, truncation and undersampling operations are performed to obtain undersampled low-resolution MRI, denoted as I LR (Taking scale = 2 as an example, I LR has a size of 128×128). Input I LR into the network of the present invention, and the target image I SR can be obtained, with a size of 256×256. The specific implementation steps are as follows:

[0044] S1. Read the undersampling mask I mask , the original high-resolution image I HR (height×width). Here, height represents the vertical resolution of the image, and width represents the horizontal resolution of the image.

[0045] S2. Perform normalization on the read image I HR , and normalize the pixel values of the image to 0-1. Use the image truncation and undersampling mask I mask to construct the grayscale image that needs to be reconstructed and super-resolved

[0046] S3. Generate 4 types of patches from the input image I LR in different cutting ways. Among them, corresponding in pairs, two groups of input data are constructed, namely (I LR，top , I LR，bottom ) and (I LR，left , I LR，right ). Use the multi-channel feature enhancement module to obtain the multi-channel feature map, and the formula is as follows:

[0047]

[0048]

[0049] Among them, MFE 1 (), MFE 2 () respectively represent the multi-channel feature enhancement modules of the upper and lower branches.

[0050] S4: Use the encoder-decoder to extract the features in different patches. Among them, the channel attention module is used in both the encoder-decoder module and the skip connection to enhance the expression of effective information and suppress the expression of invalid information, and the formula is as follows:

[0051]

[0052]

[0053]

[0054] Among them represents the outputs of the encoders of the first, second, and third layers, where i ∈ {top, bottom, left, right}. enc CA represents the output of the skip connection, dec i represents the output of the last decoder. Encoder j () and Decoder j () represent the encoder module and the decoder module respectively, where j ∈ {1, 2}, 1 represents the upper branch, 2 represents the lower branch, as Figure 2 shown.

[0055] S5. Concatenate each group of feature maps corresponding to each branch, and then add the feature maps of the two branches. The formula is as follows:

[0056] F = cat(dec top , dec bottom ) + cat(dec left , dec right ) (6)

[0057] where cat() represents the concatenation operation, and F represents the mixed enhanced features extracted from different patches.

[0058] S6. Use the self-attention module to extract the enhanced features of the complete image. The formula is as follows:

[0059] F 0 = SAB(F) (7)

[0060] where SAB() represents the self-attention module. F 0 represents the first-stage features obtained by extracting features from F, as Figure 3 shown.

[0061] S7. Use the encoder-decoder architecture and the self-attention module to extract the second-stage features, where the second-stage encoder-decoder architecture and the self-attention module are the same as those in the first stage. The formula is as follows:

[0062] F i = UNet i (F i-1 ), i ∈ 1, 2, 3...N (8)

[0063] F i = SAB i (F i ), i ∈ 1, 2, 3...N (9)

[0064] where UNet i () represents the abbreviation of the encoder-decoder architecture, SAB i() represents the self-attention module, and i represents the i-th module in the second stage.

[0065] S8. Use depth residual convolution to extract spatial information in the third stage. The formula is as follows:

[0066]

[0067] Where S() represents the depth residual convolution module, which is used to extract features rich in spatial accuracy information. F N represents the output result of the last module in the second stage, and G i represents the output result of the i-th depth residual convolution module, as Figure 4 shown.

[0068] S9. Perform upsampling and convolution operations on the output result (G N ) of the second stage to obtain the target image I SR . The formula is as follows:

[0069] I SR = Conv(Upsample(G N )) (11)

[0070] The content described in the embodiments of this specification is only an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms stated in the examples. The protection scope of the present invention also extends to equivalent technical means that those skilled in the art can think of based on the inventive concept of the present invention.

Claims

1. A method for magnetic resonance image reconstruction and super-resolution based on a multi-stage and multi-module neural network, comprising the following steps: S1. Read the undersampling mask I mask , the original high-resolution image I HR (height×width); where height represents the vertical resolution of the image and width represents the horizontal resolution of the image; S2. Normalize the read image I HR Perform normalization processing to normalize the pixel values of the image to 0-1; use the image truncation and undersampling mask I mas k to construct the grayscale image to be reconstructed and super-resolved where scale is the image scaling factor; S3. Generate the input image I LR Generate 4 types of patches according to different cutting methods; among them, pair them up to construct two sets of input data, namely (I LR,top , I LR,bottom ) and (I LR,left , I LR,right ); Obtain the multi-channel feature map using the multi-channel feature enhancement module. The formula is as follows: Among them, MFE 1 (), MFE 2 () respectively represent the multi-channel feature enhancement modules of the upper and lower branches; S4: Use an encoder-decoder to extract features within different patches; among them, channel attention modules are used in both the encoder-decoder module and the skip connection to enhance the expression of effective information and suppress the expression of invalid information. The formula is as follows: Among them represents the outputs of the encoders of the first, second, and third layers, where i ∈ {top, bottom, left, right}; enc CA represents the output of the skip connection, dec i represents the output of the last decoder; Encoder j () and Decoder j () represent the encoder module and the decoder module respectively, where j ∈ {1, 2}, 1 represents the upper branch, and 2 represents the lower branch; S5. Concatenate each group of feature maps corresponding to each branch, and then add the feature maps of the two branches. The formula is as follows: F = cat(dec top , dec bottom ) + cat(dec left , dec right ) (6) where cat() represents the concatenation operation and F represents the mixed enhanced features extracted from different patches; S6. Use a self-attention module to extract the enhanced features of the complete image; the formula is as follows: F 0 = SAB(F) (7) where SAB() represents the self-attention module; F 0 represents the first-stage features obtained by feature extraction of F; S7. Use an encoder-decoder architecture and a self-attention module to extract features in the second stage, where the encoder-decoder architecture and the self-attention module in the second stage are consistent with those in the first stage; the formula is as follows: F i = UNet i (F i-1 ), i ∈ 1, 2, 3…N (8) F i = SAB i (F i ), i ∈ 1, 2, 3…N (9) Among them, UNet i () represents the abbreviation of the encoder-decoder architecture, SAB i () represents the self-attention module, and i represents the i-th module in the second stage; S8. Use a deep residual convolution to extract spatial information in the third stage; the formula is as follows: where S() represents the depth residual convolution module, which is used to extract features rich in spatial accuracy information; F N represents the output result of the last module in the second stage, and G i represents the output result of the i-th depth residual convolution module; S9. Apply upsampling and convolution operations to the output result (G N ) of the second stage to obtain the target image I SR ; The formula is as follows: I SR = Conv(Upsample(G N )) (11).

Images