Multi-mode 5.0 T 23Na magnetic resonance imaging artificial intelligence reconstruction method

Abstract

The invention discloses a multi-modal 5.0 T 23Na magnetic resonance imaging artificial intelligence reconstruction method, and the method comprises the steps: constructing a multi-modal reconstruction network which comprises a multi-scale feature extraction network, a cross attention reconstruction network module and a fusion network. The multi-scale feature extraction network extracts multi-level features of undersampled target modal and auxiliary modal magnetic resonance images. The cross attention reconstruction network learns information between different modes. And the fusion network fuses the multi-modal multi-scale information to obtain a final reconstructed image. In addition, a structural similarity loss function is utilized, so that the multi-modal reconstruction network fully learns uniform contrast information of 5.0 T magnetic resonance image data. According to the method, the information complementarity between the multi-modal MRI data is utilized, the detail features of the reconstructed image are enhanced, the characteristic of uniform contrast of 5.0 T data is fully utilized, and the reconstruction quality of the target modal can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal 5.0T 23 Na magnetic resonance imaging artificial intelligence reconstruction method, suitable for magnetic resonance image reconstruction. Background Art

[0002] Sodium magnetic resonance imaging 23 Na MRI provides a non-invasive method to measure tissue activity and can provide information about the physiological and biochemical processes of cells. 1 H signal ratio 23 The MRI signal of Na is about 12,000 times higher. 23 The low signal-to-noise ratio caused by the low Na concentration and the bi-exponential decay of the signal limits the image quality and requires longer imaging time to compensate.

[0003] K-space undersampling is one of the common methods to reduce MRI acquisition time. Compressed sensing uses the sparsity of data in the sparse domain and combines iterative algorithms to recover undersampled k-space data, thereby removing artifacts caused by undersampling. However, compressed sensing algorithms have problems such as long reconstruction time and insufficient reconstruction accuracy.

[0004] Deep learning methods have shown great potential in accelerating MRI imaging in recent years. Deep learning methods have successfully improved undersampling by learning abstract features of data. 1 H magnetic resonance image quality. Huang et al. used a self-supervised method to train the model to expand the network DURED-Net and successfully reconstructed the details of undersampled images under high acceleration. Adlung et al. used U-Net and ResNet structures to reconstruct undersampled images. 23 Na brain magnetic resonance images, successfully improving the quality of undersampled images. 23 Na data has a low signal-to-noise ratio. 23 Na data training may not achieve the best reconstruction results. Multimodal reconstruction strategy can use auxiliary modality information to help target modality reconstruction, thereby improving the target modality image quality.

[0005] In pursuit of higher signal-to-noise ratio, higher resolution, and higher tissue contrast, high-field magnetic resonance imaging of the human body has been a major research hotspot in recent years. Ultra-high field MRI (such as 7.0T) can be imaged in many scenes that were previously impossible to image, and can provide more image details. However, ultra-high field MRI faces many challenges. For example, the main challenges of 7.0T MRI are the inhomogeneity of the emitted B1 field and the deposition of radio frequency power in the tissue. The recently developed 5.0T MRI scanner not only has good spatial resolution and signal-to-noise ratio, but also has a more uniform B1 field, and has good application prospects.

[0006] In summary, the prior art has the following problems:

[0007] 1. Currently used for 23 The deep learning reconstruction technology of Na MRI imaging is relatively simple. Using simple network structures such as Unet or ResNet may not fully exploit the redundant information of under-sampled data, resulting in insufficient acceleration;

[0008] 2. 23 Na MR image data has a low signal-to-noise ratio. 23 Na data may affect deep learning network training and degrade reconstruction quality.

[0009] 3. Ultra-high field MRI equipment is expensive, the technology is more complex, and B1 field inhomogeneity may lead to image quality degradation. Summary of the invention

[0010] In view of the above problems existing in the prior art, the present invention proposes a multi-modal 5.0T 23 Artificial intelligence reconstruction method for Na magnetic resonance imaging.

[0011] The above-mentioned purpose of the present invention is achieved by the following method:

[0012] Multi-mode 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging comprises the following steps:

[0013] Step 1, obtaining a training set, the training set comprising a plurality of sample groups, each sample group comprising a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D under-sampled target modality magnetic resonance image and a 2D under-sampled auxiliary modality magnetic resonance image of a subject;

[0014] Step 2: construct a multimodal reconstruction network, which includes a multi-scale feature extraction network, a cross-attention reconstruction network module and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks.

[0015] The 2D under-sampled target modality magnetic resonance images and 2D under-sampled auxiliary modality magnetic resonance images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then are input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain a target modality preliminary reconstruction feature block and an auxiliary modality preliminary reconstruction feature block. The target modality preliminary reconstruction feature block and the auxiliary modality preliminary reconstruction feature block corresponding to each cross-attention reconstruction network are respectively reorganized to obtain a target modality preliminary reconstructed image and an auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascaded image. All auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascaded image. The target modality cascaded image and the auxiliary modality cascaded image are input into a fusion network to obtain a target modality MRI reconstructed image and an auxiliary modality MRI reconstructed image.

[0016] Step 3: Set the total loss function;

[0017] Step 4: According to the total loss function, the multimodal reconstruction network is trained using the sample group in the training set;

[0018] Step 5: Input the 2D under-sampled target modality MRI image to be reconstructed and the corresponding 2D under-sampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

[0019] As described above, the multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network, a 2D under-sampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and a 2D under-sampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network, and the target modality branch feature extraction network and the auxiliary modality branch feature extraction network both include a plurality of cascaded encoders, and each encoder includes a convolution layer, an activation layer, and a maximum pooling layer in sequence;

[0020] The scales of the target modality feature images output by encoders at various levels in the target modality branch feature extraction network are different, and the scales of the auxiliary modality feature images output by encoders at various levels in the auxiliary modality branch feature extraction network are different; the target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks;

[0021] The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network.

[0022] Each cross-attention reconstruction network as described above includes a target modality attention branch network and an auxiliary modality attention branch network; after the target modality feature image and the auxiliary modality feature image of the same scale are equally divided into a plurality of target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input into the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input into the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network;

[0023] Both the target modality attention branch network and the auxiliary modality attention branch network include multiple cascaded transformer blocks;

[0024] In each criss-cross attention reconstruction network:

[0025] The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network;

[0026] The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the auxiliary modality attention branch network includes the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstructed feature block of the current cross-attention reconstruction network.

[0027] In each cross-attention reconstruction network as described above,

[0028] The input of the first-level transformer block of the target modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks in the target modality attention branch network except the first-level transformer block also includes the corresponding learnable vector;

[0029] The input of the first-level transformer block of the auxiliary modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block also includes the corresponding learnable vector;

[0030] The learnable vectors corresponding to the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network are the same; and the learnable vector of the first-level transformer block is the output vector obtained after the target modality feature block and the auxiliary modality feature block input into the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network; the learnable vectors corresponding to other transformer blocks except the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output vector of the previous-level transformer block in the auxiliary modality attention branch network after being input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network.

[0031] In the above-mentioned modal mapping module, the target modal feature block passes through the output of the first linear mapping layer of the modal mapping module, and the auxiliary modal feature block passes through the output of the second linear mapping layer of the modal mapping module, and then undergoes channel splicing, and then passes through the third linear mapping layer of the modal mapping module and the normalization layer of the modal mapping module in sequence, to obtain an output vector of the modal mapping module;

[0032] In each learnable vector update module, the output feature block of the previous transformer block in the target modality attention branch network and the output feature block of the previous transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

[0033] Each transformer block as described above includes a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer, and a fully connected layer in sequence;

[0034] The feature block input by the transformer block is input into the position encoding layer of the current transformer, and the learnable vector corresponding to the current transformer block is input into the attention mechanism layer of the current transformer block;

[0035] The feature block input by the transformer block is sequentially encoded by the position encoding layer, linear mapped by the linear mapping layer, and normalized by the first normalization layer in the transformer block to obtain the vector Q. In the attention mechanism layer, the vector Q can be learned and cascaded with the vector Q as the vector K and the vector V. The vectors Q, K, and V are calculated as follows to obtain the intermediate quantity z:

[0036]

[0037] Where Q represents vector Q, K represents vector K, V represents vector V, d k is the dimension of vector K, softmax is the normalized exponential function, and T represents transpose;

[0038] The intermediate quantity z is added to the vector Q through the residual connection, and then passes through the linear mapping layer of the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

[0039] As described above, the fusion network includes an input layer, multiple convolutional layers and an output layer; the target modality cascade image and the auxiliary modality cascade image are sequentially input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image, respectively.

[0040] The above fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork. The target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, multiple convolutional layers and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain the target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain the auxiliary modality MRI reconstructed image.

[0041] As mentioned above, the total loss function Loss is:

[0042] Loss = L(x ref ,f θ (y ref ))+L(x tar ,f θ (y tar ))+L SSIM (x ref ,f θ(y ref ))+L SSIM (x tar ,f θ (y tar ))

[0044] Where L is the error function, L SSIM is the structural similarity loss, x ref is a 2D fully sampled auxiliary modality MRI image, x tar is the 2D fully sampled target modality MRI image, y ref and tar The distribution is a 2D undersampled auxiliary modality MRI image and a 2D undersampled target modality MRI image, f θ is a multimodal reconstruction network with parameters θ.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements steps 2-5 of the reconstruction method according to any one of claims 1 to 9 when executing the computer program.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. The present invention builds a multi-scale feature extraction network and uses multiple encoder structures to extract image features to obtain multi-scale features from coarse to fine, thereby improving the reconstruction quality.

[0048] 2. The present invention uses the attention mechanism to learn multi-modal and multi-scale information, fully explores the information association and information redundancy between the auxiliary modality and the target modality, and uses the auxiliary modality information to help reconstruct the target modality and improve the reconstruction performance.

[0049] 3. Based on the good B1 field uniformity of 5.0T MRI, the image contrast is more uniform. Therefore, the present invention uses a structural similarity loss function to ensure that the brightness and contrast of the reconstruction result are consistent with the full sampling data, making full use of the characteristics of 5.0T MRI data, improving the details of the reconstruction result, and thus further improving the reconstruction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the present invention;

[0051] Figure 2 Schematic diagram of the overall structure of the multimodal reconstruction network;

[0052] Figure 3 Schematic diagram of the overall structure of the cross-attention reconstruction network;

[0053] Figure 4Schematic diagram of the attention mechanism layer structure, where (a) is a schematic diagram of the attention mechanism layer structure in the target modality attention branch network, and (b) is a schematic diagram of the attention mechanism layer structure in the auxiliary modality attention branch network; represents multiplication, Represents residual connection addition; Q represents vector Q, K represents vector K, and V represents vector V;

[0054] Figure 5 It is a schematic diagram of the fusion network structure;

[0055] Figure 6 It is a schematic diagram of the structure of the modal mapping module;

[0056] Figure 7 This is a schematic diagram of the structure of the learnable vector update module;

[0057] in, Indicates channel splicing;

[0058] Figure 8 Rebuild for the test set 23 Comparison of Na magnetic resonance imaging results, where (a) is 2D full sampling 23 Na magnetic resonance image, (b) is zero-filled 23 Na magnetic resonance image, (c) is the final reconstruction 23 Na magnetic resonance images. Specific implementation plan

[0059] In order to facilitate those skilled in the art to understand and implement the present invention, the following examples and Figure 1-Figure 3 The present invention is described in further detail. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and do not limit the present invention.

[0060] Multi-mode 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging specifically comprises the following steps:

[0061] Step 1: Obtain a training set and a test set, wherein the training set and the test set each include multiple sample groups, each sample group includes a 2D fully sampled target modality MRI image, a 2D fully sampled auxiliary modality MRI image, a 2D undersampled target modality MRI image, and a 2D undersampled auxiliary modality MRI image of a subject. In this example, the target modality is 23 Na, auxiliary modality is T1-weighted 1 H. The specific method is as follows:

[0062] Step 1.1, scan multiple subjects to obtain corresponding 3D full-sampling target modality MRI k-space data and 3D full-sampling auxiliary modality MRI k-space data.

[0063] In this embodiment, multiple subjects are scanned to obtain corresponding 3D full sampling 23 Na MRI k-space data and 3D fully sampled T1-weighted 1 H MRI k-space data.

[0064] Step 1.2, extract 3D fully sampled target modality MRI k-space data and 3D fully sampled auxiliary modality MRI k-space data layer by layer, and obtain multiple 2D fully sampled target modality MRI k-space data and 2D fully sampled auxiliary modality MRI k-space data respectively; undersample the 2D fully sampled target modality MRI k-space data and 2D fully sampled auxiliary modality MRI k-space data according to the undersampled sampling matrix, and obtain corresponding 2D undersampled target modality MRI k-space data and 2D undersampled auxiliary modality MRI k-space data respectively.

[0065] In this embodiment, 3D full sampling is extracted layer by layer. 23 Na MRI k-space data, obtain multiple 2D full samples 23 NaMRI k-space data, 2D full sampling according to the undersampling sampling matrix 23 Na MRI k-space data is undersampled to obtain the corresponding 2D undersampling 23 Na MRI k-space data. For 3D fully sampled T1-weighted 1 The H MRI k-space data is processed in the same way, that is, 3D full-sampled T1-weighted images are first extracted layer by layer. 1 H MRI k-space data, obtain multiple 2D full sampling 1 H MRI k-space data, 2D full sampling according to the under-sampling sampling matrix 1 H MRI k-space data is undersampled to obtain 2D undersampling 1 H MRIk space data.

[0066] Step 1.3, perform two-dimensional inverse Fourier transform on the 2D fully sampled target modality MRI k-space data, the 2D fully sampled auxiliary modality MRI k-space data, the 2D under-sampled target modality MRI k-space data, and the 2D under-sampled auxiliary modality MRI k-space data, respectively, to obtain a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D under-sampled target modality magnetic resonance image, and a 2D under-sampled auxiliary modality magnetic resonance image, respectively.

[0067] In this embodiment, 2D full sampling is obtained by two-dimensional inverse Fourier transform processing. 23 Na MRI, 2D full-sample T1-weighted 1 H MRI, 2D undersampling 23Na MRI and 2D undersampled T1-weighted 1 H magnetic resonance image.

[0068] Step 1.4: Take the 2D fully sampled target modality MRI images, 2D fully sampled auxiliary modality MRI images, 2D under-sampled target modality MRI images and 2D under-sampled auxiliary modality MRI images of the same subject as a sample group, and divide all sample groups into training sets and test sets. Both the training set and the test set include multiple sample groups.

[0069] In this embodiment, the 2D undersampling of a subject 23 Na magnetic resonance imaging, 2D full sampling 23 Na MRI, 2D undersampled T1-weighted 1 H MRI and 2D full-sample T1-weighted 1 H magnetic resonance images are used as sample groups, and all sample groups are divided into training sets and test sets. Both the training set and the test set include multiple sample groups.

[0070] In this embodiment, the undersampling mode selects variable density undersampling, and the undersampling acceleration factor is 4. 23 Na MRI and T2-weighted 1 The contrast of H magnetic resonance images is similar. This embodiment uses the T2 weighted image of the IXI public dataset. 1 H MRI images were used to simulate the target modality 5.0T 23 Na MRI images were used for experiments, using the IXI public dataset T1-weighted 1 H magnetic resonance images were used as auxiliary modalities, and 1000 sample groups were selected as training sets and 300 sample groups were selected as test sets. 23 Na MRI images and 1 H magnetic resonance image.

[0071] Step 2: Construct a multimodal reconstruction network. The multimodal reconstruction network includes a multi-scale feature extraction network, a cross-attention reconstruction network module and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks.

[0072] The 2D under-sampled target modality magnetic resonance images and 2D under-sampled auxiliary modality magnetic resonance images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then are input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain a target modality preliminary reconstruction feature block and an auxiliary modality preliminary reconstruction feature block. The target modality preliminary reconstruction feature block and the auxiliary modality preliminary reconstruction feature block corresponding to each cross-attention reconstruction network are respectively reorganized to obtain a target modality preliminary reconstructed image and an auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascade image, and all auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascade image. The target modality cascade image and the auxiliary modality cascade image are input into a fusion network to obtain a target modality MRI reconstructed image and an auxiliary modality MRI reconstructed image. In this embodiment, the cross-attention reconstruction networks at different levels only have different inputs, and the structures and parameters of the cross-attention reconstruction networks are exactly the same; multiple cross-attention reconstruction networks are parallel, that is, there is no input-output relationship between the cross-attention reconstruction networks.

[0073] (1) Multi-scale feature extraction network

[0074] The multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network. The 2D under-sampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and the 2D under-sampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network. The target modality branch feature extraction network and the auxiliary modality branch feature extraction network both include multiple cascaded encoders, and each encoder includes a convolution layer, an activation layer, and a maximum pooling layer in sequence. Each level of encoder continuously reduces the image space dimension while improving the feature depth. The scales of the target modality feature images output by each level of encoder in the target modality branch feature extraction network are different, and the scales of the auxiliary modality feature images output by each level of encoder in the auxiliary modality branch feature extraction network are different. The target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks. The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network. The scale of the feature image output in the cascaded encoder decreases successively, and the scales of all target modality feature blocks and auxiliary modality feature blocks are the same. The multi-scale feature extraction network can extract multi-scale features from coarse to fine, helping the subsequent network to learn image features more easily. After the target modality feature image and auxiliary modality feature image of the same scale are divided into multiple target modality feature blocks and auxiliary modality feature blocks respectively, they are all input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module. The target modality feature image and auxiliary modality feature image of the same scale correspond to the same cross-attention reconstruction network.

[0075] In this example, the target modality branch feature extraction network and the auxiliary modality branch feature extraction network of the multi-scale feature extraction network both include three encoders. Each encoder has five convolutional layers, the convolution kernel size is 3×3, and each convolutional layer is followed by a ReLU activation layer and a maximum pooling layer.

[0076] (2) Cross-attention reconstruction network module

[0077] The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks, each of which includes a target modality attention branch network, an auxiliary modality attention branch network, and a learnable vector network; after the target modality feature image and the auxiliary modality feature image of the same scale are evenly divided into multiple target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input to the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input to the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network, so as to help process feature extraction and global relationship modeling, etc. The target modality attention branch network and the auxiliary modality attention branch network both include multiple cascaded transformer blocks, and the learnable vector network includes a modality mapping module and multiple learnable vector update modules in turn.

[0078] In each criss-cross attention reconstruction network:

[0079] The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network and the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the corresponding learnable vector and the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network;

[0080] The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network and the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks except the first-level transformer block in the auxiliary modality attention branch network includes the corresponding learnable vector and the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstruction feature block of the current cross-attention reconstruction network;

[0081] In the same cross-attention reconstruction network, the number of encoder levels of the target modality branch feature extraction network corresponding to the target modality attention branch network is the same as the number of encoder levels of the auxiliary modality branch feature extraction network corresponding to the auxiliary modality attention branch network;

[0082] In the same cross-attention reconstruction network, the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network correspond to the same learnable vectors; the first-level transformer blocks in the target modality attention branch network and the first-level transformer blocks in the auxiliary modality attention branch network correspond to the modality mapping module of the learnable vector network; the i-th level transformer blocks in the target modality attention branch network and the i-th level transformer blocks in the auxiliary modality attention branch network correspond to the i-1-th level learnable vector update module of the learnable vector network, where i represents the level of the transformer block, i∈{2, 3, …I}, and I represents the total number of transformer blocks in the target modality attention branch network or the auxiliary modality attention branch network in a cross-attention reconstruction network.

[0083] The learnable vector of the first-level transformer block in each cross-attention reconstruction network is the output vector obtained after the target modality feature block and the auxiliary modality feature block input to the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network; the learnable vectors corresponding to other transformer blocks except the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output feature block of the previous-level transformer block in the auxiliary modality attention branch network are input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network. The learnable vector network performs weighted updates on the features of the input transformer block to improve the learning ability of the network.

[0084] In the modal mapping module, the target modal feature block passes through the output of the first linear mapping layer of the modal mapping module, and the auxiliary modal feature block passes through the output of the second linear mapping layer of the modal mapping module, and then undergoes channel splicing, and then passes through the third linear mapping layer of the modal mapping module and the normalization layer of the modal mapping module in sequence, to obtain an output vector of the modal mapping module;

[0085] In each learnable vector update module, the output feature block of the previous transformer block in the target modality attention branch network and the output feature block of the previous transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

[0086] Furthermore, each transformer block includes a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer, and a fully connected layer in sequence. For each transformer block, the feature block input by the current transformer block first enters the position encoding layer of the current transformer block, adds the corresponding position information vector, and then passes through the linear mapping layer and the first normalization layer of the current transformer block in sequence before being input into the attention mechanism layer of the current transformer block. The learnable vector corresponding to the current transformer block is also input into the attention mechanism layer of the current transformer block; the output of the attention mechanism layer of the current transformer block then passes through the second normalization layer and the fully connected layer of the current transformer block in sequence to obtain the output feature block of the current transformer block. The output feature block of the current transformer block is input into the corresponding first-level learnable vector update module and the next-level transformer block of the same modality attention branch network.

[0087] Furthermore, for the attention mechanism layer of each transformer block, the feature block input by the transformer block is sequentially encoded by the position encoding layer, linear mapped by the linear mapping layer, and normalized by the first normalization layer in the transformer block to obtain vector Q (vector Q is an important feature that needs to be paid attention to). The learnable vector and vector Q are cascaded as vectors K and vector V (vectors K and vector V represent the global features of the image). The vectors Q, K, and V are calculated as follows to obtain the intermediate quantity z:

[0088]

[0089] Where Q represents vector Q, K represents vector K, V represents vector V, d kis the dimension of vector K, softmax is the normalized exponential function, and T represents transpose; the intermediate quantity z is added to the vector Q through a residual connection, and then passes through the linear mapping layer in the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

[0090] In this embodiment, the cross-attention reconstruction network module includes three cross-attention reconstruction networks, and the target modality attention branch network and the auxiliary modality attention branch network of each cross-attention reconstruction network include three cascaded transformer blocks. In addition, the learnable vector network includes one modality mapping module and two learnable vector update modules. Each cross-attention reconstruction network processes the primary features output by the multi-scale feature extraction network respectively.

[0091] (3) Converged Network

[0092] The scale of the target modality preliminary reconstructed image corresponding to the same cross-attention reconstruction network is the same as the scale of the target modality feature image input to the cross-attention reconstruction network, and the scale of the auxiliary modality preliminary reconstructed image corresponding to the same cross-attention reconstruction network is the same as the scale of the auxiliary modality feature image input to the cross-attention reconstruction network; all target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascade image, and all auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascade image; the target modality cascade image and the auxiliary modality cascade image are input into the fusion network.

[0093] The fusion network includes an input layer, multiple convolutional layers and an output layer. The number of channels of the input layer of the fusion network is the same as the number of channels of the target modality cascade image and the number of channels of the auxiliary modality cascade image. The number of channels of the output layer of the fusion network is 2, representing the real part and the imaginary part of the output result of the fusion network respectively. The complex result composed of the two-channel results output by the output layer is the final output MRI reconstructed image of the multimodal reconstruction network. The target modality cascade image and the auxiliary modality cascade image are input into the fusion network in sequence to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image respectively.

[0094] Or the fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork, and the target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, a plurality of convolutional layers, and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain a target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain an auxiliary modality MRI reconstructed image;

[0095] In this example, the fusion network includes 5 convolutional layers, the convolution kernel of each convolutional layer is 3×3, and each convolutional layer is added with a ReLU activation layer.

[0096] Step 3, set the total loss function. In this example, the total loss function includes the mean square error between the predicted target modality MRI reconstructed image and the corresponding 2D fully sampled target modality magnetic resonance image and the mean square error between the predicted auxiliary modality MRI reconstructed image and the corresponding 2D fully sampled auxiliary modality magnetic resonance image. The total loss function Loss is set as:

[0097] Loss = L(x ref ,f θ (y ref ))+L(x tar ,f θ (y tar ))+L SSIM (x ref ,f θ (y ref ))

[0098] +L SSIM (x tar ,f θ (y tar ))

[0099] Where L represents the error function, such as the commonly used mean square error or mean absolute error. In this example, the error function L is selected as the mean square error value, L SSIM is the structural similarity (SSIM) loss, x ref is a 2D fully sampled auxiliary modality MRI image, x tar is the 2D fully sampled target modality MRI image, y ref and tar are 2D undersampled auxiliary modality MRI image and 2D undersampled target modality MRI image, respectively, θ is a multimodal reconstruction network with parameter θ. The structural similarity loss enables the multimodal reconstruction network to fully learn the brightness and contrast of the 2D fully sampled auxiliary modality MRI image and the 2D fully sampled target modality MRI image, thereby improving the reconstruction details.

[0100] Step 4: According to the total loss function set in step 3, the multimodal reconstruction network constructed in step 2 is trained end-to-end using the training set generated in step 1, and the parameters of the multimodal reconstruction network are saved;

[0101] According to the multimodal reconstruction network built in step 2, the network learning rate is initialized to 0.0001, the batch size is set to 8, and the network training is performed on the PyTorch platform using the Adam optimizer. The 2D undersampled auxiliary modality MRI images and 2D undersampled target modality MRI images in the training set in step 1 are input into the multimodal reconstruction network to obtain the predicted target modality MRI reconstructed images and auxiliary modality MRI reconstructed images, and the network training is performed according to the total loss function set in step 4. The training is stopped after the total number of iterations of the network training reaches 200, and the parameters of the corresponding multimodal reconstruction network are saved.

[0102] Step 5: Input the 2D undersampled target modality MRI image to be reconstructed and the corresponding 2D undersampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

[0103] In this example, the 2D undersampled target modality magnetic resonance image and the 2D undersampled auxiliary modality magnetic resonance image to be reconstructed are the 2D undersampled images in the test set generated in step 1. 23 Na MRI with 2D undersampled T1-weighted 1 H MRI, 2D undersampling 23 Na MRI with 2D undersampled T1-weighted 1 The H magnetic resonance image is input into the multimodal reconstruction network trained in step 4, and the corresponding 23 Na MRI reconstructed images and T1-weighted 1 H MRI reconstructed image.

[0104] The target modality MRI reconstructed image obtained in step 5 is compared with the 2D fully sampled target modality MRI image corresponding to the test set, and the Peak Signal-to-Noise Ratio (PSNR) and structural similarity index between the two are calculated; the auxiliary modality MRI reconstructed image obtained in step 5 is compared with the 2D fully sampled auxiliary modality MRI image in the test set, and the Peak Signal-to-Noise Ratio (PSNR) and structural similarity index between the two are calculated.

[0105] Figure 3 Shows the 2D undersampling of one subject in the test set 23 The reconstruction results of Na magnetic resonance images, with PSNR / SSIM values ​​marked at the bottom of the image. (a) is 2D full sampling 23 Na magnetic resonance image, (b) is zero-filled 23 Na magnetic resonance image, (c) is the final image obtained by the method of the present invention23 Na MRI reconstructed image. As can be seen from the result figure, the multimodal deep learning-based 23 Na MRI accelerated imaging reconstruction method can be used to reconstruct 2D images from highly undersampled images. 23 High-quality reconstruction of Na magnetic resonance images 23 Na magnetic resonance images.

[0106] Example 2

[0107] Multi-mode 5.0T 23 Na magnetic resonance imaging artificial intelligence reconstruction device, using the multimodal 5.0T described in Example 1 23 Na MRI AI reconstruction methods include:

[0108] A network module, used to construct a multimodal reconstruction network in step 2 of embodiment 1;

[0109] A loss function construction module, used to construct and implement the total loss function in step 3 of embodiment 1;

[0110] The training module is used to implement the above step 4 and train the multimodal reconstruction network based on the training set.

[0111] Example 3

[0112] A computer device includes a memory and a processor, wherein a computer program is stored in the memory, and steps 2-5 in the above embodiment 1 are implemented when the processor executes the computer program.

[0113] Example 4

[0114] A computer-readable storage medium stores a computer program, which implements steps 2-5 in the above-mentioned embodiment 1 when executed by a processor.

[0115] Example 5

[0116] A computer program product includes a computer program, and when the computer program is executed by a processor, steps 2-5 in the above embodiment 1 are implemented.

[0117] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. Multi-mode 5.0T 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The following steps are involved: Step 1, obtaining a training set, the training set comprising a plurality of sample groups, each sample group comprising a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D under-sampled target modality magnetic resonance image and a 2D under-sampled auxiliary modality magnetic resonance image of a subject; Step 2: construct a multimodal reconstruction network, which includes a multi-scale feature extraction network, a cross-attention reconstruction network module and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks. The 2D under-sampled target modality magnetic resonance images and 2D under-sampled auxiliary modality magnetic resonance images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then are input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain a target modality preliminary reconstruction feature block and an auxiliary modality preliminary reconstruction feature block. The target modality preliminary reconstruction feature block and the auxiliary modality preliminary reconstruction feature block corresponding to each cross-attention reconstruction network are respectively reorganized to obtain a target modality preliminary reconstructed image and an auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascaded image. All auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascaded image. The target modality cascaded image and the auxiliary modality cascaded image are input into a fusion network to obtain a target modality MRI reconstructed image and an auxiliary modality MRI reconstructed image. Step 3: Set the total loss function; Step 4: According to the total loss function, the multimodal reconstruction network is trained using the sample group in the training set; Step 5: Input the 2D under-sampled target modality MRI image to be reconstructed and the corresponding 2D under-sampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

2. The multi-modal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network, a 2D under-sampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and a 2D under-sampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network, and the target modality branch feature extraction network and the auxiliary modality branch feature extraction network both include a plurality of cascaded encoders, and each encoder includes a convolution layer, an activation layer, and a maximum pooling layer in sequence; The scales of the target modality feature images output by encoders at various levels in the target modality branch feature extraction network are different, and the scales of the auxiliary modality feature images output by encoders at various levels in the auxiliary modality branch feature extraction network are different; the target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks; The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network.

3. The multi-modal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that Each of the cross-attention reconstruction networks includes a target modality attention branch network and an auxiliary modality attention branch network; after the target modality feature image and the auxiliary modality feature image of the same scale are respectively divided into a plurality of target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input into the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input into the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network; Both the target modality attention branch network and the auxiliary modality attention branch network include multiple cascaded transformer blocks; In each criss-cross attention reconstruction network: The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network; The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the auxiliary modality attention branch network includes the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstructed feature block of the current cross-attention reconstruction network.

4. The multi-modal 5.0T according to claim 3 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that In each of the cross-attention reconstruction networks, The input of the first-level transformer block of the target modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks in the target modality attention branch network except the first-level transformer block also includes the corresponding learnable vector; The input of the first-level transformer block of the auxiliary modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block also includes the corresponding learnable vector; The learnable vectors corresponding to the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network are the same; and the learnable vector of the first-level transformer block is the output vector obtained after the target modality feature block and the auxiliary modality feature block input into the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network; the learnable vectors corresponding to other transformer blocks except the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output vector of the previous-level transformer block in the auxiliary modality attention branch network after being input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network.

5. The multi-modal 5.0T according to claim 4 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that In the modal mapping module, the target modal feature block passes through the output of the first linear mapping layer of the modal mapping module, and the auxiliary modal feature block passes through the output of the second linear mapping layer of the modal mapping module, and then undergoes channel splicing, and then passes through the third linear mapping layer of the modal mapping module and the normalization layer of the modal mapping module in sequence, to obtain an output vector of the modal mapping module; In each learnable vector update module, the output feature block of the previous transformer block in the target modality attention branch network and the output feature block of the previous transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

6. The multi-modal 5.0T according to claim 5 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that Each of the transformer blocks includes a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer and a fully connected layer in sequence; The feature block input by the transformer block is input into the position encoding layer of the current transformer, and the learnable vector corresponding to the current transformer block is input into the attention mechanism layer of the current transformer block; The feature block input by the transformer block is sequentially encoded by the position encoding layer, linear mapped by the linear mapping layer, and normalized by the first normalization layer in the transformer block to obtain the vector Q. In the attention mechanism layer, the vector Q can be learned and cascaded with the vector Q as the vector K and the vector V. The vectors Q, K, and V are calculated as follows to obtain the intermediate quantity z: Where Q represents vector Q, K represents vector K, V represents vector V, d k is the dimension of vector K, softmax is the normalized exponential function, and T represents transpose; The intermediate quantity z is added to the vector Q through the residual connection, and then passes through the linear mapping layer of the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

7. The multi-modal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The fusion network includes an input layer, multiple convolutional layers and an output layer; the target modality cascade image and the auxiliary modality cascade image are sequentially input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image respectively.

8. The multi-modal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork, and the target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, multiple convolutional layers and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain a target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain an auxiliary modality MRI reconstructed image.

9. The multi-modal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The total loss function Loss is: Loss=L(x ref ,f θ (y ref ))+L(x tar ,f θ (y tar ))+L SSIM (x ref ,f θ (y ref ))+L SSIM (x tar ,f θ (y tar )) Where L is the error function, L SSIM is the structural similarity loss, x ref is a 2D fully sampled auxiliary modality MRI image, x tar is the 2D fully sampled target modality MRI image, y ref and tar The distribution is a 2D undersampled auxiliary modality MRI image and a 2D undersampled target modality MRI image, f θ is a multimodal reconstruction network with parameters θ.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, steps 2 to 5 of the reconstruction method according to any one of claims 1 to 9 are implemented.

Images