Multimodal 5.0T 23 Artificial Intelligence Reconstruction Method for Na Magnetic Resonance Imaging

Abstract

The present invention discloses a multi-modal 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging constructs a multimodal reconstruction network, which includes a multi-scale feature extraction network, a cross-attention reconstruction network module and a fusion network. The multi-scale feature extraction network extracts multi-level features of undersampled target modality and auxiliary modality magnetic resonance images. The cross-attention reconstruction network learns information between different modalities. The fusion network fuses the multi-scale information of the multimodality to obtain the final reconstructed image. In addition, the present invention uses a structural similarity loss function to enable the multimodal reconstruction network to fully learn the uniform contrast information of 5.0T magnetic resonance image data. The present invention utilizes the information complementarity between multimodal MRI data to enhance the detailed features of the reconstructed image, and fully utilizes the uniform contrast characteristics of 5.0T data, which can significantly improve the reconstruction quality of the target modality.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal 5.0T 23 Na magnetic resonance imaging artificial intelligence reconstruction method, suitable for magnetic resonance image reconstruction. Background Art

[0002] Sodium magnetic resonance imaging 23 Na MRI) provides a non-invasive method to measure tissue activity and can provide information on the physiological and biochemical processes of cells. It can quantitatively measure the sodium ion concentration in human brain tissue. 1 H signal ratio 23 The MRI signal of Na is about 12,000 times higher. 23 The low Na concentration and biexponential decay of the signal result in a low signal-to-noise ratio, which limits image quality and requires longer imaging times to compensate.

[0003] K-space undersampling is a common method for reducing MRI acquisition time. Compressed sensing (CS) exploits the sparsity of data in the sparse domain and, combined with iterative algorithms, can recover undersampled k-space data, thereby removing artifacts caused by undersampling. However, CS algorithms suffer from long reconstruction times and limited reconstruction accuracy.

[0004] Deep learning methods have shown great potential in accelerating MRI imaging in recent years. Deep learning methods have successfully improved undersampling by learning abstract features of data. 1 H magnetic resonance image quality. Huang et al. used a self-supervised method to train the model to expand the network DURED-Net and successfully reconstructed the details of undersampled images under high acceleration. Adlung et al. used U-Net and ResNet structures to reconstruct undersampled images. 23 Na brain magnetic resonance images, successfully improved the quality of undersampled images. However, at present 23 Na data has a low signal-to-noise ratio, and only 23 Na data training may not achieve the best reconstruction results. Multimodal reconstruction strategies can use auxiliary modality information to help target modality reconstruction, thereby improving the target modality image quality.

[0005] In pursuit of higher signal-to-noise ratios, higher resolution, and higher tissue contrast, high-field magnetic resonance imaging of the human body has been a major research hotspot in recent years. Ultra-high-field MRI (e.g., 7.0T) can image in many scenarios that were previously impossible to image and can provide more image details. However, ultra-high-field MRI faces many challenges. For example, the main challenges of 7.0T MRI are the inhomogeneity of the transmitted B1 field and the deposition of radiofrequency power in tissues. The recently developed 5.0T MRI scanner not only has good spatial resolution and signal-to-noise ratio, but also has a more uniform B1 field, which has good application prospects.

[0006] In summary, the existing technology has the following problems:

[0007] 1. Currently used for 23 Deep learning reconstruction technology for Na MRI imaging is relatively simple. Using simple network structures such as Unet or ResNet may not fully exploit the redundant information of undersampled data, resulting in insufficient acceleration.

[0008] 2. 23 Na MR image data has a low signal-to-noise ratio. 23 Na data may affect deep learning network training and degrade reconstruction quality.

[0009] 3. Ultra-high field MRI equipment is expensive, the technology is more complex, and B1 field inhomogeneity may lead to image quality degradation. Summary of the Invention

[0010] In view of the above problems existing in the prior art, the present invention proposes a multi-modal 5.0T 23 Artificial intelligence reconstruction method for Na magnetic resonance imaging.

[0011] The above-mentioned purpose of the present invention is achieved by the following method:

[0012] Multimodal 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging comprises the following steps:

[0013] Step 1: obtaining a training set, where the training set includes multiple sample groups, each sample group including a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D undersampled target modality magnetic resonance image, and a 2D undersampled auxiliary modality magnetic resonance image of a subject;

[0014] Step 2: Construct a multimodal reconstruction network. The multimodal reconstruction network includes a multi-scale feature extraction network, a cross-attention reconstruction network module, and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks.

[0015] The 2D undersampled target modality magnetic resonance images and 2D undersampled auxiliary modality magnetic resonance images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain a target modality preliminary reconstructed feature block and an auxiliary modality preliminary reconstructed feature block. The target modality preliminary reconstructed feature block and the auxiliary modality preliminary reconstructed feature block corresponding to each cross-attention reconstruction network are respectively recombined to obtain a target modality preliminary reconstructed image and an auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascaded image. All auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascaded image. The target modality cascaded image and the auxiliary modality cascaded image are input into a fusion network to obtain a target modality MRI reconstructed image and an auxiliary modality MRI reconstructed image.

[0016] Step 3: Set the total loss function;

[0017] Step 4: Based on the total loss function, the multimodal reconstruction network is trained using the sample group in the training set;

[0018] Step 5: Input the 2D undersampled target modality MRI image to be reconstructed and the corresponding 2D undersampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

[0019] As described above, the multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network, a 2D undersampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and a 2D undersampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network, and the target modality branch feature extraction network and the auxiliary modality branch feature extraction network each include a plurality of cascaded encoders, each encoder including a convolution layer, an activation layer, and a maximum pooling layer in sequence;

[0020] The scales of target modality feature images output by encoders at various levels in the target modality branch feature extraction network are different, and the scales of auxiliary modality feature images output by encoders at various levels in the auxiliary modality branch feature extraction network are different; the target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks;

[0021] The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network.

[0022] Each cross-attention reconstruction network as described above includes a target modality attention branch network and an auxiliary modality attention branch network; after the target modality feature image and the auxiliary modality feature image of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input into the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input into the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network;

[0023] Both the target modality attention branch network and the auxiliary modality attention branch network include multiple cascaded transformer blocks;

[0024] In each cross-attention reconstruction network:

[0025] The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network;

[0026] The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block includes the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstructed feature block of the current cross-attention reconstruction network.

[0027] In each cross-attention reconstruction network as described above,

[0028] The input of the first-level transformer block of the target modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks in the target modality attention branch network except the first-level transformer block also includes the corresponding learnable vector;

[0029] The input of the first-level transformer block of the auxiliary modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block also includes the corresponding learnable vector;

[0030] The learnable vectors corresponding to the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network are the same; and the learnable vector of the first-level transformer block is the output vector obtained after the target modality feature block and the auxiliary modality feature block input into the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network; the learnable vectors corresponding to other transformer blocks except the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output vector of the previous-level transformer block in the auxiliary modality attention branch network after being input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network.

[0031] In the modal mapping module described above, the output of the first linear mapping layer of the modal mapping module is used for the target modal feature block, and the output of the second linear mapping layer of the modal mapping module is used for the auxiliary modal feature block. After channel concatenation, the concatenation is sequentially performed through the third linear mapping layer and the normalization layer of the modal mapping module to obtain the output vector of the modal mapping module.

[0032] In each learnable vector update module, the output feature block of the previous level transformer block in the target modality attention branch network and the output feature block of the previous level transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous level transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

[0033] Each transformer block as described above includes, in sequence, a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer, and a fully connected layer;

[0034] The feature block input by the transformer block is input into the position encoding layer of the current transformer, and the learnable vector corresponding to the current transformer block is input into the attention mechanism layer of the current transformer block;

[0035] The feature block input by the transformer block is sequentially position-encoded by the position encoding layer, linearly mapped by the linear mapping layer, and layer-normalized by the first normalization layer in the transformer block to obtain a vector Q. In the attention mechanism layer, the learning vector and vector Q are cascaded as vectors K and V. Vectors Q, K, and V are calculated as follows to obtain the intermediate quantity z:

[0036]

[0037] Among them, Q represents vector Q, K represents vector K, V represents vector V, d k is the dimension of vector K, softmax is the normalized exponential function, and T represents transpose;

[0038] The intermediate quantity z is added to the vector Q through a residual connection, and then passes through the linear mapping layer of the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

[0039] As described above, the fusion network includes an input layer, multiple convolutional layers and an output layer; the target modality cascade image and the auxiliary modality cascade image are sequentially input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image, respectively.

[0040] As shown above, the fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork. The target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, multiple convolutional layers and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain the target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain the auxiliary modality MRI reconstructed image.

[0041] As mentioned above, the total loss function Loss is:

[0042] Loss=L(x ref ,f θ (y ref ))+L(x tar ,f θ (y tar ))+L SSIM (x ref ,f θ(y ref ))

[0043] +L SSIM (x tar ,f θ (y tar ))

[0044] Where L is the error function, L SSIM is the structural similarity loss, x ref is a 2D fully sampled auxiliary modality magnetic resonance image, x tar is the 2D fully sampled target modality MRI image, y ref and y tar The distribution is a 2D undersampled auxiliary modality MRI image and a 2D undersampled target modality MRI image, f θ is a multimodal reconstruction network with parameters θ.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements steps 2-5 of the reconstruction method according to any one of claims 1 to 9 when executing the computer program.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. The present invention builds a multi-scale feature extraction network and uses multiple encoder structures to extract image features, thereby obtaining multi-scale features from coarse to fine, thereby improving the reconstruction quality.

[0048] 2. The present invention uses the attention mechanism to learn multimodal and multi-scale information, fully explores the information association and information redundancy between the auxiliary modality and the target modality, and uses the auxiliary modality information to help reconstruct the target modality and improve the reconstruction performance.

[0049] 3. Based on the good B1 field uniformity of 5.0T MRI, the image contrast is more uniform. Therefore, the present invention uses a structural similarity loss function to ensure that the brightness and contrast of the reconstructed results are consistent with the full sampling data, making full use of the characteristics of 5.0T MRI data to improve the details of the reconstruction results, thereby further improving the reconstruction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the present invention;

[0051] Figure 2 Schematic diagram of the overall structure of the multimodal reconstruction network;

[0052] Figure 3 Schematic diagram of the overall structure of the cross attention reconstruction network;

[0053] Figure 4 Schematic diagram of the attention mechanism layer structure, where (a) is a schematic diagram of the attention mechanism layer structure in the target modality attention branch network, and (b) is a schematic diagram of the attention mechanism layer structure in the auxiliary modality attention branch network; represents multiplication, Represents residual connection addition; Q represents vector Q, K represents vector K, and V represents vector V;

[0054] Figure 5 This is a schematic diagram of the fusion network structure;

[0055] Figure 6 This is a schematic diagram of the modal mapping module structure;

[0056] Figure 7 This is a schematic diagram of the structure of the learnable vector update module;

[0057] in, Indicates channel splicing;

[0058] Figure 8 Rebuild for the test set 23 Comparison of Na magnetic resonance imaging results, where (a) is 2D full sampling 23 Na magnetic resonance image, (b) is zero-filled 23 Na magnetic resonance image, (c) is the final reconstruction 23 Na magnetic resonance images. Specific implementation plan

[0059] In order to facilitate those skilled in the art to understand and implement the present invention, the following examples and Figure 1-Figure 3 The present invention will be described in further detail. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, and are not intended to limit the present invention.

[0060] Multimodal 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging specifically includes the following steps:

[0061] Step 1: Obtain a training set and a test set, wherein both the training set and the test set include multiple sample groups, each sample group includes a 2D fully sampled target modality MRI image, a 2D fully sampled auxiliary modality MRI image, a 2D undersampled target modality MRI image, and a 2D undersampled auxiliary modality MRI image of a subject. In this example, the target modality is 23 Na, auxiliary modality is T1-weighted 1 H. The specific method is as follows:

[0062] Step 1.1: Scan multiple subjects to obtain corresponding 3D full-sampling target modality MRI k-space data and 3D full-sampling auxiliary modality MRI k-space data.

[0063] In this embodiment, multiple subjects are scanned to obtain corresponding 3D full sampling 23 Na MRI k-space data and 3D fully sampled T1-weighted 1 H MRI k-space data.

[0064] Step 1.2: Extract 3D fully sampled target modality MRI k-space data and 3D fully sampled auxiliary modality MRI k-space data layer by layer, and obtain corresponding multiple 2D fully sampled target modality MRI k-space data and 2D fully sampled auxiliary modality MRI k-space data respectively; undersample the 2D fully sampled target modality MRI k-space data and 2D fully sampled auxiliary modality MRI k-space data according to the undersampled sampling matrix, and obtain corresponding 2D undersampled target modality MRI k-space data and 2D undersampled auxiliary modality MRI k-space data respectively.

[0065] In this embodiment, 3D full sampling is extracted layer by layer. 23 Na MRI k-space data, obtaining multiple 2D full samples 23 NaMRI k-space data, 2D full sampling according to the undersampling sampling matrix 23 Na MRI k-space data is undersampled to obtain the corresponding 2D undersampling 23 Na MRI k-space data. For 3D fully sampled T1-weighted 1 The H MRI k-space data is processed in the same way, that is, 3D full-sample T1-weighted images are first extracted layer by layer. 1 H MRI k-space data, obtaining multiple 2D full samples 1 H MRI k-space data, 2D full sampling according to the undersampling sampling matrix 1 H MRI k-space data is undersampled to obtain 2D undersampling 1 H MRIk-space data.

[0066] Step 1.3, perform two-dimensional inverse Fourier transform on the 2D fully sampled target modality MRI k-space data, the 2D fully sampled auxiliary modality MRI k-space data, the 2D undersampled target modality MRI k-space data, and the 2D undersampled auxiliary modality MRI k-space data, respectively, to obtain a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D undersampled target modality magnetic resonance image, and a 2D undersampled auxiliary modality magnetic resonance image, respectively.

[0067] In this embodiment, 2D full sampling is obtained by two-dimensional inverse Fourier transform processing. 23 Na magnetic resonance imaging, 2D full-sample T1-weighted 1 H MRI, 2D undersampling23 Na magnetic resonance images and 2D undersampled T1-weighted 1 H magnetic resonance image.

[0068] Step 1.4: The 2D fully-sampled target modality MRI images, 2D fully-sampled auxiliary modality MRI images, 2D under-sampled target modality MRI images, and 2D under-sampled auxiliary modality MRI images of the same subject are taken as a sample group, and all sample groups are divided into training sets and test sets. Both the training set and the test set include multiple sample groups.

[0069] In this embodiment, the 2D undersampling of a subject 23 Na magnetic resonance imaging, 2D full sampling 23 Na magnetic resonance image, 2D undersampled T1-weighted 1 H MRI and 2D full-sample T1-weighted 1 H magnetic resonance images are used as sample groups, and all sample groups are divided into training sets and test sets. Both the training set and the test set include multiple sample groups.

[0070] In this embodiment, the undersampling mode selects variable density undersampling, and the undersampling acceleration factor is 4. 23 Na MRI and T2-weighted 1 The contrast of H magnetic resonance images is similar. This example uses the T2 weighted image of the IXI public dataset. 1 H MRI images were used to simulate the target modality 5.0T 23 The experiments were conducted using the IXI public dataset T1-weighted MRI images. 1 H magnetic resonance images are used as auxiliary modalities, and 1000 sample groups are selected as training sets and 300 sample groups are selected as test sets. 23 Na MRI images and 1 H magnetic resonance image.

[0071] Step 2: Construct a multimodal reconstruction network. The multimodal reconstruction network includes a multi-scale feature extraction network, a cross-attention reconstruction network module and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks.

[0072] The 2D undersampled target modality MRI images and 2D undersampled auxiliary modality MRI images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain the target modality preliminary reconstructed feature block and the auxiliary modality preliminary reconstructed feature block. The target modality preliminary reconstructed feature block and the auxiliary modality preliminary reconstructed feature block corresponding to each cross-attention reconstruction network are respectively recombined to obtain the target modality preliminary reconstructed image and the auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-wise cascaded to obtain the target modality cascaded image, and all auxiliary modality preliminary reconstructed images are channel-wise cascaded to obtain the auxiliary modality cascaded image. The target modality cascaded image and the auxiliary modality cascaded image are input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image. In this embodiment, the cross-attention reconstruction networks at different levels only have different inputs, and the structures and parameters of the cross-attention reconstruction networks are exactly the same; multiple cross-attention reconstruction networks are parallel, that is, there is no input-output relationship between the cross-attention reconstruction networks.

[0073] (1) Multi-scale feature extraction network

[0074] The multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network. A 2D undersampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and a 2D undersampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network. Both the target modality branch feature extraction network and the auxiliary modality branch feature extraction network include multiple cascaded encoders, each of which sequentially includes a convolutional layer, an activation layer, and a maximum pooling layer. Each encoder level continuously reduces the spatial dimension of the image while increasing the feature depth. The target modality feature images output by each encoder level in the target modality branch feature extraction network have different scales, and the auxiliary modality feature images output by each encoder level in the auxiliary modality branch feature extraction network have different scales. The target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks. The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network. The scale of the feature image output by the cascaded encoders decreases successively, and the scale of all target modality feature blocks and auxiliary modality feature blocks is the same. The multi-scale feature extraction network can extract multi-scale features from coarse to fine, helping subsequent networks to learn image features more easily. The target modality feature image and auxiliary modality feature image of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module. The target modality feature image and auxiliary modality feature image of the same scale correspond to the same cross-attention reconstruction network.

[0075] In this example, the target modality branch feature extraction network and the auxiliary modality branch feature extraction network of the multi-scale feature extraction network each include three encoders. Each encoder has five convolutional layers with a convolution kernel size of 3×3. Each convolutional layer is followed by a ReLU activation layer and a maximum pooling layer.

[0076] (2) Cross-attention reconstruction network module

[0077] The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks, each of which includes a target modality attention branch network, an auxiliary modality attention branch network, and a learnable vector network. After the target modality feature image and auxiliary modality feature image of the same scale are divided into multiple target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input into the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input into the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network, thereby assisting in feature extraction and global relationship modeling. Both the target modality attention branch network and the auxiliary modality attention branch network include multiple cascaded transformer blocks, and the learnable vector network includes a modality mapping module and multiple learnable vector update modules.

[0078] In each cross-attention reconstruction network:

[0079] The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network and the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the corresponding learnable vector and the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network;

[0080] The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network and the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block includes the corresponding learnable vector and the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstructed feature block of the current cross-attention reconstruction network;

[0081] In the same cross-attention reconstruction network, the number of encoder levels of the target modality branch feature extraction network corresponding to the target modality attention branch network is the same as the number of encoder levels of the auxiliary modality branch feature extraction network corresponding to the auxiliary modality attention branch network;

[0082] In the same cross-attention reconstruction network, the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network correspond to the same learnable vectors; the first-level transformer block in the target modality attention branch network and the first-level transformer block in the auxiliary modality attention branch network correspond to the modality mapping module of the learnable vector network; the i-th level transformer block in the target modality attention branch network and the i-th level transformer block in the auxiliary modality attention branch network correspond to the i-1-th level learnable vector update module of the learnable vector network, where i represents the number of transformer blocks, i∈{2, 3,…I}, and I represents the total number of transformer blocks in the target modality attention branch network or the auxiliary modality attention branch network in a cross-attention reconstruction network.

[0083] The learnable vectors of the first-level transformer blocks in each cross-attention reconstruction network are the output vectors obtained after the target modality feature block and the auxiliary modality feature block of the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network. The learnable vectors corresponding to transformer blocks other than the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output feature block of the previous-level transformer block in the auxiliary modality attention branch network, after being input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network. The learnable vector network performs weighted updates on the features of the input transformer block to improve the network's learning ability.

[0084] In the modal mapping module, the output of the first linear mapping layer of the modal mapping module is used for the target modal feature block, and the output of the second linear mapping layer of the modal mapping module is used for the auxiliary modal feature block. After channel concatenation, the concatenation is performed sequentially through the third linear mapping layer and the normalization layer of the modal mapping module to obtain the output vector of the modal mapping module.

[0085] In each learnable vector update module, the output feature block of the previous level transformer block in the target modality attention branch network and the output feature block of the previous level transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous level transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

[0086] Furthermore, each transformer block includes a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer, and a fully connected layer. For each transformer block, the feature block input by the current transformer block first enters the position encoding layer of the current transformer block, is added with the corresponding position information vector, and then passes through the linear mapping layer and the first normalization layer of the current transformer block before being input into the attention mechanism layer of the current transformer block. The learnable vector corresponding to the current transformer block is also input into the attention mechanism layer of the current transformer block; the output of the attention mechanism layer of the current transformer block then passes through the second normalization layer and the fully connected layer of the current transformer block to obtain the output feature block of the current transformer block. The output feature block of the current transformer block is input into the corresponding first-level learnable vector update module and the next-level transformer block of the same modality attention branch network.

[0087] Furthermore, for the attention mechanism layer of each transformer block, the feature block input by the transformer block is sequentially position-encoded by the position encoding layer, linearly mapped by the linear mapping layer, and layer-normalized by the first normalization layer in the transformer block to obtain vector Q (vector Q is the important feature that needs to be paid attention to). The learnable vector and vector Q are cascaded to form vectors K and vector V (vectors K and V represent the global features of the image). Vectors Q, K, and V are calculated as follows to obtain the intermediate quantity z:

[0088]

[0089] Among them, Q represents vector Q, K represents vector K, V represents vector V, d kis the dimension of vector K, softmax is the normalized exponential function, and T represents transpose; the intermediate quantity z is added to the vector Q through a residual connection, and then passes through the linear mapping layer in the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

[0090] In this embodiment, the cross-attention reconstruction network module includes three cross-attention reconstruction networks. The target modality attention branch network and the auxiliary modality attention branch network of each cross-attention reconstruction network include three cascaded transformer blocks. In addition, the learnable vector network includes one modality mapping module and two learnable vector update modules. Each cross-attention reconstruction network processes the first-level features output by the multi-scale feature extraction network.

[0091] (3) Converged Network

[0092] The scale of the target modality preliminary reconstructed image corresponding to the same cross-attention reconstruction network is the same as the scale of the target modality feature image input to the cross-attention reconstruction network, and the scale of the auxiliary modality preliminary reconstructed image corresponding to the same cross-attention reconstruction network is the same as the scale of the auxiliary modality feature image input to the cross-attention reconstruction network; all target modality preliminary reconstructed images are channel-concatenated to obtain the target modality cascade image, and all auxiliary modality preliminary reconstructed images are channel-concatenated to obtain the auxiliary modality cascade image; the target modality cascade image and the auxiliary modality cascade image are input into the fusion network.

[0093] The fusion network consists of an input layer, multiple convolutional layers, and an output layer. The number of channels in the input layer of the fusion network is the same as that of the target modality cascade image and the auxiliary modality cascade image. The output layer of the fusion network has two channels, representing the real and imaginary parts of the fusion network output, respectively. The two-channel output of the output layer forms a complex result, which is the final output of the multimodal reconstruction network, the MRI reconstructed image. The target modality cascade image and the auxiliary modality cascade image are sequentially input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image, respectively.

[0094] Alternatively, the fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork, and the target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, multiple convolutional layers, and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain a target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain an auxiliary modality MRI reconstructed image;

[0095] In this example, the fusion network consists of 5 convolutional layers, each with a 3×3 convolution kernel, and each convolutional layer is added with a ReLU activation layer.

[0096] Step 3: Set the total loss function. In this example, the total loss function includes the mean square error between the predicted target modality MRI reconstructed image and the corresponding 2D fully sampled target modality MRI image, and the mean square error between the predicted auxiliary modality MRI reconstructed image and the corresponding 2D fully sampled auxiliary modality MRI image. The total loss function Loss is set as:

[0097] Loss=L(x ref ,f θ (y ref ))+L(x tar ,f θ (y tar ))+L SSIM (x ref ,f θ (y ref ))

[0098] +L SSIM (x tar ,f θ (y tar ))

[0099] Where L represents the error function, such as the commonly used mean square error or mean absolute error. In this example, the error function L is selected as the mean square error value, L SSIM is the structural similarity (SSIM) loss, x ref is a 2D fully sampled auxiliary modality magnetic resonance image, x tar is the 2D fully sampled target modality MRI image, y ref and y tar are 2D undersampled auxiliary modality MRI images and 2D undersampled target modality MRI images, respectively, f θ is a multimodal reconstruction network with parameter θ. The structural similarity loss enables the multimodal reconstruction network to fully learn the brightness and contrast of the 2D fully sampled auxiliary modality MRI image and the 2D fully sampled target modality MRI image, thereby improving the reconstruction details.

[0100] Step 4: Based on the total loss function set in step 3, the multimodal reconstruction network constructed in step 2 is trained end-to-end using the training set generated in step 1, and the parameters of the multimodal reconstruction network are saved.

[0101] Based on the multimodal reconstruction network constructed in step 2, the network learning rate was initialized to 0.0001, the batch size was set to 8, and the network was trained using the Adam optimizer on the PyTorch platform. The 2D undersampled auxiliary modality MRI images and 2D undersampled target modality MRI images from the training set in step 1 were input into the multimodal reconstruction network to obtain the predicted target modality MRI reconstructed images and auxiliary modality MRI reconstructed images. The network was then trained according to the total loss function set in step 4. Training was terminated after the total number of network iterations reached 200, and the corresponding multimodal reconstruction network parameters were saved.

[0102] Step 5: Input the 2D undersampled target modality MRI image to be reconstructed and the corresponding 2D undersampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

[0103] In this example, the 2D undersampled target modality MRI image and the 2D undersampled auxiliary modality MRI image to be reconstructed are the 2D undersampled images in the test set generated in step 1. 23 Na MRI and 2D undersampled T1-weighted 1 H MRI, 2D undersampling 23 Na MRI and 2D undersampled T1-weighted 1 The H magnetic resonance image is input into the multimodal reconstruction network trained in step 4, and the corresponding 23 Na MRI reconstructed images and T1-weighted 1 H MRI reconstructed image.

[0104] The target modality MRI reconstructed image obtained in step 5 is compared with the 2D fully sampled target modality MRI image corresponding to the test set, and the Peak Signal-to-Noise Ratio (PSNR) and structural similarity index between the two are calculated; the auxiliary modality MRI reconstructed image obtained in step 5 is compared with the 2D fully sampled auxiliary modality MRI image in the test set, and the Peak Signal-to-Noise Ratio (PSNR) and structural similarity index between the two are calculated.

[0105] Figure 3 Shows the 2D undersampling corresponding to one subject in the test set 23 The reconstruction results of Na magnetic resonance images, with PSNR / SSIM values ​​marked at the bottom of the image. (a) is 2D full sampling 23 Na magnetic resonance image, (b) is zero-filled 23 Na magnetic resonance image, (c) is the final image obtained by the method of the present invention23 Na MRI reconstructed image. As can be seen from the result graph, the multimodal deep learning-based 23 Na magnetic resonance accelerated imaging reconstruction method can be used to reconstruct 2D images from highly undersampled images. 23 High-quality reconstruction of Na magnetic resonance images 23 Na magnetic resonance images.

[0106] Example 2

[0107] Multimodal 5.0T 23 Na magnetic resonance imaging artificial intelligence reconstruction device, using the multimodal 5.0T described in Example 1 23 Na MRI artificial intelligence reconstruction methods include:

[0108] A network module, used to construct a multimodal reconstruction network for implementing step 2 of embodiment 1;

[0109] A loss function construction module, used to construct the total loss function in step 3 of embodiment 1;

[0110] The training module is used to implement the above step 4 and train the multimodal reconstruction network based on the training set.

[0111] Example 3

[0112] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements steps 2-5 in the above embodiment 1 when executing the computer program.

[0113] Example 4

[0114] A computer-readable storage medium stores a computer program, which implements steps 2-5 in the above embodiment 1 when executed by a processor.

[0115] Example 5

[0116] A computer program product includes a computer program, which implements steps 2-5 in the above embodiment 1 when executed by a processor.

[0117] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. Multi-mode 5.0T 23 The artificial intelligence reconstruction method for Na magnetic resonance imaging is characterized by: The following steps are involved: Step 1: obtaining a training set, where the training set includes multiple sample groups, each sample group including a 2D fully sampled target modality magnetic resonance image, a 2D fully sampled auxiliary modality magnetic resonance image, a 2D undersampled target modality magnetic resonance image, and a 2D undersampled auxiliary modality magnetic resonance image of a subject; Step 2: Construct a multimodal reconstruction network. The multimodal reconstruction network includes a multi-scale feature extraction network, a cross-attention reconstruction network module, and a fusion network. The cross-attention reconstruction network module includes multiple parallel cross-attention reconstruction networks. The 2D undersampled target modality magnetic resonance images and 2D undersampled auxiliary modality magnetic resonance images of the same sample group are input into a multi-scale feature extraction network to extract target modality feature images of different scales and auxiliary modality feature images of different scales. The target modality feature images and auxiliary modality feature images of the same scale are respectively divided into multiple target modality feature blocks and auxiliary modality feature blocks, and then input into a corresponding cross-attention reconstruction network in the cross-attention reconstruction network module to obtain a target modality preliminary reconstructed feature block and an auxiliary modality preliminary reconstructed feature block. The target modality preliminary reconstructed feature block and the auxiliary modality preliminary reconstructed feature block corresponding to each cross-attention reconstruction network are respectively recombined to obtain a target modality preliminary reconstructed image and an auxiliary modality preliminary reconstructed image. All target modality preliminary reconstructed images are channel-concatenated to obtain a target modality cascaded image. All auxiliary modality preliminary reconstructed images are channel-concatenated to obtain an auxiliary modality cascaded image. The target modality cascaded image and the auxiliary modality cascaded image are input into a fusion network to obtain a target modality MRI reconstructed image and an auxiliary modality MRI reconstructed image. Step 3: Set the total loss function; ， in, is the total loss function, is the error function, is the structural similarity loss, is a 2D fully sampled auxiliary modality magnetic resonance image, is a 2D fully sampled target modality magnetic resonance image, and The distribution is a 2D undersampled auxiliary modality MRI image and a 2D undersampled target modality MRI image, The parameter is Multimodal reconstruction network; Step 4: Based on the total loss function, the multimodal reconstruction network is trained using the sample group in the training set; Step 5: Input the 2D undersampled target modality MRI image to be reconstructed and the corresponding 2D undersampled auxiliary modality MRI image to be reconstructed into the multimodal reconstruction network trained in step 4 to obtain the corresponding target modality MRI reconstructed image and auxiliary modality MRI reconstructed image.

2. The multimodal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The multi-scale feature extraction network includes a target modality branch feature extraction network and an auxiliary modality branch feature extraction network, a 2D undersampled target modality magnetic resonance image is input into the target modality branch feature extraction network, and a 2D undersampled auxiliary modality magnetic resonance image is input into the auxiliary modality branch feature extraction network, and the target modality branch feature extraction network and the auxiliary modality branch feature extraction network each include a plurality of cascaded encoders, each encoder including a convolution layer, an activation layer and a maximum pooling layer in sequence; The scales of target modality feature images output by encoders at various levels in the target modality branch feature extraction network are different, and the scales of auxiliary modality feature images output by encoders at various levels in the auxiliary modality branch feature extraction network are different; the target modality feature image is evenly divided into target modality feature blocks, and the auxiliary modality feature image is evenly divided into auxiliary modality feature blocks; The scale of the target modality feature image output by the encoder in the target modality branch feature extraction network is the same as the scale of the auxiliary modality feature image output by the encoder at the same level in the auxiliary modality branch feature extraction network.

3. The multimodal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that Each of the cross-attention reconstruction networks includes a target modality attention branch network and an auxiliary modality attention branch network; after the target modality feature image and the auxiliary modality feature image of the same scale are respectively divided into a plurality of target modality feature blocks and auxiliary modality feature blocks, the target modality feature blocks are input into the target modality attention branch network in the corresponding cross-attention reconstruction network, and the auxiliary modality feature blocks are input into the auxiliary modality attention branch network in the corresponding cross-attention reconstruction network; Both the target modality attention branch network and the auxiliary modality attention branch network include multiple cascaded transformer blocks; In each cross-attention reconstruction network: The input of the first-level transformer block of the target modality attention branch network includes the target modality feature block corresponding to the first-level encoder in the target modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks except the first-level transformer block in the target modality attention branch network includes the output feature block of the previous level transformer block in the same target modality attention branch network; the output feature block of the last level transformer block in the same target modality attention branch network is the target modality preliminary reconstruction feature block of the current cross-attention reconstruction network; The input of the first-level transformer block of the auxiliary modality attention branch network includes the auxiliary modality feature block corresponding to the first-level encoder in the auxiliary modality branch feature extraction network corresponding to the cross-attention reconstruction network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block includes the output feature block of the previous level transformer block in the same auxiliary modality attention branch network; the output of the last level transformer block in the same auxiliary modality attention branch network is the auxiliary modality preliminary reconstructed feature block of the current cross-attention reconstruction network.

4. The multimodal 5.0T according to claim 3 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that In each of the cross-attention reconstruction networks, The input of the first-level transformer block of the target modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the target modality attention branch network; the input of other transformer blocks in the target modality attention branch network except the first-level transformer block also includes the corresponding learnable vector; The input of the first-level transformer block of the auxiliary modality attention branch network also includes the corresponding learnable vector of the first-level transformer block of the auxiliary modality attention branch network; the input of other transformer blocks in the auxiliary modality attention branch network except the first-level transformer block also includes the corresponding learnable vector; The learnable vectors corresponding to the transformer blocks at the same level of the target modality attention branch network and the auxiliary modality attention branch network are the same; and the learnable vector of the first-level transformer block is the output vector obtained after the target modality feature block and the auxiliary modality feature block input into the cross-attention reconstruction network pass through the modality mapping module of the learnable vector network; the learnable vectors corresponding to other transformer blocks except the first-level transformer block are: the learnable vector corresponding to the previous-level transformer block, the output feature block of the previous-level transformer block in the target modality attention branch network, and the output vector of the previous-level transformer block in the auxiliary modality attention branch network after being input into the learnable vector update module of the corresponding level in the same cross-attention reconstruction network.

5. The multimodal 5.0T according to claim 4 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that In the modal mapping module, the output of the first linear mapping layer of the modal mapping module is coupled with the output of the second linear mapping layer of the modal mapping module is coupled with the output of the auxiliary modal feature block is coupled with the output of the second linear mapping layer of the modal mapping module. The coupled components are then sequentially coupled with the third linear mapping layer and the normalization layer of the modal mapping module to obtain an output vector of the modal mapping module. In each learnable vector update module, the output feature block of the previous level transformer block in the target modality attention branch network and the output feature block of the previous level transformer block in the auxiliary modality attention branch network are respectively input into the first linear mapping layer and the second linear mapping layer of the current learnable vector update module and then channel splicing is performed. The result of channel splicing passes through the third linear mapping layer and the fourth linear mapping layer of the current learnable vector update module in turn. At the same time, the learnable vector corresponding to the previous level transformer block is input into the fourth linear mapping layer of the current learnable vector update module, and then the output of the fourth linear mapping layer is input into the normalization layer of the current learnable vector update module to obtain the output vector of the current learnable vector update module.

6. The multimodal 5.0T according to claim 5 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that Each transformer block includes a position encoding layer, a linear mapping layer, a first normalization layer, an attention mechanism layer, a second normalization layer and a fully connected layer in sequence; The feature block input by the transformer block is input into the position encoding layer of the current transformer, and the learnable vector corresponding to the current transformer block is input into the attention mechanism layer of the current transformer block; The feature block input by the transformer block is sequentially encoded by the position encoding layer, linearly mapped by the linear mapping layer, and normalized by the first normalization layer in the transformer block to obtain the vector , vectors and vectors can be learned in the attention mechanism layer Concatenation as a vector and vector ,vector 、 、 The intermediate quantity z is obtained by the following calculation: ， in, Represents a vector , Represents a vector , Represents a vector , is a vector Dimensions, is the normalized exponential function, represents transpose; Intermediate quantity z and vector The results are added through residual connections and then passed through the linear mapping layer of the attention mechanism layer as the output of the attention mechanism layer. The output of the attention mechanism layer is input into the second normalization layer and the fully connected layer of the current level transformer block to obtain the output feature block of the current level transformer block.

7. The multimodal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The fusion network includes an input layer, multiple convolutional layers and an output layer; the target modality cascade image and the auxiliary modality cascade image are sequentially input into the fusion network to obtain the target modality MRI reconstructed image and the auxiliary modality MRI reconstructed image, respectively.

8. The multimodal 5.0T according to claim 1 23 Na magnetic resonance imaging artificial intelligence reconstruction method, characterized in that The fusion network includes a target modality fusion subnetwork and an auxiliary modality fusion subnetwork, and the target modality fusion subnetwork and the auxiliary modality fusion subnetwork each include an input layer, multiple convolutional layers and an output layer in sequence; the target modality cascade image is input into the target modality fusion subnetwork to obtain a target modality MRI reconstructed image; the auxiliary modality cascade image is input into the auxiliary modality fusion subnetwork to obtain an auxiliary modality MRI reconstructed image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, steps 2 to 5 of the reconstruction method according to any one of claims 1 to 8 are implemented.

Images