Transform-based multi-modal MRI reconstruction method

Abstract

The invention discloses a multi-modal MRI (Magnetic Resonance Imaging) reconstruction method based on Transform, which comprises the following steps of: inputting paired under-sampling and full-sampling MRI image pairs into a pre-trained multi-modal MRI image reconstruction network model to obtain a high-resolution image after target modal reconstruction; wherein the multi-modal MRI image reconstruction network model comprises a spatial registration module and an image reconstruction module, the spatial registration module is used for carrying out spatial position alignment on an auxiliary modal MRI image to a target modal MRI image, and the image reconstruction module is used for extracting auxiliary information from an MRI image pair; the multi-modal MRI image reconstruction method is suitable for multi-modal MRI image reconstruction, and the detail quality of the reconstructed MRI image can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a Transformer-based multimodal MRI reconstruction method. Background Art

[0002] Image reconstruction is an innovative direction in computer vision and artificial intelligence, and is mainly used in many fields such as remote sensing imaging, security monitoring and autonomous driving. Image reconstruction uses the prior knowledge of the degradation process to remove or reduce the degradation effects such as blur and noise in the image through computer processing, so that the image is restored to or close to its original true state. Image reconstruction technology relies on advanced deep learning models and algorithms, such as convolutional neural networks (CNNs) and vision transformers, to extract local and global features and generate high-quality images. Therefore, images have become a research hotspot in the field of computer vision and artificial intelligence, and are an important part of modern digital media and image content restoration systems.

[0003] Nuclear Magnetic Resonance Imaging (MRI) technology, with its high-resolution and multi-parameter imaging features, provides a more reliable basis for medical and biological research. However, compared with other imaging methods, MRI scanning takes too long and imaging is slow, usually taking 15 minutes to 1 hour. In addition, during the MRI scanning process, it is very sensitive to the physiological and body movements of the scanned person. For example, heartbeat, breathing, coughing, and swallowing will cause varying degrees of artifacts in the image, all of which limit the promotion and development of MRI.

[0004] In early studies, the reconstruction process of MRI mainly relied on parallel imaging and compressed sensing. Parallel imaging technology uses multiple coils to sample a certain part at the same time, using sensitivity information to assist spatial positioning, and compressed sensing recovers the original data from the undersampled k-space by utilizing the sparsity of the image in the total variation (TV) or wavelet transform (WT). Recent studies have shown that considering the complementary information between magnetic resonance images of different contrasts, full sampling auxiliary mode MRI can help reconstruct undersampled target mode MRI more effectively. In view of this, it is believed that the key to optimizing MRI reconstruction lies in two aspects: reconstructing anatomical details and removing tissue artifacts. Summary of the invention

[0005] The purpose of the present invention is to provide a Transformer-based multimodal MRI reconstruction method, which can improve the quality of reconstructed image details and ensure high signal-to-noise ratio and fidelity; To achieve the above functions, the present invention designs a Transformer-based multimodal MRI reconstruction method, which performs the following steps S1 to S3 to reconstruct the low-resolution target modality MRI image into a high-resolution image: Step S1: acquiring a low-resolution target modality MRI image and a corresponding high-resolution auxiliary modality MRI image; Step S2: constructing an initial multimodal MRI image reconstruction network model, forming an MRI image pair with the target modality MRI image and the auxiliary modality MRI image, and inputting the initial multimodal MRI image reconstruction network model for training, outputting the reconstructed image, and obtaining a trained multimodal MRI image reconstruction network model; The multimodal MRI image reconstruction network model includes a spatial registration module and an image reconstruction module, wherein the spatial registration module performs spatial position alignment of the auxiliary modality MRI image to the target modality MRI image, and the image reconstruction module extracts auxiliary information from the MRI image pair; Step S3: input the paired under-sampled and fully-sampled MRI image pairs into the trained multi-modal MRI image reconstruction network model, output the high-resolution image after target modality reconstruction, and complete the reconstruction of the target modality MRI image.

[0006] Beneficial effects: Compared with the prior art, the advantages of the present invention include: The present invention designs a multimodal MRI image reconstruction network model, which is cascaded and predicts the flow field from coarse to fine, and can obtain accurate registration results; The present invention improves the CSWin Transformer (Alternating Cross Window Transformer) by using alternating horizontal and vertical window attention instead of cross-window attention, thus avoiding the window direction order bias; We employ many different mask types, including Cartesian, Gaussian, and Poisson, as well as different undersampling ratios, and conduct extensive experiments and result visualization to verify the robustness of our designed model to different conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a model framework diagram of a multimodal MRI image reconstruction network model provided according to an embodiment of the present invention; Figure 2 is a schematic structural diagram of a spatial registration module provided according to an embodiment of the present invention; Figure 3 is a schematic diagram of space transformation provided according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of an image reconstruction module provided according to an embodiment of the present invention; Figure 5 is a schematic diagram of the structure of SCRWin Transformer Unet provided according to an embodiment of the present invention; Figure 6 is a schematic diagram of a CSWin Transformer provided according to an embodiment of the present invention; Figure 7 is a qualitative comparison experiment result diagram using a 50% Cartesian sampling mode provided according to an embodiment of the present invention; Figure 8 is a qualitative comparison experiment result diagram using a 50% Poisson sampling mode provided according to an embodiment of the present invention; Fig. 9 is a graph showing ablation experiment results of a spatial registration module provided in an embodiment of the present invention; Fig.10 It is a graph of CSWin Transformer ablation experiment results provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0008] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0009] The Transformer-based multimodal MRI reconstruction method provided in an embodiment of the present invention performs the following steps S1 to S3 to reconstruct a low-resolution target modality MRI image into a high-resolution image: Step S1: acquiring a low-resolution target modality MRI image and a corresponding high-resolution auxiliary modality MRI image; Step S2: constructing an initial multimodal MRI image reconstruction network model, forming an MRI image pair with the target modality MRI image and the auxiliary modality MRI image, and inputting the initial multimodal MRI image reconstruction network model for training, outputting the reconstructed image, and obtaining a trained multimodal MRI image reconstruction network model; The multimodal MRI image reconstruction network model includes a spatial registration module and an image reconstruction module, wherein the spatial registration module performs spatial position alignment of the auxiliary modality MRI image to the target modality MRI image, and the image reconstruction module extracts auxiliary information from the MRI image pair; like Figure 1As shown, the training of the multimodal MRI reconstruction network model is further introduced, and the undersampled target modality MRI image and the auxiliary modality MRI image are input into the model. After registration, the auxiliary modality MRI image and the undersampled target modality MRI image are output to the image reconstruction module. After learning and processing by the image reconstruction module, the final reconstructed image is obtained. Specifically, the auxiliary modality MRI image with clearer relevant information and the low-resolution target modality MRI image are first input into the spatial registration module in pairs to achieve spatial feature alignment between the auxiliary modality MRI image and the target modality MRI image, thereby effectively extracting the real pixel information of the target modality MRI image from the auxiliary modality MRI image.

[0010] After registration, the auxiliary modality MRI image is fused with the target modality MRI image to obtain the initial auxiliary feature. Due to the inaccuracy of this feature, the subsequent ASCRWin Transformer is required for further extraction. At the same time, in order to avoid the interference of the auxiliary modality, only the target modality MRI image is input into the USCRWin Transformer to extract the unique feature, and the auxiliary feature U and the unique feature A are added to obtain the residual image, and then the residual image is supplemented and added with the original target modality MRI image to generate the reconstructed image. The reconstructed image generated in the previous stage is regarded as the original target modality MRI image, and the auxiliary and unique features are updated in the same way as initialization to generate the residual image of this stage, which is added to the original target modality MRI image to obtain the reconstructed image of this stage.

[0011] The specific steps of step S2 are as follows: Step S2.1: The structure diagram of the spatial registration module is as follows Figure 2 The architecture of this module is designed to predict the flow field through the input image pair, perform spatial transformation on the auxiliary modality MRI image, eliminate the spatial position difference between the auxiliary modality MRI image and the target modality MRI image, and achieve alignment of the same tissue structure.

[0012] In the spatial registration module, the flow field of the auxiliary modality MRI image is predicted and then spatial transformation (SpatialTransformer, STN) is implemented to obtain the registered image; The specific steps of step S2.1 are as follows: Step S2.1.1: perform channel stitching on the target modality MRI image and the auxiliary modality MRI image to obtain a combined image, and input the combined image into the VoxelMorph network to obtain a predicted flow field; Specifically, the undersampled target modality MRI image and the auxiliary modality MRI image are input into the model, and after registration, the auxiliary modality MRI image and the undersampled target modality MRI image are input into the reconstruction module. After learning and processing by the reconstruction module, the final reconstructed image is obtained. Figure 2 In the figure, flow is the deformation field predicted by the VoxelMorph network, and warp is the result of spatial transformer registration of the moving image by the flow field. The VoxelMorph network consists of a U-Net network, in which the encoder uses maximum pooling downsampling. The combined image is input into the encoder of the U-Net network for three maximum pooling downsamplings. Each time the feature map is downsampled, the size is reduced to half of the original size, and the number of channels is increased to 32, 64, and 128. The obtained high-level feature map is input into the decoder after passing through the bottleneck layer. The decoder uses transposed convolution high-level features to upsample. Each time the feature map is upsampled, the size is increased to twice the original size, and the number of channels is reduced to half the original size. Jump connections are made between the encoder and the decoder to fuse high-level features and low-level features. Finally, a 2-channel predicted flow field is output.

[0013] Step S2.1.2: Use the predicted flow field to perform spatial transformation on the auxiliary modality MRI image, and use the spatial transformation network to perform correction to obtain the current registration image; Reference Figure 3 , the specific steps of step S2.1.2 are as follows: Step S2.1.2.1: The localization network predicts the affine transformation matrix using the image features extracted by the CNN network θ ; Step S2.1.2.2: The mesh generator is based on the affine transformation matrix θ Generate a position transformation mapping relationship before and after the transformation; Figure 3 middle Indicates the position transformation mapping relationship; Step S2.1.2.3: Sample pixel values ​​according to the position transformation mapping relationship and perform bilinear interpolation to solve the problem of decimal positions in the grid sampler, as shown in the following formula: ; ; In the formula, is the output image The pixel value of the position, ; and are the height and width of the output image, respectively. C represents the channel dimension; is the pixel value at position (n,m,c) of the combined image, , H and W are the height and width of the combined image, respectively. θ 11 ~ θ 23 represents the parameters of the affine transformation, k represents bilinear interpolation, and represents the parameters of the interpolation function, Represents the coordinates of the original image.

[0014] Step S2.1.3: Use the last registered image as the auxiliary modality MRI image, repeat steps S2.1.1 to S2.1.2, generate a flow field from coarse to fine, and obtain a registered image without position deviation.

[0015] Step S2.2: Input the registration image and the target modality MRI image into the initial multimodal MRI image reconstruction network model, pass through the image reconstruction module, output the reconstructed image, and obtain the trained multimodal MRI image reconstruction network model.

[0016] like Figure 4 As shown in FIG. 1 , the reconstruction module receives the original low-resolution target modality MRI image, the high-resolution auxiliary modality MRI image and the fusion features as all the inputs of the reconstruction module. The unique features and the auxiliary features are added to obtain the reconstructed image of this iteration. The reconstructed image is used as the new target modality MRI image to generate the next fusion features, and the iteration is repeated. Figure 4 The USCRWin Transformer Unet is used to output unique features, and the ASCRWin Transformer Unet is used to output auxiliary features. The SCRWin Transformer (Separate Cross Window Transformer) Unet architecture used to output unique features and auxiliary features in the reconstruction module is shown in the figure. Figure 5As shown, residual connections are used to preserve contextual features of the encoder and decoder to speed up convergence during training. The encoder consists of a patch embedding layer, a SCRWin Transformer, and a patch merging layer. The patch embedding layer uses a 7x7 convolution kernel, which is neutralized between the 16x16 convolution kernel used in the visual transformer and the 3x3 convolution kernel used in EarlyConv to improve the nonlinear expression ability of the model while avoiding low receptive field. With patch embedding, the input image is divided into several non-overlapping blocks of size 4x4, and the feature dimensions of the feature blocks can be projected to any dimension using an additional linear projection layer. There is a patch merging layer between every two SCRWin Transformer layers to fuse different patches in the feature map, playing a role in downsampling and expanding the feature dimension to construct feature maps of different scales. In the encoding stage, the embedding sequence is downsampled a total of three times. For each scale of the feature map, it is input into the corresponding feature map generated by the decoder using a residual connection. The decoder consists of interleaved SCRWin Transformer and patch expansion layers. The patch expansion layer upsamples the feature map while reducing the feature dimension, gradually restoring the original resolution of the image.

[0017] The specific steps of step S2.2 are as follows: Step S2.2.1: Input the registration image and the target modality MRI image into the CNN network to obtain the fusion features; Step S2.2.2: Input the fused features and the target modality MRI image into a U-Net network with Transformer as the basic component and no parameter sharing, i.e., SCRWin Transformer Unet, to predict auxiliary features and unique features; CSWin Transformer (Alternating Cross Window Transformer) proposes a cross-shaped window multi-head self-attention mechanism, such as Figure 6As shown. The attention head is divided into two parts: one part calculates the local attention of the vertical sliding window, and the other part calculates the local attention of the horizontal sliding window. The window width is adjusted according to the resolution of the feature map. For high-resolution feature maps, the window width is set smaller to consider image details, while for low-resolution feature maps, the window width is set larger to fuse more information. The present invention improves the calculation method of CSWin Transformer to obtain SCRWin Transformer (Separated Cross Window Transformer). In the original model, multiple cross-window and multi-head self-attention calculations will be performed at each stage. The method of calculating attention by dividing the attention head will only consider unidirectional information in two feature maps. To solve this problem, the present invention calculates the sliding window attention in an alternating manner of "first horizontal then vertical, then front vertical then horizontal" in each CSWin Transformer layer. Specifically, if there are 24 attentions at this time, they are divided into one group according to the first 12 attentions, and the last 12 attentions are used as the second group. In CSWin Transformer, each group of feature maps only performs attention on a single direction sliding window at all stages, while SCRWin Transformer performs direction alternating sliding window attention calculation. The alternating order of the two sets of feature map windows is opposite to minimize the model's bias towards fixed-directional windows and fixed order. Each attention head fully considers both horizontal and vertical information, which can generate more accurate weight maps compared to calculating attention only in a unidirectional window. In addition, based on the input image size of 256x256, the window width is set to 1, 2, and 8, respectively.

[0018] The formula of SCRWin Transformer is as follows: ; ; ; ; in, d k is a matrix Q and K The dimensions of the column vectors in , Q is the query matrix, V is the value matrix, K It is the content of attention, head i Representative i Head attention, and It works on Q Matrix and K Weights on the matrix, subscripti The calculation represents the i Head attention, W O is the output weight represented by the convolution kernel, QK T is a dot product operation used to calculate Q exist V The purpose of scaling with softmax is to avoid too large a dot product, because when the dot product is too large, the gradient through softmax will be very small. In addition, the advantage of softmax is that it facilitates the gradient calculation of back propagation and smoothes the result to the range of 0-1. 1...c / 2 is the first half of the channel, c / 2+1...c is the second half of the channel, stage If it is an odd number, the vertical local window attention is calculated, otherwise the horizontal window attention is calculated.

[0019] Step S2.2.3: Input the auxiliary features and unique features into the CNN network to obtain residual features; Step S2.2.4: Add the residual features and the target modality MRI image to obtain the reconstructed image; Step S2.2.5: Use the reconstructed image as the target modality MRI image, input the target modality MRI image and the registration image into the CNN network, obtain the next fusion feature, and iterate steps S2.2.1-S2.2.4; until the residual feature can restore the lost details of the under-sampled image and remove unnecessary artifacts, stop the iteration and obtain a trained multi-modal MRI image reconstruction network model.

[0020] The iterative process of step S2.2.5 is as follows: ; ; ; ; ; Among them, U is the auxiliary feature, A is the unique feature, ED A , ED U are the codecs for extracting auxiliary features and unique features, respectively. t is the number of iterations, diff is the residual graph, Rec x is the reconstructed image, Rec A is the fusion feature, is a fully sampled auxiliary modality MRI image.

[0021] Step S3: input the paired under-sampled and fully-sampled MRI image pairs into the trained multi-modal MRI image reconstruction network model, output the high-resolution image after target modality reconstruction, and complete the reconstruction of the target modality MRI image.

[0022] Figure 7-Figure 8 Qualitative experimental results of reconstruction on different Tranformer-based MRI reconstruction algorithms using the IXI brain dataset are shown.

[0023] The first set of experiments: reference Figure 7 , using a 50% Cartesian sampling pattern, the 5 baseline models including the method of the present invention, the reconstructed brain MRI images are relatively clear, and the obvious artifacts in the initial state are basically removed. MTrans can still see some residual vertical line smears due to undersampling. The details of the Task-Transformer method are mixed and look fuzzy. The edges of the SwinRN method are not sharp enough. The internal tissue edges of the DuDReTLU-net method are not clear enough. The image restored by the method of the present invention has clear edges and texture details, and the detailed tissue structure can be seen, while aliasing and artifacts are basically completely eliminated.

[0024] The second set of experiments: reference Figure 8 , using a 50% Poisson sampling pattern, the images reconstructed by the MTrans and Task-Transformer methods still have artifacts in the empty area, and both have jagged edges. The internal tissue of the SwinRN method is darker. DSFormer performs poorly in areas with denser tissue structures. DuDReTLU-net has blurred edges on some of the external contours and internal tissue structures. The method of the present invention reconstructs an image that is basically free of the above problems.

[0025] When the sampling rate is the same, the aliasing phenomenon of the image generated by Poisson sampling is not as serious as that of Cartesian sampling. The reason is k The center of space contains important imaging information. k If more information about the center of space is lost, the collected images will also have more ghosting.

[0026] In order to quantify the comparison results, the present invention uses PSNR, SSIM and RMSE as indicators to measure the quality of the reconstructed image. Different tranformer-based MRI reconstruction models were tested using the IXI brain dataset, and all models were tested under the same conditions. The sampling mask is Cartesian and Poisson mode, the sampling rate is set to 50%, and the acceleration factor AF=2. The experimental results are shown in Table 1:

[0027] The PSNR, SSIM and RMSE obtained by the method of the present invention are all optimal. PSNR is the peak signal-to-noise ratio. A high PSNR indicates that the difference between the reconstructed image and the original image is very small, the image details and color restoration are relatively good, the distortion is low, and the visual effect is close to the original image; SSIM takes into account the changes in brightness, contrast and structure. A high SSIM indicates that the structural features of the reconstructed image and the original image are similar, and even if there is a slight color or texture difference, it is difficult for the human eye to perceive obvious distortion; RMSE is the root mean square difference between the actual value and the predicted value. The lower the value, the smaller the gap between the predicted value and the true value.

[0028] In order to fully illustrate the effectiveness of the method of the present invention, the present invention carried out ablation experiments on the spatial registration module and CSWinTransformer (alternating cross window Transformer). Specifically, in order to study the impact of image registration on the reconstruction algorithm, the present invention conducted experiments by deleting the spatial registration module while keeping the rest of the original model settings unchanged. Compared with the original model and other models with retained spatial registration modules, the multimodal MRI image reconstruction model without the spatial registration module has the worst performance. Without the spatial registration module, the MRI images lack spatial alignment, which makes it difficult for the image reconstruction module to extract effective complementary features. This not only increases the learning burden of the image reconstruction module, but also leads to poor reconstruction quality. The experimental results are shown in Fig. 9 shown.

[0029] Similarly, using the Cartesian and Poisson sampling modes, the model without the introduction of the spatial registration module is compared with the model with the spatial registration module. The edges of the reconstructed image are blurred, the anatomical tissue is not fine enough, and the lines of the complex parts are slightly distorted. The image reconstructed by the model with the spatial registration module is almost exactly the same as the real image. From the residual graph between the two, it can be seen that the reconstructed image perfectly restores the high-resolution original image.

[0030] The present invention also conducted an ablation experiment on CSWin Transformer, and the experimental results are as follows: Fig.10As shown. In order to verify the effectiveness of the SCRWin Transformer proposed in the present invention, CSWinTransformer was used to replace the SCRWinTransformer, while the rest of the settings of the original model were retained for experiments. The results show that the SCRWinTransformer proposed in the present invention can indeed improve the performance of the model to a certain extent. Compared with the model based on CSWin Transformer, the PSNR and SSIM of the IXI brain dataset are improved by 1.31% and 1.02%, respectively. SCRWinTransformer can achieve more stable loss reduction, which helps the model to converge quickly and optimize performance, improve generalization ability, and ensure reliable model evaluation, so that the model can learn data features more effectively and perform well on new data. Fig.10 The bar graphs in clearly show the difference between PSNR and SSIM reconstructed using SCRWin Transformer and CSWin Transformer.

[0031] In summary, the present invention proposes a two-stage end-to-end model. The first stage is the registration stage, in which the auxiliary modality MRI image is aligned with the target modality MRI image to eliminate the spatial position difference between the two. The second stage is the reconstruction stage, in which a reconstruction model is designed based on the differential image. For the undersampled target modality MRI images, the residual images between them and the underlying high-resolution magnetic resonance images are modeled. The residual images are modeled as unique and auxiliary features extracted from the input target image and the fused feature map, respectively. This ensures the accuracy of feature extraction and the accuracy of the consistent information extracted from the auxiliary modality MRI images.

[0032] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A Transformer-based multimodal MRI reconstruction method, characterized in that: Perform the following steps S1 to S3 to reconstruct the low-resolution target modality MRI image into a high-resolution image: Step S1: acquiring a low-resolution target modality MRI image and a corresponding high-resolution auxiliary modality MRI image; Step S2: constructing an initial multimodal MRI image reconstruction network model, forming an MRI image pair with the target modality MRI image and the auxiliary modality MRI image, and inputting the initial multimodal MRI image reconstruction network model for training, outputting the reconstructed image, and obtaining a trained multimodal MRI image reconstruction network model; The multimodal MRI image reconstruction network model includes a spatial registration module and an image reconstruction module, wherein the spatial registration module performs spatial position alignment of the auxiliary modality MRI image to the target modality MRI image, and the image reconstruction module extracts auxiliary information from the MRI image pair; Step S3: input the paired under-sampled and fully-sampled MRI image pairs into the trained multi-modal MRI image reconstruction network model, output the high-resolution image after target modality reconstruction, and complete the reconstruction of the target modality MRI image.

2. The Transformer-based multimodal MRI reconstruction method according to claim 1, characterized in that: The specific steps of step S2 are as follows: Step S2.1: In the spatial registration module, the auxiliary modality MRI image is subjected to flow field prediction and then spatial transformation to obtain a registered image; Step S2.2: Input the registration image and the target modality MRI image into the initial multimodal MRI image reconstruction network model, pass through the image reconstruction module, output the reconstructed image, and obtain the trained multimodal MRI image reconstruction network model.

3. The Transformer-based multimodal MRI reconstruction method according to claim 2, characterized in that: The specific steps of step S2.1 are as follows: Step S2.1.1: perform channel stitching on the target modality MRI image and the auxiliary modality MRI image to obtain a combined image, and input the combined image into the VoxelMorph network to obtain a predicted flow field; Step S2.1.2: spatially transform the auxiliary modality MRI image using the predicted flow field to obtain the current registration image; Step S2.1.3: Use the last registered image as the auxiliary modality MRI image, repeat steps S2.1.1-S2.1.2 to obtain the registered image.

4. The Transformer-based multimodal MRI reconstruction method according to claim 3, characterized in that: The specific method of step S2.1.1 is as follows: The combined image is input into the encoder of the U-Net network for three maximum pooling downsamplings. Each time the feature map is downsampled, the size is reduced to half of the original size, and the number of channels is increased to twice the original size. The obtained high-level feature map passes through the bottleneck layer and is input into the decoder. The decoder uses the transposed convolution high-level feature pairs for upsampling. Each time the feature map is upsampled, the size is increased to twice the original size, and the number of channels is reduced to half the original size. Finally, a 2-channel predicted flow field is output.

5. The Transformer-based multimodal MRI reconstruction method according to claim 3, characterized in that: The specific steps of step S2.1.2 are as follows: Step S2.1.2.1: The localization network predicts the affine transformation matrix using the image features extracted by the CNN network θ ; Step S2.1.2.2: The mesh generator is based on the affine transformation matrix θ Generate a position transformation mapping relationship before and after the transformation; Step S2.1.2.3: Sample pixel values ​​according to the position transformation mapping relationship and perform bilinear interpolation, as shown in the following formula: ； ； In the formula, is the output image The pixel value of the position, ; and are the height and width of the output image, respectively. C represents the channel dimension; is the pixel value at position (n,m,c) of the combined image, , H and W are the height and width of the combined image, respectively. θ 11 ~ θ 23 represents the parameters of the affine transformation, k represents bilinear interpolation, and represents the parameters of the interpolation function, Represents the coordinates of the original image.

6. The Transformer-based multimodal MRI reconstruction method according to claim 2, characterized in that: The specific steps of step S2.2 are as follows: Step S2.2.1: Input the registration image and the target modality MRI image into the CNN network to obtain the fusion features; Step S2.2.2: Input the fused features and the target modality MRI image into a U-Net network with Transformer as the basic component and no parameter sharing to predict auxiliary features and unique features; Step S2.2.3: Input the auxiliary features and unique features into the CNN network to obtain residual features; Step S2.2.4: Add the residual features and the target modality MRI image to obtain the reconstructed image; Step S2.2.5: Use the reconstructed image as the target modality MRI image, input the target modality MRI image and the registration image into the CNN network, obtain the next fusion feature, and iterate steps S2.2.1-S2.2.4; until the residual feature can restore the lost details of the under-sampled image and remove unnecessary artifacts, stop the iteration and obtain a trained multi-modal MRI image reconstruction network model.

7. The Transformer-based multimodal MRI reconstruction method according to claim 6, characterized in that: The specific method of step S2.2.2 is as follows: The U-Net network predicts auxiliary features and unique features based on cross-window multi-head attention, as shown in the following formula: ； ； ； ； in, d k is a matrix Q and K The dimensions of the column vectors in , Q is the query matrix, V is the value matrix, K It is the content of attention, head i Representative i Head attention, and It works on Q Matrix and K The weight on the matrix, the subscript i represents the calculation of the first i Head attention, W O is the output weight represented by the convolution kernel, QK T Used for calculation Q exist V The attention weights on , 1…c / 2 is the first half of the channel, c / 2+1…c is the second half of the channel, stage If it is an odd number, the vertical local window attention is calculated, otherwise the horizontal window attention is calculated.

8. The Transformer-based multimodal MRI reconstruction method according to claim 6, characterized in that: The iterative process of step S2.2.5 is as follows: ； ； ； ； ； Among them, U is the auxiliary feature, A is the unique feature, ED A , ED U are the codecs for extracting auxiliary features and unique features, respectively. t is the number of iterations, diff is the residual graph, Rec x is the reconstructed image, Rec A is the fusion feature, is a fully sampled auxiliary modality MRI image.

Images