A three-dimensional multimodal magnetic resonance image reconstruction method

Abstract

The present invention provides a three-dimensional multimodal nuclear magnetic resonance image reconstruction method, including: step 1, processing the data set, selecting the auxiliary mode and the target mode, further downsampling the target mode and retaining the true value; step 2, establishing a multimodal encoder as a feature extraction module, for extracting and interacting the effective features of multimodal data; step 3, establishing a decoding module based on multi-scale feature fusion, for cross-domain linking of effective features in the encoding process and layer-by-layer decoding of target modality features; step 4, performing network training based on supervised learning to obtain a trained 3DMRI reconstruction model, thereby meeting the needs of accelerating nuclear magnetic resonance imaging. Through the method of the present invention, multiple modal features and depth information can be better extracted and fused, greatly improving the accuracy of model reconstruction images.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging, and in particular relates to a three-dimensional multimodal magnetic resonance image reconstruction method. Background Art

[0002] Magnetic resonance imaging (MRI) has the advantages of no radiation, high resolution, and multi-scale. It is widely used in medical imaging and has become an important clinical medical examination method, providing very valuable information for clinical imaging diagnosis. However, compared with other imaging methods, MRI requires a longer scanning time and slow imaging, which can cause discomfort to patients. In addition, the patient's organ movement, such as breathing, swallowing, and natural heartbeat, will cause image blur and contrast distortion, which cannot meet the requirements of real-time dynamic high-precision imaging, affecting the further promotion and development of MRI.

[0003] MRI reconstruction technology can be used to reduce scanning time, improve spatial resolution and imaging quality, and also reduce patient discomfort. At present, with the development and promotion of deep learning technology, reconstruction methods based on deep learning have achieved good results in this task. Existing MRI reconstruction technology often uses deep convolutional neural networks to learn the feature information of multiple modalities, and uses this information to help the network learn the reconstruction relationship of the target modality.

[0004] Some current multimodal MRI reconstruction methods based on deep learning have achieved advanced results in multiple reconstruction tasks, but there are still some shortcomings:

[0005] Existing multimodal MRI reconstruction methods can often only select one auxiliary modality to assist the reconstruction of the target modality, and do not fully utilize other possible effective modalities. The information of these auxiliary modalities is crucial for image reconstruction. They can provide more complete and accurate image contour and detail information. Ignoring this multimodal information will greatly affect the performance of reconstruction.

[0006] Existing reconstruction methods often learn the reconstruction mapping relationship of a single image, that is, 2D MRI reconstruction, while ignoring the three-dimensional depth information. This depth information often contains more details and spatial relationships than a single image, and can provide the model with more complete and accurate spatial dimension information, thereby improving the accuracy and robustness of reconstruction;

[0007] By utilizing these neglected multimodal and deep information, the model can be better helped to learn to reconstruct relationships and the performance of existing methods can be greatly improved, thereby ensuring high-precision and stable reconstruction performance in various complex tasks. Summary of the invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide a three-dimensional multimodal nuclear magnetic resonance image reconstruction method in view of the shortcomings of the prior art, which greatly improves the shortcomings of the prior art and greatly improves the difficulty of MRI imaging in clinical practice. The method of the present invention comprises the following steps:

[0009] Step 1, processing the data, selecting multi-modality, further under-sampling the target modality and retaining the true value; the multi-modality includes an auxiliary modality and a target modality;

[0010] Step 2: Use the multimodal encoder as a feature extraction module to extract and interact effective features of multimodal data: data of multiple different modalities have different feature spaces. During the encoding process, a multimodal interaction module based on the attention mechanism is used to interact the effective features in the multimodal data to obtain the final features of the target modality.

[0011] Step 3: Establish a decoding module based on multi-scale feature fusion to link the effective features across domains and decode the target modality features layer by layer in the encoding process: the data processed in step 1 is used as the input of the multi-modal encoder, and the final features of the target modality obtained in step 2 are used as the input of the decoding module. The features obtained step by step in the encoding process of multiple modalities are used for cross-domain linking after multi-scale fusion to assist the decoding and reconstruction of the target modality.

[0012] Step 4: Perform network training based on supervised learning: The multimodal encoder and decoding modules established in steps 2 and 3 constitute a multimodal MRI reconstruction model based on multi-scale feature fusion and attention mechanism. A supervised learning strategy is adopted to train the multimodal MRI reconstruction model using the data processed in step 1 and the true value, so as to reconstruct the target MRI image.

[0013] Step 1 includes:

[0014] Step 1-1, preprocess the 3D MRI image data, eliminate the minimum and maximum outliers in the image data, normalize the grayscale value to 0-255, and crop the useless black edges at the edge of the image to retain the effective part;

[0015] Step 1-2, select n MRI imaging modalities as the used data, denoted as , It represents the nth modality in MRI imaging, and selects one of them as the target modality, and the remaining modalities as auxiliary modalities to assist in the reconstruction of the target modality. Then, the data of the target modality is undersampled while backing up a copy of the true value, and the undersampled data and the data of the other two modalities are used as model training data.

[0016] Step 2 includes:

[0017] Step 2-1, the image encoding process is the process of feature extraction. In the encoding, the data of each modality is downsampled step by step. First, the downsampling module composed of the convolutional neural network CNN is used to reduce the dimension and extract features. The formula is:

[0018]

[0019] in represents the characteristics of the m-mode input at the t-th downsampling, Represents the characteristics of the m-mode obtained after the t-th downsampling, m is One of them, is the initial input data of the model, , They represent the learnable weights and bias terms of the convolutional layer when the data of the m-mode is downsampled for the tth time, Relu activation function (linear rectification function);

[0020] Step 2-2, the multimodal interaction module based on the attention mechanism performs the following steps: interacting with the data of n modalities after each downsampling: first, concatenating the data of n modalities in the channel dimension:

[0021]

[0022] in Represents a splicing operation, Indicates intermediate parameters; Represents the characteristics of the nth mode obtained after the tth downsampling;

[0023] Then Input to the designed attention interaction module, which is a convolutional block attention module CBAM:

[0024]

[0025] in represents the channel attention map, represents the spatial attention map, represents element-wise multiplication, represents the intermediate parameter, Represents the output of the attention interaction module;

[0026] The channel attention is calculated as follows:

[0027]

[0028] in represents the sigmoid activation function, express After average pooling As a result, express After max pooling As a result, MLP represents the linear layer; and Represents the linear layer at the t-th multimodal interaction Two learnable weights, note that they share the input and ;

[0029] The spatial attention is calculated as follows:

[0030]

[0031] in Indicates that the spatial attention is calculated using a convolutional layer with a convolution kernel size of 7×7 during the t-th multimodal interaction. and Represents the learnable weights and bias items of the convolution layer with a convolution kernel size of 7×7. After average pooling and maximum pooling, the results of average pooling and maximum pooling are concatenated along the channel dimension and then convolved.

[0032] The final output is equally divided into three feature maps along the channel dimension, corresponding to the previous splicing operation, the formula is:

[0033] ;

[0034] in Indicates equal splitting along the channel dimension, Represents the final output of the nth mode of the tth multimodal interaction;

[0035] Step 2-3: Connect the multimodal interaction module to the downsampling module in the form of a residual block. The input of each downsampling is composed of the output of the previous downsampling block and the output of the multimodal interaction module:

[0036]

[0037] in, represents the m-modal result obtained by the t-th multimodal interaction in step 2-2;

[0038] Step 2-4, Step 2-1 to Step 2-3 constitute the multimodal encoder described in Step 2, acting as a feature extraction module; if there are n downsamplings in total, the final feature of the target modality is obtained , represents the target mode, which is One of them.

[0039] Step 3 includes:

[0040] Step 3-1, design an upsampling block to upsample the features of the target modality finally extracted by the multimodal encoder in step 2. The upsampling block includes a convolution layer and a deconvolution layer, and the formula is:

[0041]

[0042] in Represents the input of the tth upsampling. If there are n upsampling and downsampling, the input of the first upsampling is The output of the last downsampling ,Right now ; represents the convolutional layer, and denote the learnable weights and bias terms of the convolutional layer in the tth upsampling block, respectively. represents the deconvolution layer, and They represent the learnable weights and bias terms of the deconvolution layer in the tth upsampling block respectively; Represents the result obtained after the tth upsampling;

[0043] Step 3-2: For the multimodal features obtained in the decoding process, they are first concatenated along the channel dimension before cross-domain linking:

[0044]

[0045] in express The features obtained after the modality is downsampled for the n+1-tth time; after the tth upsampling, the features after the n+1-tth downsampling are used for cross-domain linking; Indicates intermediate parameters;

[0046] Step 3-3, design a cross-domain link module based on multi-scale feature fusion. After each upsampling, the multimodal features obtained in the encoding process are fused and cross-domain linked to the encoding process to provide sufficient feature information for upsampling: For the spliced ​​multimodal features, three convolution layers with kernel sizes of 1*1, 3*3, and 5*5 are used to capture auxiliary information, and then spliced ​​with the upsampled feature map to obtain multi-scale features. The formula is:

[0047]

[0048] in represents the feature map obtained after the tth upsampling, represents the multi-scale features extracted from the t-th cross-domain link module; , , Indicates convolutional layers with kernel sizes of 1*1, 3*3, and 5*5 respectively;

[0049] For the multi-contrast features extracted by fusion, the sigmoid activation function is used to learn the normalized feature map to enhance the feature representation. Finally, a small residual module is used to integrate the multi-scale features with the enhanced feature representation:

[0050]

[0051] in represents element-wise multiplication, represents the final output of the cross-domain link module based on multi-scale feature fusion, Represents different multi-scale features Use a convolution layer with a convolution kernel size of 3*3. ,It should be noted that this formula uses three convolutional layers for three features to learn the normalized feature maps, and finally adds the three feature maps element by element;

[0052] Step 3-4, for the t+1th upsampled input, that is, the output of the tth multi-scale feature fusion module for:

[0053]

[0054] Step 3-5, after the last upsampling, no cross-domain link is used, but a convolution layer is followed, and the sigmoid activation function is used on the final feature map and mapped to a grayscale value of 0~255:

[0055]

[0056] in represents the final reconstructed target modality image, Represents the feature map obtained after the nth upsampling.

[0057] Step 4 includes:

[0058] Step 4-1, establish the mean absolute error loss function ;

[0059] Step 4-2, establish the structural similarity (SSIM) loss function ;

[0060] Step 4-3, establish a complete loss function ;

[0061] Step 4-4, using the loss function established in step 4-3, train the multimodal MRI reconstruction model through supervised training.

[0062] In step 4-1, the mean absolute error loss function The formula is:

[0063]

[0064] in, Represents the final reconstructed target modality image The index of the pixel in , express The predicted value of the location model, express The true value of the label at the position, Represents the final reconstructed target modality image The total number of pixels in .

[0065] In step 4-2, the structural similarity (SSIM) loss function The formula is:

[0066]

[0067] in represents the structural similarity function; Represent the pixel average of the output image and the pixel average of the true value, respectively. Represent the pixel value variance of the output image and the pixel value variance of the true value, respectively. represents the pixel value covariance of the output image and the true value, , is a constant used for stabilization, , is the dynamic range of pixel values, and is a set constant, , .

[0068] In step 4-3, the complete loss function The formula is:

[0069]

[0070] in, Represents the weight of the loss function. .

[0071] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0072] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0073] Compared with the prior art, the present invention has the following beneficial effects: when performing MRI reconstruction, the attention interaction module is used to interact with multimodal information; the network is trained with three-dimensional data, and the multi-scale feature fusion module is used to fuse multimodal and depth information, which provides more effective information for the model to learn the reconstruction relationship, makes up for the shortcomings of the existing methods, and ultimately effectively improves the reconstruction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of the method of the present invention.

[0075] Figure 2 It is a schematic diagram of the multimodal attention interaction module proposed in the present invention.

[0076] Figure 3 It is a schematic diagram of the multi-scale feature fusion module proposed in the present invention.

[0077] Figure 4 It is a simplified schematic diagram of the effect of using the model proposed in the present invention in the brain MRI reconstruction task. DETAILED DESCRIPTION

[0078] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0079] Reference Figure 1 In a specific embodiment of the present invention, a 3DMRI reconstruction method based on multi-scale feature fusion and attention mechanism is provided, comprising the following steps:

[0080] Step 1: Taking brain MRI reconstruction as an example, the data set is processed, T1 and T2 modes are selected as auxiliary modes, and Flair mode is selected as the target mode. The target mode is further undersampled and the true value is retained. T1, T2, and Flair modes are all common modes in brain MRI.

[0081] Step 2: Use the multimodal encoder as a feature extraction module to extract and interact effective features of multimodal data: data of multiple different modalities have different feature spaces. In the encoding process, a multimodal interaction module based on the attention mechanism is used to interact effective features in multimodal data.

[0082] Step 3: Establish a decoding module based on multi-scale feature fusion to link the effective features across domains and decode the target modality features layer by layer in the encoding process: the data processed in step 1 is used as the input of the multi-modal encoder, and the final features of the target modality obtained in step 2 are used as the input of the decoding module. The features obtained step by step in the encoding process of multiple modalities are used for cross-domain linking after multi-scale fusion to assist the decoding and reconstruction of the target modality.

[0083] Step 4: Perform network training based on supervised learning: The multimodal encoder and decoding modules established in steps 2 and 3 constitute a multimodal MRI reconstruction model based on multi-scale feature fusion and attention mechanism. A supervised learning strategy is adopted to train the multimodal MRI reconstruction model using the data processed in step 1 and the true value, so as to reconstruct the target MRI image.

[0084] Further, step 1 specifically includes:

[0085] Step 1-1, preprocess the brain 3D MRI image data, eliminate the minimum and maximum outliers in the image data, normalize the grayscale value to 0-255, and crop the useless black edges at the edge of the image to retain the effective part;

[0086] Step 1-2: select brain MRI data of T1 and T2 modalities as auxiliary modalities to assist reconstruction of Flair modality, which is more difficult to image in clinical practice. Then, undersample the data of Flair modality while backing up a copy of the true value, and use the undersampled data and the data of the other two modalities as model training data;

[0087] Further, step 2 specifically includes:

[0088] like Figure 2 As shown in the figure, this is a schematic diagram of the multimodal attention interaction module proposed in this method.

[0089] Step 2-1: The image encoding process is essentially a feature extraction process. In the encoding process, the data of each modality is downsampled step by step. First, the downsampling module composed of the convolutional neural network CNN is used to reduce the dimension and extract features. The formula is:

[0090]

[0091] in represents the characteristics of the m-mode input at the t-th downsampling, Represents the m-mode feature obtained after the t-th downsampling, where m is one of T1, T2, and Flair. is the initial input data of the model, , They represent the learnable weights and bias terms of the convolutional layer when the data of the m-mode is downsampled for the tth time, Represents the Relu activation function;

[0092] Step 2-2, the multimodal interaction module based on the attention mechanism performs the following steps: interact with the data of the three modes of T1, T2, and Flair after each downsampling: first, splice the data of the three modes of T1, T2, and Flair in the channel dimension:

[0093]

[0094] in Represents a splicing operation, Indicates intermediate parameters;

[0095] Then Input to the designed attention interaction module, which is a convolutional block attention module CBAM:

[0096]

[0097] in represents the channel attention map, represents the spatial attention map, represents element-wise multiplication, represents the intermediate parameter, Represents the output of the attention interaction module;

[0098] The channel attention is calculated as follows:

[0099]

[0100] in represents the sigmoid activation function, express After average pooling As a result, express After max pooling As a result, MLP represents the linear layer; and Represents the linear layer at the t-th multimodal interaction Two learnable weights, note that they share the input and ;

[0101] The spatial attention is calculated as follows:

[0102]

[0103] in Indicates that the spatial attention is calculated using a convolutional layer with a convolution kernel size of 7×7 during the t-th multimodal interaction. and Represents the learnable weights and bias items of the convolution layer with a convolution kernel size of 7×7. After average pooling and maximum pooling, the results of average pooling and maximum pooling are concatenated along the channel dimension and then convolved.

[0104] The final output is equally divided into three feature maps along the channel dimension, corresponding to the previous splicing operation, the formula is:

[0105] ;

[0106] in Indicates equal splitting along the channel dimension, , , They represent the final outputs of the corresponding T1, T2, and Flair modes of the t-th feature interaction respectively;

[0107] Step 2-3: Connect the multimodal interaction module to the downsampling module in the form of a residual block. The input of each downsampling is composed of the output of the previous downsampling block and the output of the multimodal interaction module:

[0108]

[0109] in, represents the m-modal result obtained by the t-th multimodal interaction in step 2-2;

[0110] Step 2-4, Step 2-1 to Step 2-3 constitute the multimodal encoder described in Step 2, acting as a feature extraction module; if there are n downsamplings in total, the final feature of the target modality is obtained , Indicates the target mode, here is Flair;

[0111] Further, step 3 specifically includes:

[0112] like Figure 3 As shown in FIG. 1 , this figure is a schematic diagram of the multi-scale feature fusion module proposed in the present invention.

[0113] Step 3-1, design an upsampling block to upsample the features of the target modality finally extracted by the multimodal encoder in step 2. The upsampling block includes a convolution layer and a deconvolution layer to better learn and use effective features. The formula is:

[0114]

[0115] in Represents the input of the tth upsampling. If there are n upsampling and downsampling, the input of the first upsampling is The output of the last downsampling ,Right now ; Indicates the target mode, which is one of T1, T2, and Flair; represents the convolutional layer, and denote the learnable weights and bias terms of the convolutional layer in the tth upsampling block, respectively. represents the deconvolution layer, and They represent the learnable weights and bias terms of the deconvolution layer in the tth upsampling block respectively; Represents the result obtained after the tth upsampling;

[0116] Step 3-2: For the multimodal features obtained in the decoding process, they are first concatenated along the channel dimension before cross-domain linking:

[0117]

[0118] in It represents the features obtained after the m-mode is downsampled for the n+1-tth time. It should be noted that after the t-th upsampling, the features after the n+1-t-th downsampling are used for cross-domain linking; Indicates intermediate parameters;

[0119] Step 3-3, design a cross-domain link module based on multi-scale feature fusion. After each upsampling, the multimodal features obtained in the encoding process are fused and cross-domain linked to the encoding process to provide sufficient feature information for upsampling: For the spliced ​​multimodal features, three convolutional layers with kernel sizes of 1*1, 3*3, and 5*5 are used to capture their auxiliary information, and then spliced ​​with the upsampled feature map to obtain multi-scale features. The formula is:

[0120]

[0121] in represents the feature map obtained after the tth upsampling, represents the multi-scale features extracted from the t-th cross-domain link module; , , Indicates convolutional layers with kernel sizes of 1*1, 3*3, and 5*5 respectively;

[0122] For the multi-contrast features extracted by fusion, the sigmoid activation function is used to learn the normalized feature map to enhance the feature representation. Finally, a small residual module is used to integrate the multi-scale features with the enhanced feature representation:

[0123]

[0124] in represents element-wise multiplication, represents the final output of the cross-domain link module based on multi-scale feature fusion, Represents different multi-scale features Use a convolution layer with a convolution kernel size of 3*3. ,It should be noted that this formula uses three convolutional layers for three features to learn the normalized feature maps, and finally adds the three feature maps element by element;

[0125] Step 3-4, for the t+1th upsampled input, that is, the output of the tth multi-scale feature fusion module for:

[0126]

[0127] Step 3-5, after the last upsampling, no cross-domain link is used, but a convolution layer is followed, and the sigmoid activation function is used on the final feature map and mapped to a grayscale value of 0~255:

[0128]

[0129] in represents the final reconstructed target modality image, Represents the feature map obtained after the nth upsampling;

[0130] Further, step 4 specifically includes:

[0131] Step 4-1, establish the mean absolute error loss function , the formula is:

[0132]

[0133] in, Represents the final reconstructed target modality image The index of the pixel in , express The predicted value of the location model, express The true value of the label at the position, Represents the final reconstructed target modality image The total number of pixels in ;

[0134] Step 4-2, establish the structural similarity (SSIM) loss function , the formula is:

[0135]

[0136] in represents the structural similarity function; Represent the pixel average of the output image and the pixel average of the true value, respectively. Represent the pixel value variance of the output image and the pixel value variance of the true value, respectively. represents the pixel value covariance of the output image and the true value, , is a constant used for stabilization, , is the dynamic range of pixel values, and is a set constant, , ;

[0137] Step 4-3, establish a complete loss function , the formula is:

[0138]

[0139] in, represents the weight of the loss function, ;

[0140] Step 4-4, using the loss function established in step 4-3, train the multimodal MRI reconstruction model through supervised training. After the training is completed, the model can be used for brain MRI reconstruction tasks, using T1 and T2 image data to assist in the reconstruction of Flair images, which greatly improves the speed of Flair image imaging and effectively eliminates artifacts;

[0141] like Figure 4 The figure shows a simplified schematic diagram of the effect of using the model in the brain MRI reconstruction task.

[0142] The present invention provides a three-dimensional multimodal nuclear magnetic resonance image reconstruction method. There are many methods and ways to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A three-dimensional multimodal magnetic resonance image reconstruction method, characterized in that: The following steps are involved: Step 1, processing the data, selecting multi-modality, further under-sampling the target modality and retaining the true value; the multi-modality includes an auxiliary modality and a target modality; Step 2: Use the multimodal encoder as a feature extraction module to extract and interact with effective features of multimodal data. During the encoding process, use a multimodal interaction module based on the attention mechanism to interact with the effective features in the multimodal data to obtain the final features of the target modality. Step 3: Establish a decoding module based on multi-scale feature fusion to link the effective features across domains and decode the target modality features layer by layer in the encoding process: the data processed in step 1 is used as the input of the multi-modal encoder, and the final features of the target modality obtained in step 2 are used as the input of the decoding module. The features obtained step by step in the encoding process of multiple modalities are used for cross-domain linking after multi-scale fusion to assist the decoding and reconstruction of the target modality. Step 4, performing network training based on supervised learning: the multimodal encoder and decoding modules established in steps 2 to 3 constitute a multimodal MRI reconstruction model based on multi-scale feature fusion and attention mechanism. A supervised learning strategy is adopted to train the multimodal MRI reconstruction model using the data processed in step 1 and the true value, so as to reconstruct the target MRI image; Step 1 includes: Step 1-1, preprocessing the 3D MRI image data, eliminating the minimum and maximum outliers in the image data, normalizing the grayscale value to 0-255, and cropping the useless black edges at the edge of the image to retain the effective part; Step 1-2, select n MRI imaging modalities as the used data, denoted as m1, m2, ..., m n , m n Represents the nth modality in MRI imaging, and selects one of them as the target modality, and the remaining modalities as auxiliary modalities to assist in the reconstruction of the target modality. Then, the data of the target modality is undersampled with a backup of the true value, and the undersampled data and the data of the other two modalities are used as model training data; Step 2 includes: Step 2-1, the image encoding process is the process of feature extraction. In the encoding, the data of each modality is downsampled step by step. First, the downsampling module composed of the convolutional neural network CNN is used to reduce the dimension and extract features. The formula is: in represents the characteristics of the m-mode input at the t-th downsampling, represents the features of the m-mode obtained after the t-th downsampling, where m is m1, m2, ..., m n One of them, is the initial input data of the model, They represent the learnable weights and bias terms of the convolutional layer when the data of the m-mode is downsampled for the tth time, and Relu represents the Relu activation function (linear rectification function). Step 2-2, the multimodal interaction module based on the attention mechanism performs the following steps: interacting with the data of n modalities after each downsampling: first, concatenating the data of n modalities in the channel dimension: Concate represents the concatenation operation, s t Indicates intermediate parameters; Represents the characteristics of the nth mode obtained after the tth downsampling; Then, s t Input to the designed attention interaction module, which is a convolutional block attention module CBAM: Among them, M c represents the channel attention map, M s represents the spatial attention map, Represents element-by-element multiplication, s t ′ represents the intermediate parameter, s t ″ represents the output of the attention interaction module; The channel attention is calculated as follows: Where σ represents the sigmoid activation function, Indicates t After the average pooling AvgPool result, Indicates t After the maximum pooling MaxPool, MLP represents the linear layer; and represents the two learnable weights of the linear layer MLP at the t-th multimodal interaction; The spatial attention is calculated as follows: in Indicates that the spatial attention is calculated using a convolutional layer with a convolution kernel size of 7×7 during the t-th multimodal interaction. and Represents the learnable weights and bias items of the convolution layer with a convolution kernel size of 7×7. After average pooling and maximum pooling, the results of average pooling and maximum pooling are concatenated along the channel dimension and then convolved. The final output is equally divided into three feature maps along the channel dimension, corresponding to the previous splicing operation, the formula is: Where Split means equal division along the channel dimension. Represents the final output of the nth mode of the tth multi-model interaction; Step 2-3: Connect the multimodal interaction module to the downsampling module in the form of a residual block. The input of each downsampling is composed of the output of the previous downsampling block and the output of the multimodal interaction module: in, represents the m-modal result obtained by the t-th multimodal interaction in step 2-2; Step 2-4, Step 2-1 to Step 2-3 constitute the multimodal encoder described in Step 2, acting as a feature extraction module; if there are n downsamplings in total, the final feature of the target modality is obtained tar represents the target mode, which is m1, m2, ..., m n One of them.

2. The method according to claim 1, characterized in that Step 3 includes: Step 3-1, design an upsampling block to upsample the features of the target modality finally extracted by the multimodal encoder in step 2. The upsampling block includes a convolution layer and a deconvolution layer, and the formula is: where y t represents the input of the tth upsampling. If there are n upsampling and downsampling, the input y of the first upsampling is 1 The output of the last downsampling Right now Conv represents the convolutional layer. and They represent the learnable weights and bias items of the convolutional layer in the tth upsampling block, respectively. Deconv represents the deconvolution layer. and They represent the learnable weights and bias terms of the deconvolution layer in the tth upsampling block respectively; u t Represents the result obtained after the tth upsampling; Step 3-2: For the multimodal features obtained in the decoding process, they are first concatenated along the channel dimension before cross-domain linking: in represents m1, m2, ..., m n The features obtained after the modality is downsampled for the n+1-tth time; after the tth upsampling, the features after the n+1-tth downsampling are used for cross-domain linking; x n+1-t Indicates intermediate parameters; Step 3-3, design a cross-domain link module based on multi-scale feature fusion. After each upsampling, the multimodal features obtained in the encoding process are fused and cross-domain linked to the encoding process to provide feature information for upsampling: For the spliced ​​multimodal features, three convolution layers with kernel sizes of 1*1, 3*3, and 5*5 are used to capture auxiliary information, and then spliced ​​with the upsampled feature map to obtain multi-scale features. The formula is: where u t represents the feature map obtained after the tth upsampling, Represents the multi-scale features extracted from the t-th cross-domain link module; Conv 1×1 、Conv 3×3 、Conv 5×5 Indicates convolutional layers with kernel sizes of 1*1, 3*3, and 5*5 respectively; For the multi-contrast features extracted by fusion, the sigmoid activation function is used to learn the normalized feature map to enhance the feature representation, and finally a residual module is used to integrate the multi-scale features with the enhanced feature representation: Where ⊙ represents element-by-element multiplication, F t represents the final output of the cross-domain link module based on multi-scale feature fusion, Represents different multi-scale features F i t Use a convolutional layer with a kernel size of 3*3, i=1,2,3; Step 3-4, for the t+1th upsampled input, that is, the output y of the tth multi-scale feature fusion module t+1 for: y t+1 =F t Step 3-5, after the last upsampling, no cross-domain link is used, but a convolution layer is added, and the sigmoid activation function is used on the final feature map and mapped to a grayscale value of 0 to 255: P=255(σ(Conv 3×3 (and n ))) Where P represents the final reconstructed target modality image, y n Represents the feature map obtained after the nth upsampling.

3. The method according to claim 2, characterized in that Step 4 includes: Step 4-1, establish the mean absolute error loss function L l1 ; Step 4-2, establish the structural similarity loss function L SSIM ; Step 4-3, establish a complete loss function L; Step 4-4, using the loss function established in step 4-3, train the multimodal MRI reconstruction model through supervised training.

4. The method according to claim 3, characterized in that In step 4-1, the mean absolute error loss function L l1 The formula is: Wherein, p represents the index of the pixel in the target modality image P finally reconstructed, x(p) represents the predicted value of the model at position p, y(p) represents the true value of the label at position p, and N represents the total number of pixels in the target modality image P finally reconstructed.

5. The method according to claim 4, characterized in that In step 4-2, the structural similarity loss function L SSIM The formula is: Where SSIM represents the structural similarity function; μ x ,μ y Represent the pixel average of the output image and the pixel average of the true value, respectively. Represent the pixel value variance of the output image and the pixel value variance of the true value, σ xy Represents the pixel value covariance of the output image and the true value, C1, C2 are constants, C1 = (k1L) 2 ,C2=(k2L) 2 , L is the dynamic range of pixel values, k1 and k2 are set constants.

6. The method according to claim 5, characterized in that In step 4-3, the complete loss function L formula is: L(P)=γL l1 (P)+(1-γ)L SSIM (P) Among them, γ represents the weight of the loss function.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.

8. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 6 are executed.

Images