A dual-branch artificial intelligence lung MRI denoising method with model mixing

Abstract

The present invention discloses a model-mixed dual-branch artificial intelligence lung MRI denoising method. First, paired high-SNR and low-SNR lung MRI images are obtained as training sets, and a Transformer and CNN hybrid model is constructed. Then, the Transformer and CNN hybrid model is pre-trained through the set pre-training self-supervised loss function, and then supervised training is performed through the mean square error loss. Due to the complementary advantages of Transformer and CNN, the present invention effectively integrates local features and contextual information to improve the performance of image denoising; self-supervised pre-training using low-SNR images is suitable for the small sample size of paired lung MRI images; and after the training of the Transformer and CNN dual-channel hybrid model is completed, no additional parameter adjustment is required, the denoising speed is fast, and the image details can be better retained, which has practical value.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a model-mixed dual-branch artificial intelligence lung MRI denoising method, which is suitable for lung MRI denoising. Background Art

[0002] Hyperpolarized gas magnetic resonance imaging (MRI) can directly image the cavity structure of the lungs, thereby providing information on lung structure and function. It is a new practical imaging technology. However, unlike traditional proton MRI, the gas polarization of hyperpolarized gas MRI decays rapidly during the sampling process, resulting in a decrease in MRI signal-to-noise ratio and detail information. In addition, the limitations of the hardware system will further reduce the image quality, thus hindering the promotion and application of hyperpolarized gas MRI [He D, Caohui D, Sa X. IEEE Trans Instrum Meas., 2019, 68: 3950.]. Hyperpolarized gas MRI denoising can improve the image signal-to-noise ratio, which is of great significance for the study of lung MRI images. Generally, MRI noise removal can be divided into two ways, one is to improve the hardware, and the other is image post-processing. The latter uses an algorithm to denoise the low signal-to-noise ratio images obtained by scanning, thereby improving the image quality. This approach does not require processing of the imaging equipment and can be integrated into the existing imaging process, which has practical application value [Sagheer SVM, George S N. Biomed Signal Process Control., 2020, 61.].

[0003] In recent years, deep learning methods have shown great potential in the field of MRI denoising. They have significant advantages in denoising effect, denoising speed and model application. MRI denoising algorithms based on deep learning can autonomously learn how to extract image features through external data set training, and establish a mapping model from noisy MRI images to noise-free MRI images. However, most of the existing deep learning-based MRI denoising algorithms are based on convolutional neural networks (CNN) as the basic structure [Augustin AM, Kesavadas C, Sudeep P V. Biomed Signal Process Control., 2022, 77: 14.], and their performance is limited by factors such as convolution kernel size and network depth, and they cannot capture contextual information well. Transformer is another deep learning model that has excellent global modeling capabilities through a global attention mechanism and has achieved great success in natural language processing. In recent years, more and more Transformer-based methods have been proposed for computer vision tasks, and their performance has exceeded that of models based on convolutional neural networks. Compared with convolutional neural networks, Transformer can use the self-attention mechanism to extract global dependencies, making up for the shortcomings of convolutional neural networks. However, Transformer has some defects in local feature extraction, and capturing local features is crucial for image denoising [Zhendong W, Xiaodong C.CVPR., 2022.] In addition, Transformer requires a large data set for training, which is difficult to achieve in lung MRI.

[0004] In summary, the prior art has the following problems:

[0005] 1. Currently, deep learning-based lung MRI denoising algorithms all use convolutional neural networks as the basic structure, and their performance is limited by their own factors.

[0006] 2. The direct application of Transformer in lung MRI denoising has poor effect and its performance needs to be improved. Summary of the invention

[0007] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and propose a model-mixed dual-branch artificial intelligence lung MRI denoising method, which utilizes the complementarity of Transformer and CNN to improve the lung MRI denoising effect. No additional parameter adjustment and optimization are required during application, and the denoising speed is fast.

[0008] The above-mentioned object of the present invention is achieved by the following technical solutions:

[0009] A model-mixed dual-branch artificial intelligence lung MRI denoising method comprises the following steps:

[0010] Step 1, obtaining paired high signal-to-noise ratio lung MRI images and low signal-to-noise ratio lung MRI images as sample pairs, and dividing the sample pairs into a training set and a test set;

[0011] Step 2: Build a hybrid model of Transformer and CNN. The hybrid model of Transformer and CNN includes an upper channel network, a lower channel network and an output layer.

[0012] The low signal-to-noise ratio lung MRI image obtained in step 1 is the input of the upper channel network and the input of the lower channel network. The output of the upper channel network and the output of the lower channel network are stacked by channel and input to the output layer;

[0013] Step 3: Set the pre-training self-supervised loss function L s ;

[0014] Step 4: Input the low signal-to-noise ratio lung MRI images of the training set in step 1 into the Transformer and CNN hybrid model to obtain a predicted denoised lung MRI image; and according to the pre-trained self-supervised loss function set in step 3, use the low signal-to-noise ratio lung MRI images in the training set obtained in step 1 to pre-train the Transformer and CNN hybrid model constructed in step 2 to obtain a pre-trained Transformer and CNN hybrid model;

[0015] Step 5: Set the supervised training loss function L d ;

[0016] Step 6: Input the low signal-to-noise ratio lung MRI image of the training set in step 1 into the pre-trained Transformer and CNN hybrid model in step 4 to obtain the corresponding denoised lung MRI image; according to the supervised training loss function set in step 5, use the training set sample pairs generated in step 1 to perform end-to-end training on the pre-trained Transformer and CNN hybrid model in step 4 to obtain the parameters of the trained Transformer and CNN hybrid model;

[0017] Step 7: Input the low signal-to-noise ratio lung MRI image to be processed into the Transformer and CNN hybrid model trained in step 6 to obtain the corresponding denoised lung MRI image.

[0018] As described above, the upper channel network in the Transformer and CNN hybrid model includes multiple upper channel feature extraction units in order, and the upper channel feature extraction units include convolutional layers, group normalization layers and activation layers in sequence. The input of the first upper channel feature extraction unit in the upper channel network is used as the input of the upper channel network, and the feature map output by the last upper channel feature extraction unit in the upper channel network is used as the output of the upper channel network.

[0019] As mentioned above, the channel network includes the input layer, CNN encoder, Transformer encoder and decoder:

[0020] The CNN encoder sequentially includes n (n ≥ 1) CNN downsampling layers;

[0021] The Transformer encoder consists of an embedding layer and multiple Transformer modules in sequence, where the number of Transformer modules is greater than n+1;

[0022] The decoder consists of n+1 Transformer feature processing modules, n+1 decoder upsampling layers, and 1 feature output layer;

[0023] The input of the input layer is used as the input of the lower channel network, and the output of the input layer is input to the first CNN downsampling layer of the CNN encoder, the embedding layer of the Transformer encoder, and the feature output layer of the decoder respectively;

[0024] The output of each CNN downsampling layer is also input to a corresponding decoder upsampling layer. The first decoder upsampling layer has no input from the CNN downsampling layer. The sum of the serial numbers of each CNN downsampling layer and the serial number of the corresponding decoder upsampling layer is n+2.

[0025] The outputs of some selected Transformer modules except the last Transformer module in the Transformer encoder are respectively input to the corresponding Transformer feature processing modules except the last Transformer feature processing module; and the output of the last Transformer module is input to the last Transformer feature processing module;

[0026] The output of each Transformer feature processing module is input to a corresponding decoder upsampling layer. The Transformer feature processing module corresponds to the decoder upsampling layer one by one, and the sum of the serial number of the Transformer feature processing module and the serial number of the corresponding decoder upsampling layer is n+2;

[0027] The output of the current decoder upsampling layer is used as an input to the next decoder upsampling layer, and the output of the last decoder upsampling layer is used as an input to the feature output layer;

[0028] The output of the feature output layer is the output of the next channel network.

[0029] As described above, the input layer includes a plurality of input layer feature extraction units in sequence, and the input layer feature extraction units include a convolution layer, a group normalization layer and an activation layer in sequence;

[0030] The CNN downsampling layers in the CNN encoder all include convolutional layers, group normalization layers, and activation layers in order;

[0031] The embedding layer in the Transformer encoder includes multiple embedding layer downsampling layers and one position encoding. The embedding layer downsampling layers include convolutional layers, group normalization layers and activation layers in sequence. The Transformer module includes the first layer normalization, multi-head self-attention, the second layer normalization and the multi-layer perceptron in order. The output of each Transformer module is the word token obtained by adding the input of the second layer normalization and the output of the multi-layer perceptron pixel by pixel in the current Transformer module. The input of the second layer normalization is the token obtained by adding the input of the first layer normalization and the output of the multi-head attention pixel by pixel. The multi-layer perceptron includes the first fully connected layer, the activation layer and the second fully connected layer in order.

[0032] The Transformer feature processing module in the decoder includes a feature processing module upsampling layer, which includes an upsampling module, a convolution layer, a group normalization layer, and an activation layer. The feature map output by the feature processing module upsampling layer is consistent in size with the feature map output by the corresponding CNN downsampling layer.

[0033] The first decoder upsampling layer in the decoder includes an upsampling module, a convolution layer, a group normalization layer, and an activation layer; other decoder upsampling layers except the first decoder upsampling layer include a decoder upsampling layer fusion module, an upsampling module, a convolution layer, a group normalization layer, and an activation layer;

[0034] The output layer consists of a convolutional layer.

[0035] As mentioned above, the decoder upsampling layer fusion module includes three branches:

[0036] The structure of the first branch is the same as that of the second branch, and both include, in sequence, a first fused convolution layer, a first fused group normalization layer, a first fused activation layer, a second fused convolution layer, and a second fused group normalization layer; and,

[0037] The input of the first branch of the decoder upsampling layer fusion module is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the first branch belongs, and the feature map output by the previous decoder upsampling layer. The feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs and the output of the second fusion group normalization layer of the first branch are summed pixel by pixel and then input into the second fusion activation layer of the first branch;

[0038] The input of the second branch of the decoder upsampling layer fusion module is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the second branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs, and the feature map output by the previous decoder upsampling layer. The feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs and the output of the second fusion group normalization layer of the second branch are summed pixel by pixel and then input into the second fusion activation layer of the second branch;

[0039] The input of the third branch of the decoder upsampling layer fusion module is the feature map output by the previous decoder upsampling layer;

[0040] The feature maps output by the three branches of the decoder upsampling layer fusion module are stacked channel by channel through the fusion layer of the decoder upsampling layer fusion module, and then pass through 1×1 convolution.

[0041] As described above, the feature output layer in the decoder includes one feature output layer fusion module and a plurality of feature output layer feature extraction units in order, wherein the feature output layer feature extraction unit includes a convolution layer, a group normalization layer and an activation layer in order;

[0042] The feature output layer fusion module includes two branches:

[0043] The first branch of the feature output layer fusion module includes a first fusion convolution layer, a first fusion group normalization layer, a first fusion activation layer, a second fusion convolution layer, and a second fusion group normalization layer in sequence. The input of the first branch of the feature output layer fusion module is a result of stacking the feature map output by the input layer and the feature map output by the last level decoder upsampling layer according to channels. The feature map output by the input layer and the feature map output by the second fusion group normalization layer of the first branch of the feature output layer fusion module are summed pixel by pixel and then input into the second fusion activation layer of the first branch of the feature output layer fusion module.

[0044] The input of the second branch of the feature output layer fusion module is the feature map output by the upsampling layer of the last decoder;

[0045] The feature maps output by the two branches of the feature output layer fusion module are stacked by channel through the fusion layer and then pass through 1×1 convolution.

[0046] As mentioned above, the self-supervised loss function L is pre-trained s for:

[0047] L s =‖(Mm)·[D(m·x)-x]‖ F

[0048] in,‖·‖ F is the F-norm of the matrix, x is a low signal-to-noise ratio lung MRI image, m is a random Bernoulli matrix, D represents the hybrid model of Transformer and CNN, D(m·x) represents the output of the hybrid model of Transformer and CNN, M is an all-1 matrix, and the operator · refers to the dot product.

[0049] As mentioned above, the supervised training loss function L d for:

[0050] L d =L1[D′(x),y]

[0051] Where L1 is the mean absolute error operation, x is a low signal-to-noise ratio lung MRI image, y is a high signal-to-noise ratio lung MRI image paired with the low signal-to-noise ratio lung MRI image x, D′ represents the pre-trained Transformer and CNN hybrid model, and D′(x) represents the output of the pre-trained Transformer and CNN hybrid model.

[0052] For the upper channel feature extraction unit as described above: the number of input channels of the convolution layer of the first upper channel feature extraction unit is 1, and the number of output channels is greater than the number of input channels; except for the first upper channel feature extraction unit, the number of input channels of the convolution layer of each other upper channel feature extraction unit is equal to the number of output channels;

[0053] For the input layer feature extraction unit: the number of input channels of the input layer feature extraction unit with the same serial number is consistent with the number of input channels of the upper channel feature extraction unit, and the number of output channels of the input layer feature extraction unit with the same serial number is consistent with the number of output channels of the upper channel feature extraction unit;

[0054] For CNN downsampling layer: the number of output channels of the convolution layer in the CNN downsampling layer is twice the number of input channels;

[0055] For the embedding layer downsampling layer: except for the last embedding layer downsampling layer, the number of output channels of the convolution layer is twice the number of input channels; the number of output channels of the convolution layer of the last embedding layer downsampling layer is 3 / 2 of the number of input channels;

[0056] For the feature output layer feature extraction unit: the number of convolutional layer input channels of the feature output layer feature extraction unit is equal to the number of output channels.

[0057] As described above, in the Transformer feature processing module, the number of output channels of the convolution layer of the upsampling layer of the first feature processing module is 2 / 3 of the number of input channels; except for the convolution layer of the upsampling layer of the first feature processing module, the number of output channels of the convolution layer of the upsampling layer of other feature processing modules is 1 / 2 of the number of input channels;

[0058] The number of output channels of the convolution layer of the decoder upsampling layer is 1 / 2 of the number of input channels.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention builds a hybrid model of Transformer and CNN, pre-trains the hybrid model of Transformer and CNN using the set pre-training self-supervised loss function, and then uses the mean square error loss for supervised training, which can effectively denoise low signal-to-noise ratio lung MRI images. Compared with the existing magnetic resonance image denoising technology based on deep learning, the present invention uses the complementary advantages of Transformer and CNN to effectively integrate local features and contextual information to improve the performance of image denoising. In addition, the present invention uses low signal-to-noise ratio images for self-supervised pre-training, which is suitable for the small sample size of lung MRI image pairing and has practical application value.

[0061] 2. Compared with traditional magnetic resonance image denoising technology, the method of the present invention does not require additional parameter adjustment after the training of the Transformer and CNN hybrid model is completed, has a fast denoising speed, can better preserve image details, and has excellent results.

[0062] In order to effectively apply the Transformer and CNN hybrid model to lung MRI denoising, the present invention designs a Transformer encoder and a CNN encoder to process the input features, respectively, which can retain contextual information and local features at the same time. In addition, in order to reduce the semantic difference between the output features of the Transformer encoder and the CNN encoder, the present invention processes the output of the Transformer encoder and the output of the CNN encoder respectively when constructing a multi-branch fusion module. Finally, in order to compensate for the loss caused by downsampling in the Transformer encoder and the CNN encoder, the upper channel network of the present invention retains the original size of the input image, thereby retaining the image information to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of the present invention;

[0064] Figure 2 Schematic diagram of the structure of the hybrid model of Transformer and CNN; wherein, (a) is a schematic diagram of the overall structure of the hybrid model of Transformer and CNN, (b) is a schematic diagram of the structure of the embedding layer included in the Transformer encoder, (c) is a schematic diagram of the structure of the decoder upsampling layer, (d) is a schematic diagram of the structure of the decoder upsampling layer fusion module, and (d) is a schematic diagram of the Transformer module structure;

[0065] Figure 3 Comparison diagram of low signal-to-noise ratio 2D lung MRI image and denoised 2D lung MRI image; (a) is a low signal-to-noise ratio 2D lung MRI image, and (b) is a denoised 2D lung MRI image;

[0066] Figure 4 This is a qualitative comparison of the performance of the present invention and other methods on the constructed data set, where (a) is a high signal-to-noise ratio lung MRI image of the test set, (b) is a low signal-to-noise ratio lung MRI image of the test set, (c) is the denoising result obtained by the NLM non-local mean filtering algorithm, (d) is the denoising result obtained by the Dncnnn CNN-based method, and (e) is the denoising result obtained by the present invention method.

[0067] Table 1 is a quantitative comparison table of the performance of the present invention and other methods. DETAILED DESCRIPTION

[0068] In order to facilitate those skilled in the art to understand and implement the present invention, the following embodiments, Figure 1 — Figure 4 The present invention is further described in detail in Table 1. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0069] Example 1

[0070] A model-mixed dual-branch artificial intelligence lung MRI denoising method specifically includes the following steps:

[0071] Step 1: Obtain paired high signal-to-noise ratio lung MRI images and low signal-to-noise ratio lung MRI images as sample pairs, and divide the sample pairs into a training set and a test set.

[0072] In this embodiment, 1059 high signal-to-noise ratio lung original MRI images with a signal-to-noise ratio greater than 15 were obtained, 847 high signal-to-noise ratio lung original MRI images were used to construct a training set, and 212 high signal-to-noise ratio lung original MRI images were used to construct a test set. Among them, the 847 high signal-to-noise ratio lung original MRI images used to obtain the training set were rotated 90°, 180°, and 270° respectively, and a total of 3388 high signal-to-noise ratio lung MRI images were obtained and included in the training set; the 212 high signal-to-noise ratio lung original MRI images used to obtain the test set were directly used as high signal-to-noise ratio lung MRI images of the test set. Then, Rice noise was added to the 3388 high signal-to-noise ratio lung MRI images of the training set and the 212 high signal-to-noise ratio lung MRI image sets of the test set, and paired low signal-to-noise ratio lung MRI images with a signal-to-noise ratio of 8 to 15 were obtained respectively. The high signal-to-noise ratio lung MRI image and the paired low signal-to-noise ratio lung MRI image obtained by adding Rice noise are used as sample pairs. The initial size of the high signal-to-noise ratio lung MRI image and the low signal-to-noise ratio lung MRI image are both 96×84, which are expanded to 96×96 by zero padding.

[0073] Step 2: Build a hybrid model of Transformer and CNN. The hybrid model of Transformer and CNN includes an upper channel network, a lower channel network and an output layer, such as Figure 2 As shown in FIG. 1 , the upper channel network includes multiple upper channel feature extraction units in order; the lower channel network includes an input layer, a CNN encoder, a Transformer encoder and a decoder in order, and the output layer includes a convolutional layer.

[0074] The low signal-to-noise ratio lung MRI images obtained in step 1 are input into the upper channel network and the lower channel network respectively. The output of the upper channel network and the output of the lower channel network are stacked by channel and input into the output layer.

[0075] In this embodiment, the upper channel network includes 6 upper channel feature extraction units in order, and the upper channel feature extraction units include convolution layers, group normalization layers and activation layers in order. The convolution kernel size in the convolution layer of the first upper channel feature extraction unit is 3×3, the input channel is 1, the output channel is 64, the input feature map size is 96×96, and the output feature map size is 96×96; except for the first upper channel feature extraction unit, the convolution kernel size in the convolution layer of each other upper channel feature extraction unit is 3×3, the input channel is 64, and the output channel is 64; in the upper channel network, the activation function of each activation layer is the LeakyReLU function. The input of the first upper channel feature extraction unit in the upper channel network is used as the input of the upper channel network, and the feature map output by the last upper channel feature extraction unit in the upper channel network is used as the output of the upper channel network.

[0076] The lower channel network includes an input layer, a CNN encoder, a Transformer encoder and a decoder in sequence. In this embodiment, the lower channel network includes 1 input layer, 1 CNN encoder, 1 Transformer encoder and 1 decoder.

[0077] In the down channel network:

[0078] The CNN encoder includes n (n≥1) CNN downsampling layers in sequence; in this embodiment, n=2.

[0079] The Transformer encoder consists of an embedding layer and multiple Transformer modules in sequence, where the number of Transformer modules is greater than n+1;

[0080] The decoder consists of n+1 Transformer feature processing modules, n+1 decoder upsampling layers, and 1 feature output layer;

[0081] The input of the input layer is used as the input of the lower channel network, and the output of the input layer is input to the first CNN downsampling layer of the CNN encoder, the embedding layer of the Transformer encoder, and the feature output layer of the decoder respectively;

[0082] The output of each CNN downsampling layer is also input to a corresponding decoder upsampling layer. The first decoder upsampling layer has no input from the CNN downsampling layer. The sum of the serial numbers of each CNN downsampling layer and the serial number of the corresponding decoder upsampling layer is n+2.

[0083] The outputs of some selected Transformer modules except the last Transformer module in the Transformer encoder are respectively input to the corresponding Transformer feature processing modules except the last Transformer feature processing module; and the output of the last Transformer module is input to the last Transformer feature processing module;

[0084] The output of each Transformer feature processing module is input to a corresponding decoder upsampling layer. The Transformer feature processing module corresponds to the decoder upsampling layer one by one, and the sum of the serial number of the Transformer feature processing module and the serial number of the corresponding decoder upsampling layer is n+2.

[0085] The output of the current decoder upsampling layer is used as an input of the next decoder upsampling layer (the next decoder upsampling layer is the decoder upsampling layer with sequence number increased by 1), and the output of the last decoder upsampling layer is used as an input of the feature output layer;

[0086] The output of the feature output layer is the output of the next channel network.

[0087] In this embodiment, the specific configuration of the down channel network is as follows:

[0088] The input layer includes multiple input layer feature extraction units in sequence. The structure of the input layer feature extraction unit is consistent with the structure of the upper channel feature extraction unit and also includes a convolution layer, a group normalization layer and an activation layer in sequence; the number of input layer feature extraction units and the number of upper channel feature extraction units are not limited. In this embodiment, the input layer includes 2 input layer feature extraction units in order. The convolution kernel size in the convolution layer of the first input layer feature extraction unit is 3×3, the input channel is 1, the output channel is 64, the input feature map size is 96×96, and the output feature map size is 96×96; the convolution kernel size in the convolution layer of the second input layer feature extraction unit is 3×3, the input channel is 64, the output channel is 64, the input feature map size is 96×96, and the output feature map size is 96×96; the activation function of the activation layer in the input layer is the LeakyReLU function.

[0089] The CNN encoder includes multiple CNN downsampling layers in sequence. In this embodiment, the CNN encoder includes 2 CNN downsampling layers in order. The CNN downsampling layers all include convolutional layers, group normalization layers, and activation layers in sequence; the convolutional layer of the first CNN downsampling layer has a convolutional kernel size of 3×3, 64 input channels, 128 output channels, 96×96 input feature map size, and 48×48 output feature map size; the convolutional layer of the second CNN downsampling layer has a convolutional kernel size of 3×3, 128 input channels, 256 output channels, 48×48 input feature map size, and 24×24 output feature map size; in the CNN encoder, the activation function of each activation layer is the LeakyReLU function.

[0090] The Transformer encoder includes one embedding layer and multiple Transformer modules. In this embodiment, the Transformer encoder includes one embedding layer and 12 Transformer modules.

[0091] The embedding layer includes multiple embedding layer downsampling layers and one position encoding in order. The embedding layer downsampling layers each include a convolution layer, a group normalization layer, and an activation layer in order. In this embodiment, the embedding layer includes 4 embedding layer downsampling layers and one position encoding in order. Among them, the convolution kernel size of the convolution layer of the k1th embedding layer downsampling layer is 2×2, and the number of input channels is 64×2 k1-1 , the output channel is 64×2 k1 , the input feature map size is The output feature map size is Among them, k1=1,2,3; the convolution kernel size of the convolution layer of the 4th embedding layer downsampling layer is 2×2, the number of input channels is 512, the number of output channels is 768, the input feature map size is 12×12, and the output feature map size is 6×6; the position code is a learnable tensor of size 1×36×768. The input of the embedding layer is the input of the 1st embedding layer downsampling layer, including the feature map output by the input layer; the output of the embedding layer is the sum of the token obtained by flattening the feature map output by the 4th embedding layer downsampling layer and the position code; the token size obtained by flattening the feature map output by the 4th embedding layer downsampling layer is 1×36×768.

[0092] The Transformer module includes the first layer of normalization, multi-head self-attention, the second layer of normalization and the multi-layer perceptron in order. The multi-head self-attention includes 12 heads, the input token size is 1×36×768, and the output token size is 1×36×768; the multi-layer perceptron includes the first fully connected layer, the activation layer and the second fully connected layer in order. The first fully connected layer has an input dimension of 768 and an output dimension of 3072, the input token size is 1×36×768, and the output token size is 1×36×3072; the second fully connected layer has an input dimension of 3072 and an output dimension of 768, the input token size is 1×36×3072, and the output token size is 1×36×768; the activation function of the activation layer of the multi-layer perceptron is the GELU function. The output of each Transformer module is the token obtained by adding the normalized input of the second layer in the current Transformer module and the output of the multi-layer perceptron pixel by pixel; the normalized input of the second layer is the token obtained by adding the normalized input of the first layer and the output of the multi-head self-attention pixel by pixel. The input of the first Transformer module is the token output by the embedding layer; except for the first Transformer module, the input of other Transformer modules is the output of the previous Transformer module.

[0093] The decoder includes multiple Transformer feature processing modules, multiple decoder upsampling layers, and one feature output layer. In this example, the decoder includes three Transformer feature processing modules, three decoder upsampling layers, and one feature output layer.

[0094] The first Transformer feature processing module includes three feature processing module upsampling layers: the feature processing module upsampling layer includes upsampling module, convolution layer, group normalization layer and activation layer. The upsampling method of the upsampling module is bilinear interpolation. The output feature map size of the upsampling module in the feature processing module upsampling layer is 4 times the input feature map size. The input feature map size is The output feature map size is where k 2-1 =1,2,3,k 2-1Indicates the sequence number of the upsampling layer of the feature processing module in the first Transformer feature processing module; in this embodiment, the token output by the fourth Transformer module is reorganized to a size of 768×6×6 and is used as the input of the upsampling layer of the first feature processing module of the first Transformer feature processing module. The convolution kernel size in the convolution layer of the upsampling layer of the first feature processing module of the first Transformer feature processing module is 3×3, the number of input channels is 768, and the number of output channels is 512; the convolution kernel size in the convolution layer of the upsampling layer of the M1th feature processing module is 3×3, and the number of input channels is The number of output channels is Where M1=2,3, M1 represents the sequence number of the convolutional layer of the upsampling layer of the feature processing module in the first Transformer feature processing module.

[0095] The second Transformer feature processing module includes two feature processing module upsampling layers: the feature processing module upsampling layer includes an upsampling module, a convolution layer, a group normalization layer and an activation layer. The upsampling method of the upsampling module is bilinear interpolation. 2-2 The output feature map size of the upsampling module in the upsampling layer of the feature processing module is 4 times the input feature map size. The input feature map size is The output feature map size is where k 2-2 =1,2,k 2-2 Indicates the serial number of the upsampling layer of the feature processing module in the second Transformer feature processing module. In this embodiment, the token output by the 8th Transformer module is resized to 768×6×6 and is used as the input of the upsampling layer of the 1st feature processing module of the 2nd Transformer feature processing module. The convolution kernel size in the convolution layer of the upsampling layer of the 1st feature processing module of the 2nd Transformer feature processing module is 3×3, the number of input channels is 768, and the number of output channels is 512; the convolution kernel size in the convolution layer of the upsampling layer of the M2th feature processing module is 3×3, the number of input channels is The number of output channels is Wherein M2=2, M2 represents the sequence number of the convolution layer of the upsampling layer of the feature processing module in the second Transformer feature processing module.

[0096] The third Transformer feature processing module includes a feature processing module upsampling layer: the feature processing module upsampling layer includes an upsampling module, a convolution layer, a group normalization layer and an activation layer. The upsampling mode of the upsampling module is bilinear interpolation. The size of the output feature map of the upsampling module in the feature processing module upsampling layer is 4 times the size of the input feature map, the input feature map size is 6×6, and the output feature map size is 12×12. The convolution kernel size in the convolution layer of the feature processing module upsampling layer is 3×3, the number of input channels is 768, and the number of output channels is 512. In this embodiment, the token output by the 12th Transformer module is reorganized to a size of 768×6×6 and is used as the input of the feature processing module upsampling layer of the third Transformer feature processing module. The input feature map size of the feature processing module upsampling layer of the third Transformer feature processing module is 6×6, and the output feature map size is 12×12.

[0097] Generally, the kth Transformer feature processing module includes n+2-k feature processing module upsampling layers, and the feature processing module upsampling layers include upsampling modules, convolution layers, group normalization layers, and activation layers. The upsampling method of the upsampling modules in the feature processing module upsampling layers is bilinear interpolation. In this embodiment, n=2, k=1,2,3. After the feature processing module upsampling layer in the Transformer feature processing module upsamples the feature map output by the Transformer module, the size of the feature map output by the corresponding CNN downsampling layer must be consistent with that of the feature map output by the corresponding CNN downsampling layer. Therefore, the first Transformer feature processing module includes 3 feature processing module upsampling layers, and the size of the feature map output by the first Transformer feature processing module is consistent with the size of the feature map output by the first CNN downsampling layer in the CNN encoder, and so on.

[0098] The first decoder upsampling layer of the three decoder upsampling layers in the decoder includes an upsampling module, a convolution layer, a group normalization layer, and an activation layer. The feature map output by the third Transformer feature processing module is used as the input of the first decoder upsampling layer. The upsampling method of the upsampling module in the decoder upsampling layer is bilinear interpolation. The output feature map size of the upsampling module in the decoder upsampling layer is 4 times the size of the input feature map. The input feature map size is 12×12, and the output feature map size is 24×24. The convolution kernel size in the convolution layer of the first decoder upsampling layer is 3×3, the number of input channels is 512, and the number of output channels is 256.

[0099] The k3th (k3>1) decoder upsampling layer among the three decoder upsampling layers in the decoder includes a decoder upsampling layer fusion module, an upsampling module, a convolution layer, a group normalization layer, and an activation layer. The feature maps output by the 4th to k3th CNN downsampling layers in the CNN encoder, the feature maps output by the 4th to k3th Transformer feature processing modules, and the feature maps output by the k3-1th decoder upsampling layer are used as inputs to the k3th decoder upsampling layer fusion module. The decoder upsampling layer fusion module includes three branches, two of which are branches with convolution layers. The decoder upsampling layer fusion module stacks the feature maps output by the 4th to k3th CNN downsampling layers in the CNN encoder, the feature maps output by the 4th to k3th Transformer feature processing modules, and the feature maps output by the k3-1th decoder upsampling layer by channels as inputs to the two branches with convolution layers, and uses the feature maps output by the k3-1th decoder upsampling layer as input to the third branch. Then, the feature maps output by the three branches are stacked by channels and subjected to 1×1 convolution, specifically:

[0100] The first branch and the second branch are both branches with convolutional layers and have the same structure. The first branch and the second branch both sequentially include a first fused convolutional layer, a first fused group normalization layer, a first fused activation layer, a second fused convolutional layer, and a second fused group normalization layer;

[0101] For the first branch of the decoder upsampling layer fusion module: the input of the first branch is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the first branch belongs, and the feature map output by the previous decoder upsampling layer stacked by channels; and the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs and the output of the second fusion group normalization layer of the first branch are summed pixel by pixel and input into the second fusion activation layer of the first branch; in this embodiment, the serial number of the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs is 4-k3.

[0102] For the second branch of the decoder upsampling layer fusion module: the input of the second branch is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the second branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs, and the feature map output by the previous level decoder upsampling layer stacked by channel; and the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs and the output of the second fusion group normalization layer of the second branch are summed pixel by pixel and input to the second fusion activation layer of the second branch; in this embodiment, the serial number of the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs is 4-k3;

[0103] For the third branch of the decoder upsampling layer fusion module: the input of the third branch is the feature map output by the upsampling layer of the previous decoder;

[0104] The feature maps output by the three branches of the decoder upsampling layer fusion module are stacked by channel through the fusion layer and then pass through 1×1 convolution.

[0105] In this embodiment: the convolution kernel size of the first fusion convolution layer in the first branch and the second branch of the k3th decoder upsampling layer fusion module is 3×3, and the input channel is The output channel is The input feature map size is The output feature map size is The convolution kernel size of the second fused convolution layer in the first branch and the second branch is 3×3, and the input channel is The output channel is The input feature map size is The output feature map size is The second fusion activation layer input in the first branch with a convolution layer in the decoder upsampling layer fusion module includes the feature map output by the 4th to k3th CNN downsampling layer in the CNN encoder and the feature map output by the second fusion group normalization layer. The second fusion activation layer input in the second branch with a convolution layer includes the feature map output by the 4th to k3th Transformer feature processing module and the feature map output by the second fusion group normalization layer. The feature map obtained by summing the pixels; the output of the third branch is the feature map output by the upsampling layer of the previous decoder. The fusion layer stacks the feature maps output by the three branches according to the channels, and then reduces the number of channels to 1×1 through 1 convolution. The input feature map size of the k3th decoder upsampling layer fusion module Output feature map size The upsampling mode of the upsampling module in the decoder upsampling layer is bilinear interpolation. The output feature map size of the upsampling module is 4 times the input feature map size. The output feature map size is The convolution kernel size in the convolution layer of the k3th decoder upsampling layer is 3×3, and the number of input channels is The number of output channels is The input feature map size of the k3th decoder upsampling layer is The output feature map size is Among them, k=2,3.

[0106] The feature output layer in the decoder includes, in order, one feature output layer fusion module and a plurality of feature output layer feature extraction units. In this embodiment, the feature output layer in the decoder includes, in order, one feature output layer fusion module and two feature output layer feature extraction units. The structure of the feature output layer feature extraction unit is consistent with that of the upper channel feature extraction unit. The feature output layer fusion module includes two branches, one fusion layer and a 1×1 convolution layer. The first branch in the feature output layer fusion module has the same structure as the first branch in the upsampling layer fusion module of the decoder, and both include, in order, a first fusion convolution layer, a first fusion group normalization layer, a first fusion activation layer, a second fusion convolution layer, and a second fusion group normalization layer. The feature map output by the first layer, the feature map output by the input layer and the feature map output by the normalization layer of the second fusion group are summed pixel by pixel and then input into the second fusion activation layer of the first branch in the feature output layer fusion module; but the number of convolutional layer channels in the feature output layer fusion module is different from the number of convolutional layer channels in the decoder upsampling layer fusion module; the input of the first branch is the result of channel stacking of the feature map output by the input layer and the feature map output by the last level decoder upsampling layer; the input of the second branch is the feature map output by the last level decoder upsampling layer; the fusion layer of the feature output layer fusion module stacks the feature maps output by the two branches through the fusion layer of the feature output layer fusion module according to channels, and then passes through 1×1 convolution.

[0107] In this embodiment, in the feature output layer fusion module, the convolution kernel size of the first convolution layer of the first branch is 3×3, the input channel is 128, the output channel is 64, the input feature map size is 96×96, and the output feature map size is 96×96; the convolution kernel size of the second convolution layer in the first branch is 3×3, the input channel is 64, the output channel is 64, the input feature map size is 96×96, and the output feature map size is 96×96; the feature map output by the input layer and the feature map output by the third decoder upsampling layer are used as the input of the feature output layer fusion module: the feature output layer fusion module stacks the feature map output by the input layer and the feature map output by the third decoder upsampling layer by channel as the first fusion of the first branch of the feature output layer fusion module. The input of the convolutional layer is taken as the input of the second branch of the feature output layer fusion module; the feature map output by the third decoder upsampling layer is taken as the input of the second branch of the feature output layer fusion module; the fusion layer of the feature output layer fusion module stacks the feature maps output by the two branches according to the channel, and then reduces the number of channels to 64 through 1×1 convolution; the feature map output by the input layer and the feature map output by the third decoder upsampling layer are both 96×96 in size, and the feature map output by the feature output layer fusion module is 96×96 in size; the convolution kernel size in the convolutional layer of the feature extraction unit of the feature output layer is 3×3, the input channel is 64, the output channel is 64, the input feature map size is 96×96, the output feature map size is 96×96, and the feature map output by the last feature output layer feature extraction unit is the output feature map of the lower channel network.

[0108] The output layer includes a convolution layer, in which the convolution kernel size is 3×3, the input channel is 128, the output channel is 1, the input feature map size is 96×96, the output feature map size is 96×96, and the feature map output by the lower channel network and the feature map output by the upper channel network are stacked by channel as the input of the output layer. Among them, the feature map output by the last upper channel feature extraction unit in the upper channel network is used as the feature map output by the upper channel network, and the feature map output by the feature output layer in the lower channel network is used as the feature map output by the lower channel network.

[0109] Step 3: Set the pre-training self-supervised loss function L s . Pre-trained self-supervised loss function L s Set to:

[0110] L s =‖(Mm)·[D(m·x)-x]‖ F

[0111] in,‖·‖ Fis the F norm of the matrix, x is a low signal-to-noise ratio lung MRI image, m is a random Bernoulli matrix, D represents the Transformer and CNN hybrid model, and D(m·x) represents the output of the Transformer and CNN hybrid model. In this embodiment, x is a low signal-to-noise ratio lung MRI image of the training set, the size is 96×96, m is a random Bernoulli matrix of size 96×96, each element in the random Bernoulli matrix obeys a Bernoulli distribution of P=0.5, M is an all-1 matrix, and the operator·refers to dot multiplication.

[0112] Step 4: Input the low signal-to-noise ratio lung MRI images of the training set in step 1 into the Transformer and CNN hybrid model to obtain the predicted denoised lung MRI images. According to the pre-trained self-supervised loss function set in step 3, the low signal-to-noise ratio lung MRI images of the training set in step 1 are used to pre-train the Transformer and CNN hybrid model constructed in step 2. The learning rate of the network is initialized to 0.0005, the batch size is set to 8, and the network is trained on the PyTorch platform using the ADAMW optimizer. The training is stopped after the total number of iterations reaches 60, and the pre-trained Transformer and CNN hybrid model is obtained, and the corresponding Transformer and CNN hybrid model parameters are saved.

[0113] Step 5: Set the supervised training loss function L d . Supervised training loss function L d Set to

[0114] L d =L1[D′(x),y]

[0115] Where L1 is the mean absolute error operation, x is a low signal-to-noise ratio lung MRI image, y is a high signal-to-noise ratio lung MRI image paired with the low signal-to-noise ratio lung MRI image x, D′ represents the pre-trained Transformer and CNN hybrid model, and D′(x) represents the output of the pre-trained Transformer and CNN hybrid model. In this example, x is a low signal-to-noise ratio lung MRI image of the training set, with a size of 96×96, and y is a high signal-to-noise ratio lung MRI image paired with the low signal-to-noise ratio lung MRI image, with a size of 96×96.

[0116] Step 6, input the low signal-to-noise ratio lung MRI image of the training set in step 1 into the pre-trained Transformer and CNN hybrid model in step 4 to obtain the corresponding denoised lung MRI image. And according to the supervised training loss function set in step 5, the training set sample pairs (i.e., the low signal-to-noise ratio lung MRI images of the training set and the paired high signal-to-noise ratio lung MRI images) are generated using step 1 to perform end-to-end training on the pre-trained Transformer and CNN hybrid model in step 4, the learning rate of the network is initialized to 0.0001, the batch size is set to 8, and the network is trained on the PyTorch platform using the ADAMW optimizer. After the total number of iterations reaches 60, the training is stopped to obtain the trained Transformer and CNN hybrid model, and the parameters of the corresponding Transformer and CNN hybrid model are saved. The present invention first uses the self-supervised training of step 4 to provide appropriate initial values ​​for the denoising network using the low signal-to-noise ratio lung MRI images, and gives the Transformer and CNN hybrid models certain denoising capabilities, and then performs supervised training in step 6, thereby improving the network denoising performance.

[0117] Step 7: Input the low signal-to-noise ratio lung MRI image to be processed into the hybrid model of Transformer and CNN trained in step 6 to obtain a denoised lung MRI image. In this embodiment, the low signal-to-noise ratio lung MRI image in the test set generated in step 1 is input into the hybrid model of Transformer and CNN trained in step 6 to obtain the corresponding denoised lung MRI image.

[0118] The denoised lung MRI image obtained in step 7 is compared with the low signal-to-noise ratio lung MRI image in the test set obtained in step 1, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) indicators of the denoised lung MRI image and the low signal-to-noise ratio lung MRI image are calculated respectively, such as Figure 3 As shown in the figure, the left side is a low signal-to-noise ratio lung MRI image, and the right side is a lung MRI image denoised by the method of the present invention. It can be seen from the result figure that the model-mixed dual-branch artificial intelligence lung MRI denoising method provided by the present invention can effectively remove noise, and the peak signal-to-noise ratio of the low signal-to-noise ratio lung MRI image is increased from 21.2 to 28.1, thereby obtaining a high-quality denoised lung MRI image.

[0119] The proposed method is compared with other methods, and the results are shown in Figure 4As shown in Table 1, NLM is a non-local mean filtering algorithm with a PSNR of 20.241 and a SSIM of 0.498; Dncnn is a CNN-based method with a PSNR of 29.433 and a SSIM of 0.855; the PSNR of the method of the present invention is 30.012 and the SSIM is 0.869. Compared with other methods, the model-mixed dual-branch artificial intelligence lung MRI denoising method provided by the present invention can better retain image details while achieving denoising.

[0120] Table 1 Quantitative comparison of the performance of the present invention and other methods

[0121]

[0122] Example 2

[0123] Utilizing the model-mixed dual-branch artificial intelligence lung MRI denoising method described in Example 1, this embodiment provides a model-mixed dual-branch artificial intelligence lung MRI denoising device, including a preprocessing module, a model construction module, a pre-trained self-supervised loss function construction module, a pre-training module, a supervised training loss function construction module, a supervised training module, and a use test module:

[0124] The preprocessing module is used to implement step 1 in Example 1, that is, to obtain paired high signal-to-noise ratio lung MRI images and low signal-to-noise ratio lung MRI images as sample pairs, and divide the sample pairs into a training set and a test set.

[0125] The model building module is used to implement step 2 in Example 1, that is, to build a hybrid model of Transformer and CNN;

[0126] The pre-trained self-supervised loss function construction module is used to implement step 3 in Example 1, i.e., setting the pre-trained self-supervised loss function;

[0127] The pre-training module is used to implement step 4 in Example 1, that is, pre-training the Transformer and CNN hybrid model;

[0128] The supervised training loss function construction module is used to implement step 5 in Example 1, i.e., setting the supervised training loss function;

[0129] The supervised training module is used to implement step 6 in Example 1, i.e., training the pre-trained Transformer and CNN hybrid model;

[0130] The test module is used to implement step 7 in Example 1, that is, processing the low signal-to-noise ratio lung MRI image to be denoised.

[0131] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A model-mixed dual-branch artificial intelligence lung MRI denoising method, characterized in that: The following steps are involved: Step 1, obtaining paired high signal-to-noise ratio lung MRI images and low signal-to-noise ratio lung MRI images as sample pairs, and dividing the sample pairs into a training set and a test set; Step 2: Build a hybrid model of Transformer and CNN. The hybrid model of Transformer and CNN includes an upper channel network, a lower channel network and an output layer. The low signal-to-noise ratio lung MRI image obtained in step 1 is the input of the upper channel network and the input of the lower channel network. The output of the upper channel network and the output of the lower channel network are stacked by channel and input to the output layer; Step 3: Set the pre-training self-supervised loss function L s ; Step 4: Input the low signal-to-noise ratio lung MRI images of the training set in step 1 into the Transformer and CNN hybrid model to obtain a predicted denoised lung MRI image; and according to the pre-trained self-supervised loss function set in step 3, use the low signal-to-noise ratio lung MRI images in the training set obtained in step 1 to pre-train the Transformer and CNN hybrid model constructed in step 2 to obtain a pre-trained Transformer and CNN hybrid model; Step 5: Set the supervised training loss function L d ; Step 6: Input the low signal-to-noise ratio lung MRI image of the training set in step 1 into the pre-trained Transformer and CNN hybrid model in step 4 to obtain the corresponding denoised lung MRI image; according to the supervised training loss function set in step 5, use the training set sample pairs generated in step 1 to perform end-to-end training on the pre-trained Transformer and CNN hybrid model in step 4 to obtain the parameters of the trained Transformer and CNN hybrid model; Step 7, input the low signal-to-noise ratio lung MRI image to be processed into the Transformer and CNN hybrid model trained in step 6 to obtain the corresponding denoised lung MRI image; The lower channel network includes an input layer, a CNN encoder, a Transformer encoder, and a decoder: The CNN encoder sequentially includes n (n ≥ 1) CNN downsampling layers; The Transformer encoder consists of an embedding layer and multiple Transformer modules in sequence, where the number of Transformer modules is greater than n+1; The decoder consists of n+1 Transformer feature processing modules, n+1 decoder upsampling layers, and 1 feature output layer; The input of the input layer is used as the input of the lower channel network, and the output of the input layer is input to the first CNN downsampling layer of the CNN encoder, the embedding layer of the Transformer encoder, and the feature output layer of the decoder respectively; The output of each CNN downsampling layer is also input to a corresponding decoder upsampling layer. The first decoder upsampling layer has no input from the CNN downsampling layer. The sum of the serial numbers of each CNN downsampling layer and the serial number of the corresponding decoder upsampling layer is n+2. The outputs of some selected Transformer modules except the last Transformer module in the Transformer encoder are respectively input to the corresponding Transformer feature processing modules except the last Transformer feature processing module; and the output of the last Transformer module is input to the last Transformer feature processing module; The output of each Transformer feature processing module is input to a corresponding decoder upsampling layer. The Transformer feature processing module corresponds to the decoder upsampling layer one by one, and the sum of the serial number of the Transformer feature processing module and the serial number of the corresponding decoder upsampling layer is n+2; The output of the current decoder upsampling layer is used as an input to the next decoder upsampling layer, and the output of the last decoder upsampling layer is used as an input to the feature output layer; The output of the feature output layer is the output of the lower channel network; The upper channel network in the Transformer and CNN hybrid model includes multiple upper channel feature extraction units in order, and the upper channel feature extraction units include convolutional layers, group normalization layers and activation layers in sequence. The input of the first upper channel feature extraction unit in the upper channel network is used as the input of the upper channel network, and the feature map output by the last upper channel feature extraction unit in the upper channel network is used as the output of the upper channel network.

2. According to claim 1, a model-mixed dual-branch artificial intelligence lung MRI denoising method is characterized in that: Said The input layer includes a plurality of input layer feature extraction units in sequence, and the input layer feature extraction units include a convolution layer, a group normalization layer and an activation layer in sequence; The CNN downsampling layers in the CNN encoder all include convolutional layers, group normalization layers, and activation layers in order; The embedding layer in the Transformer encoder includes multiple embedding layer downsampling layers and one position encoding. The embedding layer downsampling layers include convolutional layers, group normalization layers and activation layers in sequence. The Transformer module includes the first layer normalization, multi-head self-attention, the second layer normalization and the multi-layer perceptron in order. The output of each Transformer module is the word token obtained by adding the input of the second layer normalization and the output of the multi-layer perceptron pixel by pixel in the current Transformer module. The input of the second layer normalization is the token obtained by adding the input of the first layer normalization and the output of the multi-head attention pixel by pixel. The multi-layer perceptron includes the first fully connected layer, the activation layer and the second fully connected layer in order. The Transformer feature processing module in the decoder includes a feature processing module upsampling layer, which includes an upsampling module, a convolution layer, a group normalization layer, and an activation layer. The feature map output by the feature processing module upsampling layer is consistent in size with the feature map output by the corresponding CNN downsampling layer. The first decoder upsampling layer in the decoder includes an upsampling module, a convolutional layer, a group normalization layer, and an activation layer; The decoder upsampling layers other than the first decoder upsampling layer include the decoder upsampling layer fusion module, upsampling module, convolution layer, group normalization layer and activation layer; The output layer consists of a convolutional layer.

3. According to claim 2, a model-mixed dual-branch artificial intelligence lung MRI denoising method is characterized in that: The decoder upsampling layer fusion module includes three branches: The structure of the first branch is the same as that of the second branch, and both include, in sequence, a first fused convolution layer, a first fused group normalization layer, a first fused activation layer, a second fused convolution layer, and a second fused group normalization layer; and, The input of the first branch of the decoder upsampling layer fusion module is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the first branch belongs, and the feature map output by the previous decoder upsampling layer. The feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the first branch belongs and the output of the second fusion group normalization layer of the first branch are summed pixel by pixel and then input into the second fusion activation layer of the first branch; The input of the second branch of the decoder upsampling layer fusion module is the feature map output by the CNN downsampling layer corresponding to the decoder upsampling layer to which the second branch belongs, the feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs, and the feature map output by the previous decoder upsampling layer. The feature map output by the Transformer feature processing module corresponding to the decoder upsampling layer to which the second branch belongs and the output of the second fusion group normalization layer of the second branch are summed pixel by pixel and then input into the second fusion activation layer of the second branch; The input of the third branch of the decoder upsampling layer fusion module is the feature map output by the previous decoder upsampling layer; The feature maps output by the three branches of the decoder upsampling layer fusion module are stacked channel by channel through the fusion layer of the decoder upsampling layer fusion module, and then pass through 1×1 convolution.

4. According to claim 3, a model-mixed dual-branch artificial intelligence lung MRI denoising method is characterized in that: The feature output layer in the decoder includes a feature output layer fusion module and a plurality of feature output layer feature extraction units in order, The feature output layer feature extraction unit includes a convolution layer, a group normalization layer and an activation layer in sequence; The feature output layer fusion module includes two branches: The first branch of the feature output layer fusion module includes a first fusion convolution layer, a first fusion group normalization layer, a first fusion activation layer, a second fusion convolution layer, and a second fusion group normalization layer in sequence. The input of the first branch of the feature output layer fusion module is a result of stacking the feature map output by the input layer and the feature map output by the last level decoder upsampling layer according to channels. The feature map output by the input layer and the feature map output by the second fusion group normalization layer of the first branch of the feature output layer fusion module are summed pixel by pixel and then input into the second fusion activation layer of the first branch of the feature output layer fusion module. The input of the second branch of the feature output layer fusion module is the feature map output by the upsampling layer of the last decoder; The feature maps output by the two branches of the feature output layer fusion module are stacked by channel through the fusion layer and then pass through 1×1 convolution.

5. According to claim 1, a dual-branch artificial intelligence lung MRI denoising method with model mixing is characterized in that: The pre-trained self-supervised loss function L s for: L s =‖(M-m)·[D(m·x)-x]‖ F in,‖·‖ F is the F-norm of the matrix, x is a low signal-to-noise ratio lung MRI image, m is a random Bernoulli matrix, D represents the hybrid model of Transformer and CNN, D(m·x) represents the output of the hybrid model of Transformer and CNN, M is an all-1 matrix, and the operator · refers to the dot product.

6. According to claim 1, a model-mixed dual-branch artificial intelligence lung MRI denoising method is characterized in that: The supervised training loss function L d for: L d =L1[D ′ (x),y] Where L1 is the mean absolute error operation, x is the low signal-to-noise ratio lung MRI image, y is the high signal-to-noise ratio lung MRI image paired with the low signal-to-noise ratio lung MRI image x, and D ′ represents the pre-trained Transformer and CNN hybrid model, D ′ (x) represents the output of the pre-trained Transformer and CNN hybrid model.

7. According to claim 4, a dual-branch artificial intelligence lung MRI denoising method with model mixing is characterized in that: For the upper channel feature extraction unit: the number of input channels of the convolution layer of the first upper channel feature extraction unit is 1, and the number of output channels is greater than the number of input channels; except for the first upper channel feature extraction unit, the number of input channels of the convolution layer of each other upper channel feature extraction unit is equal to the number of output channels; For the input layer feature extraction unit: the number of input channels of the input layer feature extraction unit with the same serial number is consistent with the number of input channels of the upper channel feature extraction unit, and the number of output channels of the input layer feature extraction unit with the same serial number is consistent with the number of output channels of the upper channel feature extraction unit; For CNN downsampling layer: the number of output channels of the convolution layer in the CNN downsampling layer is twice the number of input channels; For the embedding layer downsampling layer: except for the last embedding layer downsampling layer, the number of output channels of the convolution layer is twice the number of input channels; the number of output channels of the convolution layer of the last embedding layer downsampling layer is 3 / 2 of the number of input channels; For the feature output layer feature extraction unit: the number of convolutional layer input channels of the feature output layer feature extraction unit is equal to the number of output channels.

8. According to claim 4, a dual-branch artificial intelligence lung MRI denoising method with model mixing is characterized in that: In the Transformer feature processing module, the number of output channels of the convolution layer of the upsampling layer of the first feature processing module is 2 / 3 of the number of input channels; the number of output channels of the convolution layers of the upsampling layers of other feature processing modules except the upsampling layer of the first feature processing module is 1 / 2 of the number of input channels; The number of output channels of the convolutional layer of the decoder upsampling layer is 1 / 2 of the number of input channels.

Images