An Artifact Removal Method and System Based on Deep Unsupervised Learning

Abstract

The present invention discloses an artifact removal method and system based on deep unsupervised learning, constructs an artifact removal network model based on deep unsupervised learning; uses the full-sampled MRI artifact-containing images and / or full-sampled MRI artifact-free images in multiple full-sampled MRI images to train the artifact removal network model, and obtains a trained artifact removal network model; this model is used to remove the artifacts of full-sampled MRI images. By using the method and system of the present invention, the problems that it is difficult to obtain paired data sets for MRI images and the supervised method requires paired data sets are effectively solved; the difficulty of network artifact removal is effectively reduced; the present invention generates specific masks for different motion artifact-containing images, can remove as many motion artifacts as possible during undersampling while retaining more texture details; at the same time, the present invention combines k-space knowledge and deep learning, gives full play to the advantages of both, and can retain texture details while removing artifacts cleanly.

Description

Technical Field

[0001] The present invention belongs to the technical fields of medical magnetic resonance imaging and deep learning, and particularly relates to an artifact removal method and system based on deep unsupervised learning. Background Art

[0002] Gadoxetic acid disodium is a common liver-specific magnetic resonance contrast agent, which is widely used in liver transplantation tasks, the diagnosis of liver cancer and other diseases. However, although it has advantages in MR imaging, there are still patients who will experience acute transient dyspnea or transient violent movement in the arterial phase after injection, which will cause motion artifacts in the image, reduce the quality, and affect the diagnostic accuracy. Therefore, a suitable motion artifact correction algorithm is needed to remove the artifacts and obtain clear images.

[0003] Currently, there are various traditional methods used to remove motion artifacts. One type is to modify the acquired K-space data to reconstruct artifact-free images. For example, orthogonal K-space data is acquired by exchanging the phase encoding direction, and the motion artifacts are corrected according to the phase difference between the two. This algorithm can significantly reduce motion artifacts; another type is to directly process the reconstructed images with motion artifacts. For example, high-resolution, high-signal-to-noise cardiac magnetic resonance images are generated by registration methods, effectively removing some motion artifacts. Traditional algorithms are based on prior knowledge. Although they can remove some motion artifacts, they have disadvantages such as slow imaging speed and limited motion artifact removal effect.

[0004] In recent years, deep learning has also been applied to motion artifact removal methods. Compared with traditional methods, deep learning methods are data-driven and can implicitly learn prior knowledge to improve the imaging speed. One type is supervised learning methods. These methods can achieve good motion artifact removal effects on simulated data, but due to the lack of paired datasets required for their training in real MRI data, the motion artifact removal effect on real data needs to be improved. Another type is unsupervised methods. These methods solve the problem of the need for paired datasets in supervised methods and can be trained on real data, but the motion artifact removal effect still needs to be improved. Summary of the Invention

[0005] In order to improve the effect of removing motion artifacts, the present invention proposes an artifact removal method and system based on deep unsupervised learning.

[0006] An artifact removal method based on deep unsupervised learning for achieving one of the purposes of the present invention includes the following steps:

[0007] Step (1) Construct an artifact removal network model based on deep unsupervised learning;

[0008] The artifact removal network model includes an MNet network and an artifact-free image generator;

[0009] The MNet network is used to perform downsampling, upsampling, and convolution operations on the fully sampled MRI image to obtain a first parameterized mask M mnet ; The first parameterized mask M mnet is used to perform undersampling on the fully sampled MRI artifact image or the fully sampled MRI artifact-free image in the fully sampled MRI image to obtain an undersampled MRI artifact image or an undersampled MRI artifact-free image;

[0010] The artifact-free image generator is used to perform artifact removal operations on the undersampled MRI artifact image or the undersampled MRI artifact-free image to obtain a fully sampled MRI artifact-free image with artifacts removed;

[0011] Step (2) uses the fully sampled MRI artifact image or the fully sampled MRI artifact-free image in multiple fully sampled MRI images to train the artifact removal network model to obtain a trained artifact removal network model; the artifact removal network model is used to remove artifacts in the fully sampled MRI image to obtain a fully sampled MRI artifact-free image with artifacts removed.

[0012] An artifact removal system based on deep unsupervised learning for achieving the second object of the present invention includes: an artifact removal network model construction module and an artifact removal network model training module;

[0013] The artifact removal network model construction module is used to construct an artifact removal network model based on deep unsupervised learning;

[0014] The artifact removal network model training module is used to train the artifact removal network model with the fully sampled MRI artifact image or the fully sampled MRI artifact-free image in multiple fully sampled MRI images to obtain a trained artifact removal network model; the artifact removal network model is used to remove artifacts in the fully sampled MRI image to obtain a fully sampled MRI artifact-free image with artifacts removed.

[0015] Beneficial effects:

[0016] 1. The present invention uses a dual-domain unsupervised network and trains with an unpaired dataset, effectively training a generator that can restore rich details, effectively solving the problem that it is difficult to obtain a paired dataset for MRI images and the problem that a supervised method requires a paired dataset;

[0017] 2. The present invention uses an adaptive binary parameterized mask to undersample the K-space, remove some difficult-to-remove motion artifacts, and then input the data into the dual-domain unsupervised network to remove artifacts, effectively reducing the difficulty of network artifact removal and improving the effect of removing motion artifacts of the present invention;

[0018] 3. The present invention utilizes the mask_backward module to train the MNet neural network to generate an adaptive binary parameterized mask. This module effectively trains the MNet network to generate specific masks for different motion artifact images, which can remove as many motion artifacts as possible during undersampling while retaining more texture details.

[0019] 4. The present invention combines k-space knowledge and deep learning to give full play to the advantages of both, enabling the removal of artifacts while retaining texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is the overall training flowchart of the present invention;

[0021] Figure 2 is the framework diagram of the mask_backward network of the present invention;

[0022] Figure 3-a is the schematic diagram of the artifact image generator in the mask_backward network of the present invention;

[0023] Figure 3-b is the schematic diagram of the artifact-free image generator in the mask_backward network of the present invention;

[0024] Figure 3-c is the schematic diagram of the cross-image generation in the mask_backward network of the present invention;

[0025] Figure 4 is the structural diagram of the MNet network of the present invention;

[0026] Figure 5 is the structural diagram of the content encoder network of the present invention;

[0027] Figure 6 is the structural diagram of the artifact encoder network of the present invention;

[0028] Figure 7 is the structural diagram of the image generator network of the present invention;

[0029] Figure 8 is the structural diagram of the discriminator network of the present invention;

[0030] Figure 9 is the residual module of the present invention;

[0031] Figure 10 is the effect comparison diagram between the present invention and Cycle-MedGAN. DETAILED DESCRIPTION OF THE INVENTION

[0032] The following specific embodiments are used to explain the technical solutions of the claims of the present invention, so that those skilled in the art can understand this claim book. The protection scope of the present invention is not limited to the following specific implementation structures. Those made by those skilled in the art that include the technical solutions of the claim book of the present invention and are different from the following specific embodiments are also within the protection scope of the present invention.

[0033] In the description of the present invention, the terms "first", "second", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0034] The embodiments of this application include an artifact removal method based on deep unsupervised learning. The specific method includes:

[0035] Step (1) Construct an artifact removal network model based on deep unsupervised learning

[0036] In this embodiment, the artifact removal network model is used to remove artifacts from the fully sampled MRI images of the liver. The artifact removal network model includes an MNet network, an artifact-free image generator, an artifact image generator, and an artifact-free image discriminator D y and an artifact image discriminator D x .

[0037] The MNet network is used to perform downsampling, upsampling, and convolution operations on the fully sampled MRI images to obtain a first parameterized mask M mnet ; the first parameterized mask M mnet is used to perform undersampling operations on the fully sampled MRI images with artifacts or the fully sampled MRI images without artifacts in the fully sampled MRI images to obtain undersampled MRI images with artifacts or undersampled MRI images without artifacts;

[0038] The artifact-free image generator is used to perform artifact removal operations on the undersampled MRI images (including undersampled MRI images with artifacts and undersampled MRI images without artifacts) to obtain fully sampled MRI images without artifacts.

[0039] The artifact-free image generator is as Figure 3-b shown, and includes a first content encoder E F and an artifact-free image decoder G F ; wherein the first content encoder E F is used to encode the content part of the input undersampled MRI image to obtain a content encoding, and the artifact-free image decoder G F is used to decode the input content encoding to obtain a fully sampled artifact-free image. The input content encoding can come from the first content encoder E F , or can come from the second content encoder E C。The input undersampled MRI images include undersampled MRI images with artifacts and undersampled MRI images without artifacts. Among them, the undersampled MRI images with artifacts include a content part and an artifact part, and the content part of the undersampled MRI images without artifacts is the other part of the image that does not include the artifact part. The first content encoder E F only encodes the content part.

[0040] The first content encoder E F has a network structure as Figure 5 shown. An image of 1×128×128 is input, and after three convolution operations, it becomes an image of 256×32×32. Then, it passes through four residual structures. Each residual module contains two convolution operations that do not change the image size. Finally, a content encoding Z of 256×32×32 is output. F 。

[0041] The artifact-free image decoder G F has a network structure as Figure 7 shown. The input is the content encoding Z of 256×32×32. F First, it passes through a large residual module, which contains four small residual modules. Each small residual module contains two convolution operations that do not change the image size, and the output size is 256×32×32. Then, it undergoes two upsampling operations and three convolution operations, and finally outputs a fully sampled artifact-free MRI image of 1×128×128.

[0042] The undersampled MRI images pass through the first content encoder E F to obtain the content information encoding Z. F Z F passes through the artifact-free image decoder G F to obtain a fully sampled artifact-free MRI image.

[0043] The artifact image generator is used to separate and merge the artifact part and the content part of the undersampled MRI images (including undersampled MRI images with artifacts and undersampled MRI images without artifacts) to obtain a fully sampled MRI image with artifacts. The process of separating and merging the artifact part and the content part is to use the encoder in the artifact image generator to encode the artifact part and the content part respectively to obtain a content encoding and an artifact encoding, thereby realizing the separation of the content part and the artifact part. Then, use the decoder in the artifact image generator to re-synthesize a single artifact image from the content encoding and the artifact encoding. Here, the content encoding and the artifact encoding do not necessarily come from the same artifact image, and the content encoding may come from the content encoding output by the artifact-free image generator. The role of the artifact image generator is to better train the artifact-free image generator.

[0044] The artifact image generator is as Figure 3-a shown and includes a second content encoder Ec , the artifact encoder E a and the artifact image decoder G c ; the second content encoder E c has the same network structure as the first content encoder E F , as Figure 5 shown, its input is the undersampled image of 1×128×128, and finally outputs the content information encoding Z of 256×32×32 c ; the network structure of the artifact encoder E a is as Figure 6 shown, including four convolutions, the input is the undersampled image of 1×128×128, and the output is the artifact information encoding Z of 256×32×32 a ; as Figure 7 shown, the network structure of the second artifact-free image decoder G c is similar to that of the artifact-free image decoder G F , the only difference is that the input of the second artifact-free image decoder G c is the encoding structure of 512×32×32, and finally also outputs the image of 1×128×128

[0045] The undersampled MRI image passes through the second content encoder E c to obtain the content information encoding Z of 256×32×32 c , the undersampled MRI image passes through the artifact encoder E A to obtain the artifact information encoding Z of 256×32×32 a . Connect Z c and Z a into the encoding structure of 512×32×32, and the encoding structure of 512×32×32 passes through the artifact image decoder G c to obtain the fully sampled MRI image with artifacts of 1×128×128

[0046] The artifact-free image discriminator D y is used to distinguish whether the input image is the fully sampled MRI artifact-free image obtained by the artifact-free image generator or the real fully sampled MRI artifact-free image. When the input image is the fully sampled MRI artifact-free image obtained by the artifact-free image generator, the artifact-free image discriminator outputs a probability value close to 0. When the input image is the real fully sampled MRI artifact-free image, it outputs a probability value close to 1, and the two types of images are distinguished according to the numerical value of the output probability

[0047] The artifact image discriminator D xUsed to identify whether the input image is a full-sampled MRI artifact image obtained by an artifact image generator or a real full-sampled MRI artifact image. When the input image is a full-sampled MRI artifact image obtained by an artifact image generator, the artifact image discriminator outputs a probability value close to 0; when the input image is a real full-sampled MRI artifact image, it outputs a probability value close to 1. The two types of images are identified according to the numerical size of the output probability.

[0048] Artifact-free image discriminator D y and the artifact image discriminator D x have the same network structure, as Figure 8 shown. The size of the input image is 1×128×128. After two downsamplings, data of two sizes, 1×64×64 and 1×32×32, can be obtained. The above three pieces of data can respectively obtain three pieces of data, 1×8×8, 1×4×4, and 1×2×2, after 5 convolutions. The three pieces of data are arranged in the above order in turn to form a list. After taking the mean of each of the three pieces of data in the list, and then taking the mean of the three mean results, the finally obtained mean is the output of the discriminator.

[0049] Step (2) Construct a liver MRI full-sampled image dataset

[0050] The liver full-sampled MRI (Magnetic Resonance Imaging) image dataset includes a liver full-sampled MRI image training dataset and a liver full-sampled MRI image test dataset; the specific process is as follows:

[0051] Step (2.1) Divide the training set and the test set

[0052] The dataset in this embodiment contains 300 mat data (a data type used for processing in matlab), among which 250 are mat data with artifacts and 50 are mat data without artifacts; 200 mat data are randomly selected from the 250 mat data with artifacts, and 30 mat data are randomly selected from the 50 mat data without artifacts as the initial liver full-sampled MRI image training dataset; the remaining mat data are used as the liver MRI full-sampled image test dataset. Each mat data contains 60 - 80 liver full-sampled MRI slice images with a size of 320×256.

[0053] Step (2.2) Preprocess the training set data

[0054] Crop each slice of the initial liver full-sampled MRI image training dataset, and crop the liver full-sampled MRI slice image with a size of 320×256 to a size of 128×128 to obtain the liver full-sampled MRI image training dataset.

[0055] The technical effect of this step is to remove redundant boundary useless information to accelerate the training speed of the undersampled dual-domain unsupervised artifact removal network model in the subsequent steps.

[0056] The fully sampled liver MRI image test dataset is not subjected to any further processing.

[0057] Step (3) trains the undersampled dual-domain unsupervised artifact removal network

[0058] The training process is as Figure 1 shown, where X represents the fully sampled MRI artifact image, Y represents the fully sampled MRI artifact-free image, and M mnet represents the parameterized mask M generated by the MNet network mnet , and after binarization, the binary mask M refined represents the optimized mask after being optimized by the mask_backward module, and after binarization, the binary optimized mask M rand represents the randomly generated random mask image. After binarizing M rand , the binary random mask is obtained Q represents the quality function value. The smaller the value, the better the effect of the corresponding mask. Q1, Q2, and Q3 respectively represent the corresponding Q values. The following will describe the specific implementation of training the model step by step:

[0059] Step (3.1) calculates the first parameterized mask M mnet and the binary mask

[0060] The method for calculating the first parameterized mask M mnet includes:

[0061] Randomly arrange the fully sampled MRI artifact images and fully sampled MRI artifact-free images in the liver fully sampled MRI image training dataset described in step (2), and take out one fully sampled MRI artifact image X1 and one fully sampled MRI artifact-free image Y1; input the fully sampled MRI artifact image X1 into the Mnet network. As Figure 4 shown in the network structure of MNet, input a single fully sampled MRI artifact image X1 with a size of 1×128×128 (number of channels × length × width, the same for the following dimensions) into the Mnet network, and convert the size to 2048×8×8 through four downsampling operations; then convert it to 128×128×128 through four upsampling operations; finally, obtain the first parameterized mask M mnet through two convolution operations. Its size is 1×128×128, and the pixel values are all between 0 and 1;

[0062] Method for calculating a binary mask includes:[[]]

[0063] Performing the following binarization operation on the first parameterized mask M mnet to obtain a binary mask with a sampling rate between 0.3 and 0.4

[0064] When the pixel value is greater than 0.5, the pixel value is set to 1, and when it is less than or equal to 0.5, it is set to 0. Its size is the same as the input image. In this embodiment, the size is 1×128×128.[[]]

[0065] The above-mentioned sampling rate means that there are only two values, 0 and 1, in the binary mask, and the proportion of 1 is the sampling rate.[[]]

[0066] Step (3.2) Generate an undersampled image with artifacts and an undersampled artifact-free image

[0067] Step (3.2.1) Update the pixel values of 0 in the binary mask to -0.1 and the pixel values of 1 to 0.1 to obtain a second parameterized mask M' mnet ;

[0068] Step (3.2.2) Perform a Fourier transform on the fully sampled MRI image with artifacts X1 to obtain the k-space data of the fully sampled image with artifacts Multiply pixel by pixel with the binary mask to obtain the k-space data of the undersampled image with artifacts Perform an inverse Fourier transform on to obtain the undersampled image with artifacts

[0069] The difference between the undersampled image and the fully sampled image is mainly manifested in the k-space data. In the k-space data of the fully sampled image, the data value of each point is not 0, while in the undersampled image, the data value of some points is 0.[[]]

[0070] Perform a Fourier transform on the fully sampled MRI artifact-free image Y1 to obtain the k-space data of the fully sampled artifact-free image Multiply pixel by pixel with the binary mask to obtain the k-space data of the undersampled artifact-free image Perform an inverse Fourier transform on to obtain the undersampled artifact-free image

[0071] The purpose of the above undersampling operation is to remove data related to motion artifacts.

[0072] The above parametric mask M mnet , the binary mask the fully sampled images X1 and Y1, the undersampled image and are input into the mask_backward module to update the above generator, discriminator, and the first parametric mask M mnet , to obtain the optimized mask M refined . This process will be described in steps (3.3) to (3.5) next.

[0073] Step (3.3) Update the artifact image discriminator D x and the artifact-free image discriminator D y Parameter step (3.3.1) Generate the fully sampled artifact-free image and the fully sampled artifact image

[0074] As Figure 3-c shown in the schematic diagram, the undersampled artifact image (i.e., Figure 3-c x in 1 ) is input into the content encoder E of the artifact image generator c to obtain the content encoding (i.e., Figure 3-c Z in c );The content encoding is input into the artifact-free image decoder G of the artifact-free image generator F to obtain the fully sampled artifact-free image

[0075] The undersampled artifact-free image ( Figure 3-c y in 1 ) is input into the content encoder E in the artifact-free image generator F to obtain the content encoding ( Figure 3-c Z in F );

[0076] The undersampled artifact image is input into the artifact encoder E of the artifact image generator a to obtain the artifact encoding ( Figure 3-c Z in A );

[0077] The content encoding and the artifact encoding The artifact image decoder G in the spliced input artifact image generator C Obtain the fully sampled artifact image The content encoding here And the artifact encoding The structures of these two encodings are the same. In this embodiment, they are both 256*32*32, and here they will be spliced into 512*32*32 according to the first dimension.

[0078] Step (3.3.2) Obtain the discrimination result

[0079] Input the fully sampled artifact image Into the artifact image discriminator D x Obtain the discrimination result

[0080] Input the initial fully sampled MRI artifact image X1 into the artifact image discriminator D x Obtain the discrimination result D x (x1);

[0081] Input the fully sampled artifact-free image Into the artifact-free image discriminator D y Obtain the discrimination result

[0082] Input the fully sampled MRI artifact-free image Y1 into the artifact-free image discriminator D y Obtain the discrimination result D y (y1);

[0083] Step (3.3.3) Calculate the discriminator loss L according to the following formula dis :

[0084]

[0085] Where E[·] represents taking the expected value.

[0086] Step (3.3.4) Use the loss L dis To update the parameters of the artifact image discriminator D x And the artifact-free image discriminator D y In this embodiment, through the derivative chain rule, calculate the derivatives of the loss function with respect to the parameters of the artifact image discriminator network and the artifact-free image discriminator network, and use the Adam optimizer to update the network parameters with an initial learning rate of 0.0001 and a learning rate that is halved every 10,000 rounds.

[0087] Step (3.3.5) Repeat the above steps (3.3.1) to 3.3.4) multiple times. In this embodiment, it is twice.

[0088] Step (3.4) Update the network parameters of the artifact image generator and the artifact-free image generator

[0089] Step (3.4.1) Input the undersampled artifact image described in step (3.2.2) into the artifact image generator to obtain the fully sampled reconstructed artifact image x rec ; Input the undersampled artifact-free image described in step (3.2.2) into the artifact-free image generator to obtain the fully sampled reconstructed artifact-free image y rec ;

[0090] Step (3.4.2) Perform the undersampling operation described in step (3.2.2) on the fully sampled artifact-free image and the fully sampled artifact image respectively to obtain the undersampled artifact-free image and the undersampled artifact image Perform the operations shown in step (3.3.1) above on the undersampled artifact-free image and the undersampled artifact image to obtain the fully sampled cyclic artifact-free image y cyc and the fully sampled cyclic artifact image x cyc .

[0091] Step (3.4.3) Calculate the total loss L gen

[0092] (1) Calculate the reconstruction loss L according to the following formula rec

[0093] L rec = E[||x1 - x rec ||1] + E[||y1 - y rec ||1] Formula (2)

[0094] In the formula:

[0095] E[·] represents taking the expected value;

[0096] ||·||1 represents taking the first norm;

[0097] x1 represents the fully sampled MRI artifact image X1;

[0098] y1 represents the fully sampled MRI artifact-free image Y1;

[0099] x rec represents the fully sampled reconstructed artifact image x obtained in step (3.4.1) rec ;

[0100] y recDenote the fully sampled reconstructed artifact-free image y obtained in step (3.4.1) rec ;

[0101] (2) Calculate the content loss L according to the following formula content

[0102]

[0103] In the formula:

[0104] denotes the sum from i = 1 to i = n (where n represents the number of matrix columns of the corresponding image represented by the matrix); the image is X1, Y1,

[0105] ·||1 represents calculating the first norm;

[0106] sum i represents the summation function;

[0107] X1 represents the initial fully sampled MRI image with artifacts X1;

[0108] Y1 represents the initial fully sampled MRI image without artifacts Y1;

[0109] denotes the fully sampled artifact-free image

[0110] denotes the fully sampled image with artifacts

[0111] (3) Calculate the cycle loss L according to the following formula cyc :

[0112] L cyc = E[||X1 - x cyc ||1] + E[||Y1 - y cyc ||1] Formula (4)

[0113] In the formula:

[0114] E[·] represents calculating the expected value;

[0115] ||·||1 represents calculating the first norm;

[0116] X1 represents the initial fully sampled MRI image with artifacts X1;

[0117] Y1 represents the initial fully sampled MRI image without artifacts Y1;

[0118] x cyc denotes the fully sampled cyclic artifact image x obtained in step (3.4.2)cyc ;

[0119] y cyc represents the fully sampled cyclic artifact-free image y obtained in step (3.4.2) cyc .

[0120] (4) Calculate the adversarial loss L adv

[0121] Calculate the adversarial loss L according to the following formula adv :

[0122]

[0123] where E[·] represents the expectation value, and D x (·) and D y (·) represent the output values of the corresponding discriminators

[0124] (5) Calculate the total loss L according to the following formula gen

[0125] L gen = λ1L adv + λ2L rec + λ3L cyc + λ4L content Formula (6)

[0126] In the formula: the corresponding weights are determined according to the loss magnitude, with the aim of keeping all losses at the same order of magnitude; in this embodiment, λ1 = λ2 = λ3 = 10, and λ4 = 0.5

[0127] In step (3.4.5), update the network parameters of the artifact-free image generator and the artifact image generator. In this embodiment, by using the derivative chain rule, calculate the derivatives of the loss function with respect to the parameters of the artifact-free image generator network and the artifact image generator network, and use the Adam optimizer to update the network parameters with an initial learning rate of 0.0001 and a learning rate that is halved every 10,000 rounds gen

[0128] Step (3.5) Update the second parameterized mask M' mnet to obtain the optimized mask M refined , which specifically includes the following steps

[0129] Step (3.5.1) Calculate the quality function Q1; the specific steps include

[0130] (1), Input the fully sampled artifact-free image described in step (3.3.1) into the artifact image discriminator D x to obtain the probability value ​Input the fully sampled artifact image described in step (3.3.1) into the artifact-free image discriminator D y to obtain a probability value

[0131] (4) Calculate the binary mask quality function Q1 according to the following formula

[0132]

[0133] In the formula

[0134] D x (x1) represents the discrimination result obtained by inputting the initial fully sampled MRI artifact image X1 into the artifact image discriminator D x ;

[0135] D y (y1) represents the discrimination result obtained by inputting the fully sampled MRI artifact-free image Y1 into the artifact-free image discriminator D y ;

[0136] In step (3.5.2), calculate the sampling rate loss L rate and the mask change loss L refine :

[0137]

[0138]

[0139] where sum(·) is the element-wise summation function, and M and N are respectively the number of rows and columns of is to calculate the binary cross-entropy

[0140] In step (3.5.3), calculate the second adversarial loss L' adv :

[0141]

[0142] In the formula

[0143] Fully sampled artifact image Input into the artifact image discriminator D x to obtain the discrimination result

[0144] D x (x1): The discrimination result obtained by inputting the initial fully sampled MRI artifact image X1 into the artifact image discriminator D x ;

[0145] Full-sampling artifact-free image Input the artifact-free image discriminator D y The obtained discrimination result;

[0146] D y (y1): The full-sampling MRI artifact-free image Y1 is input into the artifact-free image discriminator D y The obtained discrimination result.

[0147] Step (3.5.4) calculates the updated parameterized mask M according to the following formula mnet The total loss L of mask :

[0148] L mask = w1L′ adv + w2L rate + w3L refine (Formula 11)

[0149] In the formula: w1, w2, w3 are determined according to the loss magnitude to keep all losses at the same order of magnitude; in this embodiment, w1 = 1, w2 = w3 = 0.1;

[0150] Step (3.5.5) uses the loss L obtained in step (3.5.4) mask To update the second parameterized mask M' mnet To obtain the optimized mask M refined ; in this embodiment, through the derivative chain rule, calculate the derivatives of the loss function with respect to each parameter of the parameterized mask M' mnet Use the Adam optimizer with a learning rate of 0.0001 to update each parameter of M' mnet To obtain the optimized mask M refined .

[0151] Step (3.5.5) repeats steps (3.5.1) to (3.5.4) multiple times (such as 9 times).

[0152] Step (3.6) determines the reference mask for training the Mnet network Specifically, it includes the following steps:

[0153] Step (3.6.1) calculates Q2, specifically including the following steps:

[0154] (1) After binarizing M refined To obtain the mask

[0155] (2) Use To undersample the full-sampling MRI artifact-containing image X1 and the full-sampling MRI artifact-free image Y1 to obtain the undersampled artifact-containing image And the undersampled artifact-free image

[0156] (3) Input the under-sampled artifact image into the content encoder E of the artifact image generator c to obtain the content encoding Input the content encoding into the artifact-free image decoder G of the artifact-free image generator F to obtain the fully-sampled artifact-free image

[0157] Input the under-sampled artifact-free image into the content encoder E in the artifact-free image generator F to obtain the content encoding

[0158] Input the under-sampled artifact image X'1 into the artifact encoder E of the artifact image generator a to obtain the artifact encoding

[0159] Input the content encoding and the artifact encoding after addition into the artifact-free image decoder G in the artifact-free image generator F to obtain the fully-sampled artifact image

[0160] (4) Calculate the quality function Q2 according to the following formula:

[0161]

[0162] Step (3.6.2) calculates Q3, which specifically includes the following steps:

[0163] (1) Use a random mask to undersample the fully-sampled MRI artifact image X1 and the fully-sampled MRI artifact-free image Y1 to obtain the under-sampled artifact image and the under-sampled artifact-free image where is a randomly generated binary mask;

[0164] (2) Input the under-sampled artifact image into the content encoder E of the artifact image generator c to obtain the content encoding Input the content encoding into the artifact-free image decoder G of the artifact-free image generator F to obtain the fully-sampled artifact-free image

[0165] Input the under-sampled artifact-free image Input the content encoder E in the artifact-free image generator F Obtain the content encoding

[0166] Input the undersampled artifact image into the artifact encoder E of the artifact image generator a Obtain the artifact encoding

[0167] Add the content encoding and the artifact encoding and input the sum into the artifact-free image decoder G in the artifact-free image generator F Obtain the fully sampled artifact image

[0168] (3) Calculate the quality function Q3 according to the following formula:

[0169]

[0170] Step (3.6.3) Determine the reference mask

[0171] Compare Q1, Q2 and Q3; if Q2 is the largest, then is used as the reference mask If Q3 is the largest, then is used as the reference mask If Q1 is the largest, it means that a better reference mask has not been found in this training Directly skip step (3.7) and enter the next round;

[0172] Step (3.7) Train the MNet network according to the reference mask Specifically, it includes the following steps:

[0173] Step (3.7.1) Input the initial fully sampled MRI artifact image X1 into the MNet network to obtain the parameterized mask M” mnet , and calculate the loss function L according to the following formula mnet :

[0174]

[0175] Among them, is to calculate the binary cross entropy

[0176] Step (3.7.2) Use the loss function L mnet to update the network parameters of the network MNet: Through the derivative chain rule, calculate the derivative of the loss function L mnet with respect to each parameter of the MNet network, and use the Adam optimizer to update each network parameter with an initial learning rate of 0.0001 and a learning rate that is halved every 10,000 rounds.

[0177] Step (3.8) takes the next full-sampled MRI artifact image from the sequence of full-sampled MRI artifact images arranged in step (3.1), and takes the next full-sampled MRI artifact-free image from the sequence of full-sampled MRI artifact-free images arranged in step (3.1), and uses the MNet after updating each parameter in step (3.7.2) to generate a parameterized mask M mnet 。

[0178] Step (3.9) repeats the operations from (3.2) to (3.8) multiple times (e.g., 50,000 times) to obtain a trained undersampled dual-domain unsupervised artifact removal network.

[0179] Step (4) uses the trained undersampled dual-domain unsupervised artifact removal network in step (3) to test the full-sampled MRI image test dataset in step (1) to obtain the final artifact removal test result. The specific steps include:

[0180] Input the data in the full-sampled MRI image test dataset into the content encoder E in the artifact image generator trained in step (3) in sequence c to obtain the content encoding z c , and then input the content encoding z c into the artifact-free image decoder G of the artifact-free image generator F to obtain the artifact removal corrected image.

[0181] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0182] Through experimental demonstration, the method described in the present invention can retain more texture details while effectively removing MRI motion artifacts.

[0183] In the comparative experiment, the existing Cycle-MedGAN based on Cycle GAN proposed by Armanious et al. is selected as the comparative method. To ensure the fairness of the comparative experiment, both methods are experimented under the same software and hardware environment.

[0184] Software and hardware environment for the comparative experiment: Pytorch framework; Windows 10 system; Intel i7-8700K CPU; 32GB memory; NVIDIA GTX 1080Ti graphics card;

[0185] The convolutional neural network-based algorithms for comparison are:

[0186] The Cycle-MedGAN method proposed by Armanious et al., reference: Armanious, K., Tanwar, A., Abdulatif S., Kustner, T., Gatidis, S., Yang, B. (2020) Unsupervised adversarial correction of rigid MR motion artifacts. In: IEEE 17th International Symposium on Biomedical Imaging (ISBI). Iowa. pp. 1494-1498.

[0187] Evaluation metrics: The present invention evaluates images from two aspects: the degree of artifact removal and the degree of detail preservation.

[0188] Figure 10 The artifact removal results of the present invention and the Cycle-MedGAN network are shown, where the selected images are representative images. Compared with the Cycle-MedGAN network, the present invention has a better artifact removal effect and can display a clearer liver image. Moreover, in terms of detail restoration, as Figure 10 shown in the second row of pictures, it can be seen that there are severely overlapping motion artifacts in the original artifact image, and there are still slight motion artifacts in the image corrected by the Cycle-MedGAN method. The motion artifacts cover most of the lesions, reducing the image resolution and affecting the diagnosis of the condition. However, in the image corrected by the undersampling dual-domain unsupervised network of the present invention, the motion artifacts are basically completely eliminated, and the lesions can be clearly displayed. In summary, the present invention is an effective undersampling artifact removal method based on deep unsupervised learning.

[0189] The embodiment of the present application further includes an artifact removal system based on deep unsupervised learning, characterized by including an artifact removal network model construction module and an artifact removal network model training module;

[0190] The artifact removal network model construction module is used to construct an artifact removal network model based on deep unsupervised learning;

[0191] The artifact removal network model training module is used to collect multiple data pairs including artifacts and non-artifacts to train the model and obtain a trained artifact removal network model; the model is used to remove the artifacts of the MRI full-sampling image.

[0192] In the above system, the artifact removal network model includes an MNet network and an artifact-free image generator;

[0193] The MNet network is used to perform downsampling, upsampling, and convolution operations on the fully sampled MRI image to obtain the first parameterized mask M mnet , which is used to perform undersampling operations on the fully sampled MRI image with artifacts or the fully sampled MRI image without artifacts to obtain the undersampled MRI image with artifacts or the undersampled MRI image without artifacts;

[0194] The artifact-free image generator is used to perform artifact removal operations on the undersampled MRI image with artifacts or the undersampled MRI image without artifacts to obtain the fully sampled MRI image without artifacts.

[0195] The content not detailed in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. An artifact removal method based on deep unsupervised learning, characterized in that, It includes the following steps: S1. Construct an artifact removal network model based on deep unsupervised learning; The artifact removal network model includes an MNet network and an artifact-free image generator; The MNet network is used to perform downsampling, upsampling, and convolution operations on the fully sampled MRI image to obtain the first parameterized mask M mnet ; The first parameterized mask M mnet is used to perform undersampling operations on the fully sampled MRI image with artifacts or the fully sampled MRI image without artifacts in the fully sampled MRI image to obtain an undersampled MRI image with artifacts or an undersampled MRI image without artifacts; The artifact-free image generator is used to perform artifact removal operations on undersampled MRI images with artifacts or undersampled MRI images without artifacts, and obtain full-sampled MRI images without artifacts after removing the artifacts; S2. Use the undersampled MRI images with artifacts and / or undersampled MRI images without artifacts in multiple full-sampled MRI images to train the artifact removal network model, and obtain a trained artifact removal network model; the artifact removal network model is used to remove the artifacts in the full-sampled MRI images and obtain full-sampled MRI images without artifacts after removing the artifacts; The artifact-free image generator includes a first content encoder E F and an artifact-free image decoder G F ; The first content encoder E F is used to encode the content part of the input undersampled MRI artifact image or undersampled MRI artifact-free image to obtain a content encoding; The artifact-free image decoder G F is used to decode the input content encoding to obtain a fully sampled MRI artifact-free image; The content part of the undersampled MRI artifact image or undersampled MRI artifact-free image is the other part of the image that does not include the artifact part; The artifact removal network model further includes an artifact-free image discriminator D y , which is used to distinguish whether the input image is the fully sampled artifact-free MRI image output by the artifact-free image generator or the real fully sampled artifact-free MRI image. Its function is to improve the output accuracy of the artifact-free image generator, so that the output of the artifact-free image generator is more inclined to the image of the real image; The artifact removal network model further includes an artifact image generator, which is used to separate and merge the artifact part and the content part of the undersampled MRI images with artifacts or undersampled MRI images without artifacts, and obtain full-sampled MRI images with artifacts.

2. The artifact removal method based on deep unsupervised learning according to claim 1, wherein The first content encoder E F For the input image of 1×128×128, through multiple convolution operations, it becomes an image of 256×32×32; the image of 256×32×32 is successively input into multiple residual structures for convolution operations and then outputs the content encoding Z of 256×32×32 F .

3. The artifact removal method based on deep unsupervised learning according to claim 1, characterized in that, The artifact image generator includes a second content encoder E c , an artifact encoder E a and an artifact image decoder G c ; The second content encoder E c performs multiple convolution operations on the input 1×128×128 undersampled MRI with artifact image or undersampled MRI without artifact image, and outputs the content information encoding Z of 256×32×32 c ; The artifact encoder E a performs multiple convolution operations on the input 1×128×128 undersampled MRI with artifact image or undersampled MRI without artifact image, and outputs the artifact information encoding Z of 256×32×32 a ; Second artifact-free image decoder G c After performing multiple upsampling and convolution operations on the input image of 512×32×32, it outputs a fully sampled MRI artifact image of 1×128×128.

4. The artifact removal method based on deep unsupervised learning according to claim 1, characterized in that, The artifact removal network model further includes an artifact image discriminator D x , which is used to distinguish whether the input image is a full-sampled MRI artifact image output by the artifact image generator or a real full-sampled MRI artifact image.

5. The artifact removal method based on deep unsupervised learning according to claim 1, characterized in that According to the first parameterized mask M mnet A method for performing an undersampling operation on an aliased full-sampling MRI image or a non-aliased full-sampling MRI image includes: For the first parameterized mask M mnet The binarized mask is obtained through the following binarization operation : When the pixel value is greater than 0.5, the pixel value is set to 1, and when it is less than or equal to 0.5, it is set to 0; The fully sampled MRI artifact image X1 is Fourier-transformed to obtain the fully sampled artifact image represented by k-space data ; Multiply pixel by pixel with the binary mask to obtain the undersampled artifact image represented by k-space data ; Perform inverse Fourier transform on to obtain the undersampled artifact image ; The artifact-free full-sampled MRI image Y1 is Fourier-transformed to obtain the artifact-free full-sampled image represented by k-space data ; Multiply pixel by pixel with the binary mask to obtain the undersampled artifact-free image represented by k-space data , and perform inverse Fourier transform on to obtain the undersampled artifact-free image .

6. A de-artifacting system based on deep unsupervised learning, which is used to implement the de-artifacting method based on deep unsupervised learning described in claim 1, and is characterized in that, It includes an artifact removal network model construction module and an artifact removal network model training module; The artifact removal network model construction module is used to construct an artifact removal network model based on deep unsupervised learning; The artifact removal network model training module is used to train the artifact removal network model with the undersampled MRI images with artifacts or undersampled MRI images without artifacts in multiple full-sampled MRI images, and obtain a trained artifact removal network model; the artifact removal network model is used to remove the artifacts in the full-sampled MRI images and obtain full-sampled MRI images without artifacts after removing the artifacts.

7. The artifact removal system based on deep unsupervised learning according to claim 6, wherein The artifact removal network model includes an MNet network and an artifact-free image generator; The MNet network is used to perform downsampling, upsampling, and convolution operations on the fully sampled MRI image to obtain the first parameterized mask M mnet ; The first parameterized mask M mnet is used to perform undersampling operations on the fully sampled MRI artifact image or the fully sampled artifact-free MRI image in the fully sampled MRI image to obtain an undersampled MRI artifact image or an undersampled artifact-free MRI image; The artifact-free image generator is used to perform artifact removal operations on undersampled MRI images with artifacts or undersampled MRI images without artifacts, and obtain full-sampled MRI images without artifacts after removing the artifacts.

Images