Metal artifact correction network model and metal artifact removal method thereof

Abstract

The invention relates to a metal artifact correction network model and a metal artifact removal method thereof, and the metal artifact correction network model is provided with a learnable artifact encoder which is used for carrying out extraction and feature coding on artifact information in an uncorrected metal artifact-containing CT image to obtain a corresponding multi-scale artifact coding feature map; the learnable artifact remover is used for removing artifact information in the uncorrected CT image containing the metal artifacts; the learnable artifact generator is used for generating artifact information on an artifact-free CT image paired with the uncorrected metal artifact-containing CT image; the network structure of the learnable artifact remover is the same as that of the learnable artifact generator. According to the invention, a metal artifact-removed image with a good artifact correction effect can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of medical imaging quality control, and in particular to a metal artifact correction network model and a metal artifact removal method thereof. Background Art

[0002] Computed Tomography (CT) is an imaging technology that obtains cross-sectional structural images of objects in a non-invasive and non-open injury manner. Due to its fast imaging speed and high resolution, CT has been widely used in medical imaging diagnosis and has gradually become the preferred method for modern clinical medical imaging examinations. However, during the CT scanning process, the high atomic number metal fillers implanted in patients will cause severe attenuation of X-rays. At the same time, due to the influence of various physical effects such as X-ray beam hardening, scattering, and photon starvation, the signal received by the detector will jump and change, resulting in discontinuous projection data. These abnormal signals will cause light and dark stripe artifacts in the reconstructed image, which are called metal artifacts.

[0003] Existing metal artifact correction methods can be roughly divided into three categories: projection completion method, iterative reconstruction method, and deep learning method. The first category, the projection completion method, refers to interpolating the metal projection trajectory to restore the projection data damaged by metal in the CT projection, and filtering the corrected projection to reconstruct the image without metal artifacts. The restoration operation of this method is usually simple and effective, but because the hidden structural information in the metal projection trajectory is discarded, it often causes the details of the structure around the metal to be too smooth, especially when the metal shape is large or the number is large. The second category, the basic idea of ​​iterative reconstruction is to obtain high-quality images by iteratively solving the optimal solution of the optimization model. There are two methods for metal artifact correction based on iterative reconstruction: one is to use the projection data of the missing metal trajectory to iteratively reconstruct the CT image, and the other is to add a constrained regularization term in the iterative optimization process to remove the influence of metal artifacts during iterative reconstruction. This method reconstructs high-quality images, but it requires multiple front projection and reconstruction operations, which is computationally intensive and time-consuming, making it very difficult to promote its practical application. The third category, deep learning, due to its powerful feature learning ability, is gradually applied to MAR tasks.

[0004] At present, the MAR algorithm based on deep learning mainly uses supervised learning, aiming to train the transformation between artifact images or projections and artifact-free images or projections. Supervised networks are trained using simulated paired data. Although the effect is stable, it requires a large number of labeled data sets, and the performance will degrade in practical applications. Unsupervised learning uses unpaired clinical data to train the network in an adversarial manner, which can avoid the domain difference problem caused by simulation, but there is a problem of unstable network training, and the quality of the test results is uncertain. Semi-supervised MAR methods usually train network models on simulated paired data and fine-tune them using clinical unpaired data. Semi-supervised methods try to find a balance between simulation and clinical data. Although they combine the advantages of unsupervised and supervised training methods, they are prone to unsatisfactory results in both domains. In addition, the introduction of unpaired data also brings uncertainty and potential dangers to network learning.

[0005] Therefore, in view of the shortcomings of the prior art, it is necessary to provide a metal artifact correction network model and a metal artifact removal method thereof to solve the shortcomings of the prior art. Summary of the invention

[0006] The first object of the present invention is to avoid the shortcomings of the prior art and provide a metal artifact correction network model. The metal artifact correction network model obtains a metal artifact-free image with a good artifact correction effect.

[0007] The above-mentioned purpose of the present invention is achieved by the following technical measures:

[0008] A metal artifact correction network model is provided, which is provided with:

[0009] Learnable artifact encoder: extracts and features artifact information from uncorrected CT images containing metal artifacts, and obtains corresponding multi-scale artifact coding feature maps;

[0010] Learnable Artifact Remover - removes artifact information from uncorrected CT images containing metal artifacts;

[0011] A learnable artifact generator for generating artifact information on an artifact-free CT image paired with the uncorrected metal artifact-containing CT image;

[0012] The network structure of the learnable artifact remover is the same as the network structure of the learnable artifact generator.

[0013] Preferably, the network structure of the learnable artifact remover and the learnable artifact generator are both U-net structures with skip links.

[0014] Preferably, the above-mentioned learnable artifact remover and the learnable artifact generator both include eight second residual modules, four second downsampling blocks, four spatial self-attention modules, four first upsampling blocks for artifact encoding operations and four second upsampling blocks.

[0015] Preferably, the above-mentioned learnable artifact encoder includes five first downsampling blocks, a deep convolutional layer, three first residual modules and three convolutional modules.

[0016] Preferably, the spatial self-attention module SAM uses the multi-scale artifact coding feature map obtained by the learnable artifact encoder to scale and modulate the feature map of the jump connection.

[0017] Preferably, the downsampling operator in the first downsampling block is a convolution operation with a step size equal to 2.

[0018] The downsampling operator in the second downsampling block is a convolution operation with a step size equal to 2.

[0019] Preferably, the upsampling operation in the first upsampling block is implemented by interpolation through an Upsample function.

[0020] Preferably, the upsampling operator in the above-mentioned second upsampling block is a deconvolution operation.

[0021] Preferably, the normalization function of the above-mentioned learnable artifact encoder is SpectralNorm, and the activation function is LeakyRelu.

[0022] Preferably, the normalization functions of the learnable artifact remover and the learnable artifact generator are both InstanceNorm, and the activation functions are both LeakyRelu.

[0023] Preferably, the first residual module includes a convolutional layer and an activation function.

[0024] Preferably, the second residual module includes a convolutional layer, a normalization function and an activation function.

[0025] Respectively in the learnable artifact remover and the learnable artifact generator:

[0026] The four first upsampling blocks are defined as a first upsampling block 1, a first upsampling block 2, a first upsampling block 3 and a first upsampling block 4 from front to back according to the transmission direction, and the first upsampling block 1, the first upsampling block 2, the first upsampling block 3 and the first upsampling block 4 are connected in sequence, and the multi-scale artifact coding feature map of the learnable artifact encoder is input from the first upsampling block 1;

[0027] Four of the second residual modules are respectively concatenated with four corresponding second up-sampling blocks through a concatenation function Concat to obtain four first concatenated blocks, wherein the four first concatenated blocks are defined as a first concatenated block 1, a first concatenated block 2, a first concatenated block 3 and a first concatenated block 4 from front to back according to a transmission direction, and the first concatenated block 1, the first concatenated block 2, the first concatenated block 3 and the first concatenated block 4 are connected in sequence;

[0028] Splicing the other four second residual modules with the four second downsampling blocks respectively to obtain four second splicing blocks; the four second splicing blocks are defined as second splicing block 1, second splicing block 2, second splicing block 3 and second splicing block 4 from front to back according to the transmission direction; and the second splicing block 1, the second splicing block 2, the second splicing block 3 and the second splicing block 4 are connected in sequence, and the output of the second splicing block 4 is also transmitted to the first splicing block 1;

[0029] The four spatial self-attention modules are defined as spatial self-attention module 1, spatial self-attention module 2, spatial self-attention module 3 and spatial self-attention module 4 respectively;

[0030] The output of the first upsampling block 1 and the output of the second splicing block 4 are respectively input into the spatial self-attention module 1, and the output of the spatial self-attention module 1 is input into the first splicing block 1;

[0031] The output of the first upsampling block 2 and the output of the second splicing block 3 are respectively input into the spatial self-attention module 2, and the output of the spatial self-attention module 2 is input into the first splicing block 2;

[0032] The output of the first upsampling block 3 and the output of the second splicing block 2 are respectively input into the spatial self-attention module 3, and the output of the spatial self-attention module 3 is input into the first splicing block 3;

[0033] The output of the first upsampling block 4 and the output of the second splicing block 1 are respectively input into the spatial self-attention module 4 , and the output of the spatial self-attention module 4 is input into the first splicing block 4 .

[0034] When in the learnable artifact remover, the uncorrected CT image containing metal artifacts is input from the residual module side of the second splicing block 1, the output of the first splicing block 4 is added to the uncorrected CT image containing metal artifacts to output an artifact-free CT image.

[0035] When in the learnable artifact generator, the artifact-free CT image is input from the residual module side of the second splicing block 1, the output of the first splicing block 4 is added to the artifact-free CT image to output the artifact-added CT image.

[0036] In the learnable artifact encoder, an uncorrected CT image containing metal artifacts enters from one of the first downsampling blocks, then is input into the deep convolution layer, then is sequentially input into another four first downsampling blocks, then is sequentially input into three of the first residual modules, then is sequentially input into three of the convolution modules, and finally is output from the last convolution module to obtain a multi-scale artifact coding feature map.

[0037] The second objective of the present invention is to provide a metal artifact removal method to avoid the shortcomings of the prior art. The metal artifact removal method can obtain a metal artifact-free image with a better artifact correction effect.

[0038] The above-mentioned purpose of the present invention is achieved by the following technical measures:

[0039] A metal artifact removal method is provided, which is performed using the metal artifact correction network model.

[0040] The metal artifact removal method of the present invention is performed by the following steps:

[0041] S1. Input the metal artifact-removing CT image into the trained learnable artifact encoder to obtain the artifact encoding feature map F(X a );

[0042] S2, input the metal artifact to be removed CT image into the trained learnable artifact remover for decoding, and the trained learnable artifact remover gradually embeds the metal artifact to be removed CT image into the artifact coding feature map F(X) obtained in S1 during the decoding process. a ), and finally obtain the metal artifact-free image.

[0043] The metal artifact removal method of the present invention jointly trains the learnable artifact encoder, the learnable artifact remover and the learnable artifact generator according to a pre-constructed training database to obtain a trained learnable artifact encoder, a trained learnable artifact remover and a trained learnable artifact generator.

[0044] Preferably, the training database is provided with multiple pairs of paired data, and each pair of paired data consists of an image X containing metal artifacts. a And with the metal artifact image X a Paired artifact-free images X c composition.

[0045] Preferably, the above joint training includes the following steps:

[0046] A1. X a Input the learnable artifact encoder E a In the above example, we get the artifact coding feature map F(Xa );

[0047] A2. Take the same metal artifact image X a Input the learnable artifact remover G ac Decoding is performed, and then the learnable artifact remover G ac The artifact coding feature map F(X a ), and get the metal artifact-free image X ac , and X ac =G ac (X a ,F(X a ))+X a ;

[0048] A3, the artifact coding feature map F(X a ) is set to 0, and then the metal artifact image X a Input the learnable artifact remover G ac Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the image X containing metal artifacts aa , and X aa =G ac (X a ,0)+X a ;

[0049] A4. The metal artifact-free image X in the same paired data c Input the learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and get the metal artifact image X ca , and X ca =G ca (X c ,F(X a ))+X c ;

[0050] A5, the artifact encoding feature map F(X a ) is set to 0, and then the metal artifact-free image X in the same paired data c Input the learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the metal artifact-free image X cc , and X cc =G ca (X a ,0)+Xc ;

[0051] A6. Construct a total loss function consisting of the artifact removal process loss function, the artifact generation process loss function, and the overall bidirectional embedding framework consistency loss function;

[0052] A7, according to the total loss function obtained in A6, and using the stochastic gradient descent method to update the learnable artifact encoder E a The parameters of the learnable artifact remover G ac The parameters of the learnable artifact generator G ca Parameters;

[0053] A8, determine whether the number of training times is greater than or equal to the set number of training times, if so, go to A9; otherwise, return to A1;

[0054] A9. Training is over.

[0055] Preferably, the above artifact removal process loss function is constrained by L1 norm, and the metal artifact removal image X obtained by constraining A2 by L1 norm is ac Image with no metal artifacts X c , get the metal artifact-free image X ac With artifact-free image X c The error between sup , and the loss function of the artifact removal process is expressed by formula (1);

[0056] L sup =||X ac -X c ||1……Formula (1).

[0057] Preferably, the artifact generation process loss function is constrained by using a perceptual loss function, an adversarial loss function, and a characteristic loss function, and the metal artifact image X obtained by A4 is constrained by using the perceptual loss function, the adversarial loss function, and the characteristic loss function ca Image X with metal artifacts a At the feature level, we get the metal artifact image X ca Image X with metal artifacts a The error between them is given by the perceptual loss error L per , adversarial loss error L adv and feature loss error L fm composition.

[0058] Preferably, the perceptual loss error L per Calculated by formula (2);

[0059] L per =||VGG(X ca )-VGG(Xa )||1...Formula (2);

[0060] VGG(·) is the feature output by the next activation layer of the pooling layer in VGG-19 pre-trained on ImageNet, and VGG-19 is a network pre-trained using ImageNet data.

[0061] Preferably, the above-mentioned anti-loss error L adv Calculated by formula (3);

[0062]

[0063] P a (X a ) represents the data from the image X containing metal artifacts a , P c (Xc ) To represent the data artifact-free image X c , P G (X ca ) represents the data coming from the learnable artifact generator G ca The generated image X contains metal artifacts ca , D a is the scale discriminator and E is the expected value.

[0064] Preferably, the above feature loss error L fm Calculated by formula (4);

[0065]

[0066] T is the scale discriminator D a The total number of layers, N i is the number of elements in each layer, is the scale discriminator D a Extract the features of the downsampling block at the i-th layer.

[0067] Preferably, the above overall bidirectional embedding framework consistency loss function uses L1 norm constraint, and the metal artifact image X obtained by using L1 norm constraint A4 is ca The image X obtained with A3 contains metal artifacts. aa , get the metal artifact image X ca Keep the image containing metal artifacts X aa The error between them; and the metal artifact image X obtained by constraint A4 ca The metal artifact-free image obtained with A3X cc , get the metal artifact image X ca Keep metal artifact-free images with X cc The error between them, and the overall bidirectional embedding framework consistency loss function is expressed by formula (5):

[0068] L art =λ a1 ||X aa -X ca ||1+λ a2 ||X ac -X cc ||1……Formula (5);

[0069] Among them, λ a1 and λ a2 are all hyperparameters.

[0070] Preferably, the above total loss function is expressed by formula (6);

[0071] L=λ sup L sup +λ per L per +λ ad L adv +λ fm L fm +λ art L art

[0072] ...Formula (6);

[0073] Among them, λ sup , per , adv , fm and λ art is the default value.

[0074] The present invention discloses a metal artifact correction network model and a metal artifact removal method thereof, wherein the metal artifact correction network model is provided with a learnable artifact encoder, which extracts and features artifact information in an uncorrected CT image containing metal artifacts to obtain a corresponding multi-scale artifact encoding feature map; a learnable artifact remover, which removes artifact information in an uncorrected CT image containing metal artifacts; a learnable artifact generator, which generates artifact information on an artifact-free CT image paired with the uncorrected CT image containing metal artifacts; and the network structure of the learnable artifact remover is the same as the network structure of the learnable artifact generator. The present invention constructs an artifact encoder for separating metal artifacts from anatomical structures in an image containing metal artifacts, and the artifact encoder can extract multi-scale metal artifact encoding feature maps. Then, the artifact removal process and the artifact generation process are simultaneously modeled to strengthen the effective training of the artifact encoder. Specifically, a learnable artifact remover and a learnable artifact generator with exactly the same network structure are constructed. The learnable artifact remover is used to remove artifact information in uncorrected CT images. A multi-channel spatial self-attention module is provided. The spatial self-attention module uses a multi-scale artifact coding feature map to scale and modulate the feature map of the jump connection to adapt to the changes in artifact information of different scales and assist in the embedding of artifact coding information during the decoding process. The learnable artifact generator is used to generate artifact information on paired CT artifact-free images to assist in the training of the artifact encoder. The input of the artifact generator is an image without metal artifacts, and the input of the artifact remover is an image containing metal artifacts. The two encode their respective inputs respectively, and finally achieve the purpose of adding and subtracting artifacts. The present invention can obtain a metal artifact-free image with a better artifact correction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0076] Figure 1 This is a framework diagram of a metal artifact correction network model.

[0077] Figure 2 Figure 2 is a diagram of the framework of the learnable artifact encoder.

[0078] Figure 3 This is the structural diagram of the first downsampling block.

[0079] Figure 4 It is the structural diagram of the first residual module.

[0080] Figure 5 This is the structural diagram of the convolution module.

[0081] Figure 6 Figure 2 is the structural diagram of the learnable artifact generator.

[0082] Figure 7 Figure 2 is the structural diagram of the learnable artifact remover.

[0083] Figure 8 This is the structural diagram of the first upsampling block.

[0084] Fig. 9 It is the structural diagram of the second residual module.

[0085] Fig.10 This is the structural diagram of the second downsampling block.

[0086] Fig.11 This is the structural diagram of the second upsampling block.

[0087] Fig.12 This is the structural diagram of the header module.

[0088] Fig.13 This is the structural diagram of the tail module.

[0089] Fig.14 Flowchart of joint training.

[0090] Fig.15 are the CT image to be metal artifact removed and the final metal artifact removed image. DETAILED DESCRIPTION

[0091] The technical solution of the present invention is further described in conjunction with the following embodiments.

[0092] Example 1

[0093] A metal artifact correction network model, such as Figure 1 As shown, the settings are:

[0094] Learnable artifact encoder: extracts and features artifact information from uncorrected CT images containing metal artifacts, and obtains corresponding multi-scale artifact coding feature maps;

[0095] Learnable Artifact Remover - removes artifact information from uncorrected CT images containing metal artifacts;

[0096] Learnable Artifact Generator – Generates artifact information on artifact-free CT images paired with uncorrected CT images containing metal artifacts.

[0097] The network structure of the learnable artifact remover is the same as that of the learnable artifact generator. Both the network structure of the learnable artifact remover and the learnable artifact generator are U-net structures with skip links.

[0098] The learnable artifact encoder includes five first downsampling blocks, one deep convolutional layer, three first residual modules, and three convolutional modules. The upsampling operation in the first upsampling block is implemented by interpolation through the Upsample function. The normalization function of the learnable artifact encoder is SpectralNorm, which is beneficial to the learnable artifact encoder E. a It is relatively insensitive to input disturbances, which is conducive to the stability of training. The activation function is LeakyRelu. The first residual module includes a convolutional layer and an activation function.

[0099] like Figure 2 As shown, in the learnable artifact encoder, the uncorrected CT image containing metal artifacts is obtained from a first downsampling block (such as Figure 3 ) enters, and then enters the deep convolution layer, and then enters the other four first downsampling blocks in sequence, and then enters the three first residual modules in sequence (such as Figure 4 ), and then input into three convolution modules in sequence (such as Figure 5 ), and finally output from the last convolution module to obtain a multi-scale artifact encoding feature map.

[0100] Both the learnable artifact remover and the learnable artifact generator include eight second residual modules, four second downsampling blocks, four spatial self-attention modules, four first upsampling blocks for artifact encoding operations, and four second upsampling blocks. The spatial self-attention module SAM uses the multi-scale artifact coding feature map obtained by the learnable artifact encoder to scale and modulate the feature map of the jump connection. The downsampling operator in the first downsampling block is a convolution operation with a step size of 2. The downsampling operator in the second downsampling block is a convolution operation with a step size of 2. The upsampling operator in the second upsampling block is a deconvolution operation. The normalization function of the learnable artifact remover and the learnable artifact generator is InstanceNorm, that is, instance normalization, so as to achieve the conversion of the style of the image with artifacts to the style of the image without artifacts. The normalization method corresponds to the normalization method of the artifact encoder E, and the spectral normalization function SpectralNorm is selected. Then the spatial self-attention module SAM is used to convert the features of the jump connection in the U-net structure. With f i After modulation, the characteristics after modulation are Wherein, the subscript i represents the i-th scale. The superscript n∈{1,2}, n=1 represents the process of artifact generation, and n=2 represents the process of artifact removal. and It is for i Predicted hyperparameters after 3×3 convolution. The purpose of using the spatial self-attention module SAM modulation is to automatically learn and scale the embedded features to better match the feature changes at different feature scales. The modulated features The decoded features after upsampling with G.UpBlock are concatenated according to the conventional U-net architecture.

[0101] The activation function is LeakyRelu. The second residual module includes a convolution layer, a normalization function, and an activation function.

[0102] In the learnable artifact remover and learnable artifact generator respectively:

[0103] The four first upsampling blocks (such as Figure 8 ) are defined as the first upsampling block 1, the first upsampling block 2, the first upsampling block 3 and the first upsampling block 4 from front to back according to the transmission direction, and the first upsampling block 1, the first upsampling block 2, the first upsampling block 3 and the first upsampling block 4 are connected in sequence, and the multi-scale artifact coding feature map F(X) of the artifact encoder can be learned a ) is input from the first upsampling block 1.

[0104] The four second residual modules (such as Fig. 9 ) are respectively connected with the four second upsampling blocks (such as Fig.11 ) are correspondingly spliced ​​to obtain four first splicing blocks, which are defined as the first splicing block 1, the first splicing block 2, the first splicing block 3 and the first splicing block 4 from front to back according to the transmission direction, and the first splicing block 1, the first splicing block 2, the first splicing block 3 and the first splicing block 4 are connected in sequence.

[0105] The other four second residual modules are respectively connected to the four second downsampling blocks (such as Fig.10 ) are correspondingly spliced ​​to obtain four second splicing blocks; the four second splicing blocks are defined as second splicing block 1, second splicing block 2, second splicing block 3 and second splicing block 4 from front to back according to the transmission direction; and second splicing block 1, second splicing block 2, second splicing block 3 and second splicing block 4 are connected in sequence, and the output of second splicing block 4 is also transmitted to the first splicing block 1.

[0106] The four spatial self-attention modules are defined as spatial self-attention module 1, spatial self-attention module 2, spatial self-attention module 3 and spatial self-attention module 4 respectively.

[0107] The output of the first upsampling block 1 and the output of the second splicing block 4 are respectively input into the spatial self-attention module 1, and the output of the spatial self-attention module 1 is input into the first splicing block 1.

[0108] The output of the first upsampling block 2 and the output of the second splicing block 3 are respectively input to the spatial self-attention module 2, and the output of the spatial self-attention module 2 is input to the first splicing block 2.

[0109] The output of the first upsampling block 3 and the output of the second splicing block 2 are respectively input to the spatial self-attention module 3, and the output of the spatial self-attention module 3 is input to the first splicing block 3.

[0110] The output of the first upsampling block 4 and the output of the second splicing block 1 are respectively input into the spatial self-attention module 4, and the output of the spatial self-attention module 4 is input into the first splicing block 4.

[0111] The present invention further connects a head module (such as Fig.12 ), a tail module (such as Fig.13 ).

[0112] When the uncorrected CT image X containing metal artifacts is used in the learnable artifact remover a The residual module side of the second splicing block 1 is input, and the output of the first splicing block 4 is added to the uncorrected CT image containing metal artifacts to output the artifact-free CT image X ac ,like Figure 7 .

[0113] When the artifact-free CT image X c The residual module side of the second splicing block 1 is input, and the output of the first splicing block 4 is added to the artifact-free CT image to output the artifact-added CT image X ca ,like Figure 6 .

[0114] It should be noted that in Figure 2-Figure 5 and Figure 8-Figure 13 In it, K represents the convolution kernel size, S represents the convolution step size, P represents the padding size, and SF represents the upsampling scale factor.

[0115] The metal artifact correction network model constructs an artifact encoder for separating metal artifacts from anatomical structures in images containing metal artifacts, and the artifact encoder can extract multi-scale metal artifact coding feature maps. Then, the artifact removal process and the artifact generation process are modeled simultaneously to strengthen the effective training of the artifact encoder. Specifically, a learnable artifact remover and a learnable artifact generator with exactly the same network structure are constructed. The learnable artifact remover is used to remove artifact information in uncorrected CT images, and a multi-channel spatial self-attention module is provided. The spatial self-attention module uses a multi-scale artifact coding feature map to scale and modulate the feature map of the jump connection to adapt to the changes in artifact information of different scales and assist in the embedding of artifact coding information during the decoding process, and the learnable artifact generator is used to generate artifact information on paired CT artifact-free images to assist in the training of the artifact encoder. The input of the artifact generator is an image without metal artifacts, and the input of the artifact remover is an image containing metal artifacts. The two respectively encode their respective inputs, and finally achieve the purpose of adding artifacts and subtracting artifacts. The present invention can obtain a metal artifact-free image with a better artifact correction effect.

[0116] Example 2

[0117] A metal artifact removal method is implemented using the metal artifact correction network model of Example 1.

[0118] In the metal artifact removal method of the present invention, a learnable artifact encoder, a learnable artifact remover and a learnable artifact generator are jointly trained according to a pre-constructed training database to obtain a trained learnable artifact encoder, a trained learnable artifact remover and a trained learnable artifact generator.

[0119] The training database is set with multiple pairs of paired data, and each pair of paired data consists of an image X containing metal artifacts. a And with the metal artifact image X a Paired artifact-free images X c Each pair of paired data refers to the image X without metal artifacts corresponding to the same slice of the same object. c and metal artifact image X a .

[0120] The joint training includes the following steps: Fig.14 As shown:

[0121] A1. X a Input learnable artifact encoder E a In the above example, we get the artifact coding feature map F(X a );

[0122] A2. Take the same metal artifact image X aInput learnable artifact remover G ac After decoding, the artifact remover G can be learned ac The artifact coding feature map F(X a ), and get the metal artifact-free image X ac , and X ac =G ac (X a ,F(X a ))+X a ;

[0123] A3, the artifact encoding feature map F(X a ) is set to 0, and then the metal artifact image X a Input learnable artifact remover G ac Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the image X containing metal artifacts aa , and X aa =G ac (X a ,0)+X a ;

[0124] A4. The metal artifact-free image X in the same paired data c Input learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the metal artifact image X ca , and X ca =G ca (X c ,F(X a ))+X c ;

[0125] A5, the artifact encoding feature map F(X a ) is set to 0, and then the metal artifact-free image X in the same paired data c Input learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the metal artifact-free image X cc , and X cc =G ca (X a ,0)+X c ;

[0126] A6. Construct a total loss function consisting of the artifact removal process loss function, the artifact generation process loss function, and the overall bidirectional embedding framework consistency loss function;

[0127] A7. Update the learnable artifact encoder E using the total loss function obtained from A6 and the stochastic gradient descent method. a Parameters of the learnable artifact remover G ac Parameters of the learnable artifact generator G ca Parameters;

[0128] A8, determine whether the number of training times is greater than or equal to the set number of training times, if so, go to A9; otherwise, return to A1;

[0129] A9. Training is over.

[0130] The loss function of the artifact removal process of the present invention is constrained by the L1 norm. The metal artifact removal image X obtained by constraining A2 by the L1 norm is ac Image with no metal artifacts X c , get the metal artifact-free image X ac With artifact-free image X c The error between sup , and the loss function of the artifact removal process is expressed by formula (1);

[0131] L sup =||X ac -X c ||1……Formula (1).

[0132] The loss function of the artifact generation process is constrained by the perceptual loss function, the adversarial loss function and the feature loss function. The metal artifact image X obtained by A4 is constrained by the perceptual loss function, the adversarial loss function and the feature loss function. ca Image X with metal artifacts a At the feature level, we get the metal artifact image X ca Image X with metal artifacts a The error between them is given by the perceptual loss error L per , adversarial loss error L adv and feature loss error L fm composition.

[0133] Perceptual loss error L per Calculated by formula (2);

[0134] L per =||VGG(X ca )-VGG(X a )||1...Formula (2);

[0135] VGG(·) is the feature output by the next activation layer of the pooling layer in VGG-19 pre-trained on ImageNet, and VGG-19 is a network pre-trained using ImageNet data.

[0136] It should be noted that the perceptual loss function of the present invention is used to measure perceptual similarity.

[0137] Adversarial loss error L adv Calculated by formula (3);

[0138]

[0139] P a (X a ) represents the data from the image X containing metal artifacts a , P c (X c ) represents the data artifact-free image X c , P G (X ca ) represents the data coming from the learnable artifact generator G ca The generated image X contains metal artifacts ca , D a is the scale discriminator and E is the expected value.

[0140] It should be noted that the adversarial loss of the present invention utilizes a multi-scale discriminator D a To regularize X ca The rationality of D a Select the regular discriminator LayerDiscriminator.

[0141] Feature loss error L fm Calculated by formula (4);

[0142]

[0143] T is the scale discriminator D a The total number of layers, N i is the number of elements in each layer, is the scale discriminator D a Extract the features of the downsampling block at the i-th layer.

[0144] The overall bidirectional embedding framework consistency loss function uses the L1 norm constraint. The metal artifact image X obtained by A4 using the L1 norm constraint ca The image X obtained with A3 contains metal artifacts. aa , get the metal artifact image X ca Keep the image containing metal artifacts X aa The error between them; and the metal artifact image X obtained by constraint A4ca The metal artifact-free image obtained with A3X cc , get the metal artifact image X ca Keep metal artifact-free images with X cc The error between them is , and the overall bidirectional embedding framework consistency loss function is expressed by formula (5):

[0145] L art =λ a1 ||X aa -X ca ||1+λ a2 ||X ac -X cc ||1……Formula (5);

[0146] Among them, λ a1 and λ a2 are all hyperparameters.

[0147] It should be noted that the artifact consistency loss constraint of the present invention retains the image X aa and the generated artifact image X ca consistent with, rather than with X a Stay consistent. Because X aa Indicates that the artifact-containing image is generated without embedding artifact coding. In the proposed fully symmetrical bidirectional embedding framework, its corresponding target should be the artifact-containing image generated by embedding artifact coding, i.e., X ca , in order to align the consistency of the generated artifact images. Similarly, X cc represents an artifact-free image generated by non-embedded artifact coding, and its goal should be an artifact-free image X generated by embedded artifact coding ac .

[0148] The total loss function is expressed by formula (6);

[0149] L=λ sup L sup +λ per L per +λ adv L adv +λ fm L fm +λ art L art

[0150] ...Formula (6);

[0151] Among them, λ sup , per , adv , fm and λ art is the default value.

[0152] It should be noted that the present invention sup , per , ad , fm and λ art It is determined based on experiments and experience. sup =λ per =100,λ adv =1,λ fm =λ art =10,λ a1 =1,λ a2 =0.

[0153] The metal artifact removal method is performed by the following steps:

[0154] S1. Input the metal artifact-removing CT image into the trained learnable artifact encoder to obtain the artifact encoding feature map F(X a );

[0155] S2, input the metal artifact to be removed CT image into the trained learnable artifact remover for decoding, and the trained learnable artifact remover gradually embeds the metal artifact to be removed CT image into the artifact coding feature map F(X) obtained in S1 during the decoding process. a ), and finally obtain the metal artifact-free image.

[0156] The metal artifact removal method is based on the metal artifact correction network model. The present invention first trains the metal artifact correction network model, and because the metal artifact correction network model is a completely symmetrical network model architecture. In practical applications, only the trained learnable artifact encoder and the trained learnable artifact remover are needed to identify, extract and remove artifacts from the input CT image to be removed from metal artifacts, which has good accuracy and practicality, and ultimately achieves the purpose of adding and subtracting artifacts, and the present invention obtains a metal artifact-removed image with a better artifact correction effect.

[0157] Example 3

[0158] A metal artifact removal method based on the metal artifact correction network model of Example 1, other features are the same as those of Example 2, except that: sup , per , adv , fm and λ art The specific values ​​are as follows: sup =λ per =100,λ adv =1,λ fm =λ art =10,λ a1 =1,λa2 =0.

[0159] This example uses 95 sets of patient data without metal artifacts obtained by dental CBCT scanners in dental clinics and 8802 sets of paired data obtained by clinical oral implant metal mask simulation for training. Then, the training is performed according to the joint training of Example 2 under the Pytorch deep learning framework. The Adam optimizer is used, and the learning rate is increased from 2×10 -5 Linearly decreases to 1×10 -7 , the network batch size is 1, the number of training times is 50, and the trained learnable artifact encoder and the trained learnable artifact remover are obtained.

[0160] The oral CT image containing metal artifacts is scanned by using a CBCT scanner of a dental clinic at a tube voltage of 85 kVp and a tube current of 9 mAs as the CT image to be removed from the metal artifacts; the CT image to be removed from the metal artifacts is subjected to metal artifact removal according to the steps of the metal artifact removal method of Example 2 to obtain Fig.15 .

[0161] Among them Fig.15 The left side of the middle is the CT image to be metal artifact removed, and the right side is the final metal artifact removed image. Fig.15 It can be seen that the uncorrected CT image has many light and dark stripe artifacts, while the corrected image obtained by the present invention effectively suppresses the radial stripe artifacts caused by metal and effectively maintains the tooth structure. Therefore, the present invention can obtain a metal artifact-free image with better artifact correction effect.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A metal artifact correction network model, characterized in that: The settings are: Learnable artifact encoder: extracts and features artifact information from uncorrected CT images containing metal artifacts, and obtains corresponding multi-scale artifact coding feature maps; Learnable Artifact Remover - removes artifact information from uncorrected CT images containing metal artifacts; A learnable artifact generator for generating artifact information on an artifact-free CT image paired with the uncorrected metal artifact-containing CT image; The network structure of the learnable artifact remover is the same as the network structure of the learnable artifact generator.

2. The metal artifact correction network model according to claim 1, characterized in that: The learnable artifact encoder includes five first downsampling blocks, a deep convolutional layer, three first residual modules and three convolutional modules.

3. The metal artifact correction network model according to claim 2, characterized in that: The network structure of the learnable artifact remover and the learnable artifact generator are both U-net structures with skip links; The learnable artifact remover and the learnable artifact generator both include eight second residual modules, four second downsampling blocks, four spatial self-attention modules, four first upsampling blocks for artifact encoding operations, and four second upsampling blocks.

4. The metal artifact correction network model according to claim 3, characterized in that: The spatial self-attention module SAM uses the multi-scale artifact coding feature map obtained by the learnable artifact encoder to scale and modulate the feature map of the jump connection; The downsampling operator in the first downsampling block is a convolution operation with a step size equal to 2; The downsampling operator in the second downsampling block is a convolution operation with a step size equal to 2; The upsampling operation in the first upsampling block is implemented by interpolation through the Upsample function; The upsampling operator in the second upsampling block is a deconvolution operation; The normalization function of the learnable artifact encoder is SpectralNorm, and the activation function is LeakyRelu; The normalization function of the learnable artifact remover and the learnable artifact generator are both InstanceNorm, and the activation function is both LeakyRelu; The first residual module includes a convolutional layer and an activation function; The second residual module includes a convolution layer, a normalization function and an activation function.

5. The metal artifact correction network model according to claim 3, characterized in that: Respectively in the learnable artifact remover and the learnable artifact generator: The four first upsampling blocks are defined as a first upsampling block 1, a first upsampling block 2, a first upsampling block 3 and a first upsampling block 4 from front to back according to the transmission direction, and the first upsampling block 1, the first upsampling block 2, the first upsampling block 3 and the first upsampling block 4 are connected in sequence, and the multi-scale artifact coding feature map of the learnable artifact encoder is input from the first upsampling block 1; Four of the second residual modules are respectively concatenated with four corresponding second up-sampling blocks through a concatenation function Concat to obtain four first concatenated blocks, wherein the four first concatenated blocks are defined as a first concatenated block 1, a first concatenated block 2, a first concatenated block 3 and a first concatenated block 4 from front to back according to a transmission direction, and the first concatenated block 1, the first concatenated block 2, the first concatenated block 3 and the first concatenated block 4 are connected in sequence; Splicing the other four second residual modules with the four second downsampling blocks respectively to obtain four second splicing blocks; the four second splicing blocks are defined as second splicing block 1, second splicing block 2, second splicing block 3 and second splicing block 4 from front to back according to the transmission direction; and the second splicing block 1, the second splicing block 2, the second splicing block 3 and the second splicing block 4 are connected in sequence, and the output of the second splicing block 4 is also transmitted to the first splicing block 1; The four spatial self-attention modules are defined as spatial self-attention module 1, spatial self-attention module 2, spatial self-attention module 3 and spatial self-attention module 4 respectively; The output of the first upsampling block 1 and the output of the second splicing block 4 are respectively input into the spatial self-attention module 1, and the output of the spatial self-attention module 1 is input into the first splicing block 1; The output of the first upsampling block 2 and the output of the second splicing block 3 are respectively input into the spatial self-attention module 2, and the output of the spatial self-attention module 2 is input into the first splicing block 2; The output of the first upsampling block 3 and the output of the second splicing block 2 are respectively input into the spatial self-attention module 3, and the output of the spatial self-attention module 3 is input into the first splicing block 3; The output of the first upsampling block 4 and the output of the second splicing block 1 are respectively input to the spatial self-attention module 4, and the output of the spatial self-attention module 4 is input to the first splicing block 4; When in the learnable artifact remover, the uncorrected CT image containing metal artifacts is input from the residual module side of the second splicing block 1, the output of the first splicing block 4 is added to the uncorrected CT image containing metal artifacts to output an artifact-free CT image; When in the learnable artifact generator, the artifact-free CT image is input from the residual module side of the second splicing block 1, the output of the first splicing block 4 is added to the artifact-free CT image to output the artifact-added CT image; In the learnable artifact encoder, an uncorrected CT image containing metal artifacts enters from one of the first downsampling blocks, then is input into the deep convolution layer, then is sequentially input into another four first downsampling blocks, then is sequentially input into three of the first residual modules, then is sequentially input into three of the convolution modules, and finally is output from the last convolution module to obtain a multi-scale artifact coding feature map.

6. A method for removing metal artifacts, characterized in that: The metal artifact correction network model described in any one of claims 1 to 5 is used for the correction.

7. The metal artifact removal method according to claim 6, characterized in that: The following steps are performed: S1. Input the metal artifact-removing CT image into the trained learnable artifact encoder to obtain the artifact encoding feature map F(X a ); S2, input the metal artifact to be removed CT image into the trained learnable artifact remover for decoding, and the trained learnable artifact remover gradually embeds the metal artifact to be removed CT image into the artifact coding feature map F(X) obtained in S1 during the decoding process. a ), and finally obtain the metal artifact-free image.

8. The metal artifact removal method according to claim 7, characterized in that: Jointly training the learnable artifact encoder, the learnable artifact remover and the learnable artifact generator according to a pre-constructed training database to obtain a trained learnable artifact encoder, a trained learnable artifact remover and a trained learnable artifact generator; The training database is provided with multiple pairs of paired data, and each pair of paired data consists of an image X containing metal artifacts. a And with the metal artifact image X a Paired artifact-free images X c composition.

9. The metal artifact removal method according to claim 8, characterized in that: The joint training comprises the following steps: A1. X a Input the learnable artifact encoder E a In the above example, we get the artifact coding feature map F(X a ); A2. Take the same metal artifact image X a Input the learnable artifact remover G ac Decoding is performed, and then the learnable artifact remover G ac The artifact coding feature map F(X a ), and get the metal artifact-free image X ac , and X ac =G ac (X a ,F(X a ))+X a ; A3, the artifact coding feature map F(X a ) is set to 0, and then the metal artifact image X a Input the learnable artifact remover G ac Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the image X containing metal artifacts aa , and X aa =G ac (X a ,0)+X a ; A4. The metal artifact-free image X in the same paired data c Input the learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the metal artifact image X ca , and X ca =G ca (X c ,F(X a ))+X c ; A5, the artifact encoding feature map F(X a ) is set to 0, and then the metal artifact-free image X in the same paired data c Input the learnable artifact generator G ca Decode, and gradually embed the artifact coding feature map F(X a ), and obtain the metal artifact-free image X cc , and X cc =G ca (X a ,0)+X c ; A6. Construct a total loss function consisting of the artifact removal process loss function, the artifact generation process loss function, and the overall bidirectional embedding framework consistency loss function; A7, according to the total loss function obtained in A6, and using the stochastic gradient descent method to update the learnable artifact encoder E a The parameters of the learnable artifact remover G ac The parameters of the learnable artifact generator G ca Parameters; A8, determine whether the number of training times is greater than or equal to the set number of training times, if it is greater than or equal to, enter A9; Otherwise, return to A1; A9. Training is over.

10. The metal artifact removal method according to claim 9, characterized in that: The de-artifacting process loss function is constrained by the L1 norm. The metal artifact-free image X obtained by constraining A2 by the L1 norm is ac Image with no metal artifacts X c , get the metal artifact-free image X ac With artifact-free image X c The error between sup , and the loss function of the artifact removal process is expressed by formula (1); L sup =||X ac -X c ||1……Formula (1); The artifact generation process loss function is constrained by using a perceptual loss function, an adversarial loss function and a characteristic loss function, and the metal artifact image X obtained by A4 is constrained by using the perceptual loss function, the adversarial loss function and the characteristic loss function. ca Image X with metal artifacts a At the feature level, we get the metal artifact image X ca Image X with metal artifacts a The error between them is given by the perceptual loss error L per , adversarial loss error L adv and feature loss error L fm composition; The perceptual loss error L per Calculated by formula (2); L per = ||VGG(X ca ) - VGG(X a )||1…… Equation (2); VGG(·) is the feature output by the next activation layer of the pooling layer in VGG-19 pre-trained on ImageNet. VGG-19 is a network pre-trained using ImageNet data. The adversarial loss error L adv Calculated by formula (3); P a (X a ) represents the data from the image X containing metal artifacts a , P c (X c ) represents the data artifact-free image X c , P G (X ca ) represents the data coming from the learnable artifact generator G ca The generated image containing metal artifacts X ca , D a is the scale discriminator, E is the expected value; The characteristic loss error L fm Calculated by formula (4); T is the scale discriminator D a The total number of layers, N i is the number of elements in each layer, is the scale discriminator D a Extract the features of the downsampling block of the i-th layer; The overall bidirectional embedding framework consistency loss function uses the L1 norm constraint. The metal artifact image X obtained by using the L1 norm constraint A4 ca The image X obtained with A3 contains metal artifacts. aa , get the metal artifact image X ca Keep the image containing metal artifacts X aa The error between them; and the metal artifact image X obtained by constraint A4 ca The metal artifact-free image obtained with A3X cc , get the metal artifact image X ca Keep metal artifact-free images with X cc The error between them, and the overall bidirectional embedding framework consistency loss function is expressed by formula (5): L art = λ a1 ||X aa - X ca ||1 + λ a2 ||X ac - X cc ||1……Equation (5); Among them, λ a1 and λ a2 All are hyperparameters; The total loss function is expressed by formula (6); L=λ sup L sup +λ per L per +λ ad L adv +λ fm L fm +λ ar L art ...Formula (6); Among them, λ sup , per , adv , fm and λ ar is the default value.

Images