A method for removing sparse angle CT artifacts

Abstract

The present invention discloses a method for removing artifacts from sparse angular CT. The method is based on perceptual loss and feature fusion attention residual network. It includes: preprocessing the input sparse CT image with artifacts and the original CT image; extracting features from the preprocessed sparse CT image and the original CT image to obtain a feature map; training the obtained feature map with an improved feature fusion attention residual network with perceptual loss as the loss function, using the original CT image as a label, so that the sparse CT image can remove artifacts but keep its structure unchanged; finally, the sparse CT image is used as input, and the artifacts are removed using the trained network model, and the peak signal-to-noise ratio and structural similarity are calculated. The method uses perceptual loss as the loss function, which can enhance detail information. And adding an attention mechanism to the feature fusion residual network to improve the performance of the network, and finally achieving a fast and effective artifact removal effect.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, in particular to a method for removing sparse angle CT artifacts. Background Art

[0002] Computed tomography (CT) is a key medical imaging technology. However, the potential X-ray radiation problem in CT imaging has attracted people's attention, and the radiation dose increases the risk of cancer. Therefore, many studies have focused on reducing the radiation dose, which has led to the development of many effective methods. The initial methods focused on reducing X-ray exposure by reducing or adjusting the tube current. Methods that reduce the number of projections for a given scanning trajectory will cause severe streak artifacts in CT images. Reconstructing images from sparsely sampled data can solve the problem of insufficient data due to limited sampling angles. However, reconstruction of sparse view CT projection data (such as using FBP) will produce streak artifacts in CT images.

[0003] In recent years, deep learning (DL) has become a powerful feature learning method. Benefiting from the power of deep learning and end-to-end deep supervised learning methods, and with the help of convolutional neural networks, great progress has been made in natural image problems. For example, object detection, image segmentation, and image super-resolution. Many researchers have used deep learning methods for sparse CT image reconstruction and artifact removal.

[0004] The Chinese patent application "A CT sparse reconstruction artifact correction method and system based on residual learning" (patent application number CN201711097204.1, publication number CN107871332A) learns the characteristics of artifacts by establishing a residual neural network structure to obtain an artifact map; finally, the sparsely reconstructed map and the artifact map are used to make a residual to restore a clear CT image. However, this method has problems such as long training time and many model parameters, and also increases the computational complexity of the network. Summary of the invention

[0005] In view of the above-mentioned technical and material selection problems in the prior art, the present invention provides a fast, concise and robust method for removing artifacts of sparse angle CT.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] The present invention is a method for removing sparse angle CT artifacts, and the specific steps are as follows:

[0008] Step 1, preprocessing the sparse angular CT data with streak artifacts and the original CT data into .dcm format images of a specified size;

[0009] Step 2, the preprocessed sparse angular CT data is passed through the VGG19 network to extract image features for subsequent training;

[0010] Step 3, construct a feature fusion attention residual network, in which the residual skip dense block is used as the basis of the shallow layer of the network. The local feature fusion layer is located after the two convolutional layers. The residual skip dense block skips the connection between the two local feature fusion layers, and the feature fusion result of the previous module is used as the input of the next module. The features of the local feature fusion are stacked, and the residual information is used to integrate the feature information to form a basic network architecture;

[0011] Step 4: Take the sparse CT image as input, the original CT image as the label, and the perceptual loss as the loss function for training to obtain a trained feature fusion attention residual network model;

[0012] Step 5: Use the sparse CT image of the same part as input for testing to obtain an image with artifacts removed, and calculate the peak signal-to-noise ratio and structural similarity between the output CT image and the labeled CT image.

[0013] A further improvement of the present invention is that step 1 is specifically:

[0014] Step 1.1, modify the size of the head sparse CT images and original CT images in the dataset so that the resolution of each image is m×n, where m and n are the length and height of the image respectively. The sparse CT images and original CT images are in .dcm format.

[0015] Step 1.2: Convert the image of the new size in step 1.1 into a Tensor data type suitable for the Pytorch framework.

[0016] A further improvement of the present invention is that in step 2, three identical images are put together to form three channels to adapt to the VGG19 network, and the preprocessed sparse CT image and the original CT image are extracted features by the trained VGG19 model, the former is used for artifact removal training, and the latter is used for prediction training with stripe artifacts.

[0017] A further improvement of the present invention is that the specific operation of constructing the feature fusion attention residual network in step 3 is as follows: connecting between several residual dense blocks, fusing the local features in each RSDB, and passing them to the subsequent residual blocks, the f-th RSDB directly introduces the local features in the (f-1)-th RSDB or RB and performs feature fusion, where, if f=1, the model adapts to the residual block RB, and the local feature fusion LFF is expressed as:

[0018]

[0019] where F f,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 It represents the result of shallow feature extraction of the f-th RSDB input after two Conv layers, expressed as:

[0020]

[0021] Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 Represents the result of the f-1th RSDB output;

[0022] After local features are fused in each RSDB, they are passed to the attention module. The feature map obtained by the attention module is passed to the next RSDB for the same operation. The specified layer features in different basic blocks are stacked together to achieve spaced jump connections and effective feature extraction. The feature map stacking is represented as:

[0023]

[0024] Among them, F0 is the feature map of the Conv layer in the residual block RB of the first RSDB pre-model adaptation. The subsequent FFRN network structure will mention that in order to adapt to the RSDB structure, an RB block will be introduced in RSDB. Finally, local residual learning is introduced to further improve the information flow, F f，3 The representation of the transformation layer function uses a convolutional layer with a kernel size of axa.

[0025] There are two ways to obtain features from the attention mechanism module. One is the main channel, which can replace any frontier convolution module. The other is the mask channel, which mainly processes the features to generate a soft mask, which becomes a control gate to control new features and make the features more concentrated.

[0026] Assume that the input of the main road is x, the output is T(x), and the mask output is M(x). The output of the attention module is

[0027] H i,c (x) = M i,c (x)*T i,c (x)

[0028] Where i represents the spatial position of the feature, and c represents the feature channel.

[0029] In the attention module, the mask not only serves as a feature selector in the forward inference process, but also as a gradient update filter in the back-propagation process.

[0030]

[0031] When back-propagating the gradient update, the parameters θ of the mask part are not updated, but only the parameters of the main road are updated. Where θ is the parameter of the mask branch and φ is the parameter of the trunk branch. Derivatives are taken and gradients are calculated. This makes the attention module very robust to noise and can effectively reduce the impact of noise on gradient updates. This property makes the network very robust to noisy labels. Each main road will be matched with a mask road, and the feature mask of this step is learned to make the mask more targeted.

[0032] Since the value of the mask is [0,1], refer to the residual learning structure of Resnet. The output of this module is improved to a residual structure:

[0033] H i,c (x) = (1 + M i,c (x))*T i,c (x)

[0034] A further improvement of the present invention is that the perceptual loss in step 4 is expressed as:

[0035]

[0036] in and are the features of the sparse CT and original CT images extracted by the j-layer VGG19 network, where W j is the weight coefficient of the jth layer, G j and H j They represent the length and width of the j-th layer network respectively.

[0037] The absolute value loss mainly measures the difference between the pixel values ​​between the neural network output image and the reference image. Its formula is as follows:

[0038]

[0039] M and N are the number of rows and columns of the image. This formula calculates the expectation of the absolute value of the difference between the pixels of the convolutional neural network output image I and the reference image R, where I(k, l) and R(k, l) represent the input image pixel and output image pixel at the kth row and lth column, respectively;

[0040] The loss function in the network combines absolute value loss and perceptual loss, expressed as follows:

[0041] L loss =L mae +λL feat

[0042] Where λ is the weight coefficient of perceptual loss.

[0043] The beneficial effects of the present invention are:

[0044] 1. The present invention is based on perceptual loss. In the trained network model, perceptual loss and image feature loss are used as loss functions, which reduces the loss of the network and improves the quality of the output CT image.

[0045] 2. The present invention combines the feature fusion attention residual network and adopts the residual learning method. After subtracting the output CT image from the input sparse CT image with stripe artifacts, the image with stripe artifacts is used as a label to predict the stripe artifacts.

[0046] 3. Compared with the traditional residual network, the network of the present invention makes more full use of local features and introduces an attention module to learn more detailed information;

[0047] 4. The present invention does not change the size of the image during the training process, and perfectly preserves the details and edge information of the image;

[0048] 5. Compared with the traditional residual network, the network of the present invention has a simpler structure, consumes less memory resources and takes less training time;

[0049] 6. The images in the training process of the present invention can adopt other sizes, such as 256×256, 512×512, etc., which greatly expands the applicability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is an overall flow chart of the method of an embodiment of the present invention.

[0051] Figure 2 It is the RB module of the present invention.

[0052] Figure 3 It is the RSDB module of the present invention.

[0053] Figure 4 It is the attention mechanism module of the present invention.

[0054] Figure 5 The overall structure diagram of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and innovation of the present invention clearer, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The present invention is a method for removing sparse angle CT artifacts

[0057] Step 1: preprocess each image in the sparse CT and original CT datasets so that each image has the same size and is converted into a data type suitable for network training; the specific steps are as follows:

[0058] Step 1.1: Cut the original CT data (.dcm format) obtained from the hospital into the same size, with a pixel size of m*n, where m and n are the length and height of the image respectively. In the experiment, m=n=512 is taken;

[0059] Step 1.2: Convert the adjusted .dcm data into a Tensor data type suitable for the Pytorch framework.

[0060] Step 2: Extract image features from the preprocessed sparse angular CT data through the VGG19 network for subsequent training; the specific operations are as follows:

[0061] The preprocessed data are all single-channel medical images, and the input of the selected VGG19 network is a three-channel color image, so the input image needs to be increased in dimension to adapt to the network. The specific method is: select three identical images to form a three-channel (512, 512, 3) input, and extract features through the open source trained VGG19 model imagenet-vgg-verydeep-19 version.

[0062] Step 3, construct a feature fusion attention residual network, in which the residual skip dense block is used as the basis of the shallow layer of the network. The local feature fusion layer is located after the two convolutional layers. The residual skip dense block skips the connection between the two local feature fusion layers, and the feature fusion result of the previous module is used as the input of the next module. The features of the local feature fusion are stacked, and the residual information is used to integrate the feature information to form a basic network architecture;

[0063] Residual Dense Block (RDB) can extract multi-level features from the original image and further improve network performance. RDB is a residual dense block proposed in RDN. RDB makes full use of all layers through local dense connections, and then adaptively saves the accumulated features through local feature fusion (LFF). Through global residual learning, shallow features and deep features are combined for global feature fusion (GFF) to obtain global dense features.

[0064] The blocks are connected to each other, the local features in each RSDB are fused, and finally passed to the subsequent residual block; the f-th RSDB directly introduces the local features in the (f-1)-th RSDB or RB for feature fusion. If f = 1, the model adapts to the residual block RB, and the local feature fusion LFF is expressed as:

[0065]

[0066] where F f,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 It represents the result of shallow feature extraction of the f-th RSDB input after two Conv layers, which can be expressed as:

[0067]

[0068] Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 Represents the result of the f-1th RSDB output.

[0069] After local features are fused in each RSDB, they are passed to the attention module. The feature map obtained by the attention module is passed to the next RSDB for the same operation. The specified layer features in different basic blocks are stacked together to achieve spaced jump connections and effective feature extraction. The feature map stacking is represented as:

[0070]

[0071] Among them, F0 is the feature map of the Conv layer in the residual block RB of the first RSDB pre-model adaptation. The subsequent FFRN network structure will mention that in order to adapt to the RSDB structure, an RB block will be introduced in RSDB. Finally, local residual learning is introduced to further improve the information flow, F f，3 It means that the conversion layer function can use a convolutional layer with a kernel size of 1x1 or 3x3. Different kernel sizes will play different roles.

[0072] There are two ways to obtain features from the attention mechanism module. One is the main channel, which can replace any frontier convolution module. The other is the mask channel, which mainly processes the features to generate a soft mask, which becomes a control gate to control new features and make the features more concentrated.

[0073] Assume that the input of the trunk is x, the output is the trunk branch feature map T(x), and the mask branch (mask) output is M(x). At this time, the output of the attention module is

[0074] H i,c (x) = M i,c (x)*T i,c (x)

[0075] Where i represents the spatial position of the feature, and c represents the feature channel.

[0076] In the attention module, the mask not only serves as a feature selector in the forward inference process, but also as a gradient update filter in the back-propagation process.

[0077]

[0078] When back-propagating the gradient update, the parameters θ of the mask part are not updated, but only the parameters of the main road are updated. Where θ is the parameter of the mask branch and φ is the parameter of the trunk branch. Derivatives are taken and gradients are calculated. This makes the attention module very robust to noise and can effectively reduce the impact of noise on gradient updates. This property makes the network very robust to noisy labels. Each main road will be matched with a mask road, and the feature mask of this step is learned to make the mask more targeted.

[0079] Since the value of the mask is [0,1], refer to the residual learning structure of Resnet. The output of this module is improved to a residual structure:

[0080] H i,c (x) = (1 + M i,c (x))*T i,c (x)

[0081] The convolutional neural network in this example is Figure 2 As shown in , the residual block (RB) structure is: the first layer is a 3*3 convolution layer, followed by a ReLU layer, then a 3*3 convolution layer, a 3*1 convolution layer, and a 1*3 convolution layer. Figure 3 As shown in the figure, the structure of the RSDB module is: the first layer is a 3*3 convolution layer, followed by a ReLU layer, then a 3*3 convolution layer, a ReLU layer, a connection layer, then a 3*1 convolution layer, and a 1*3 convolution layer. While increasing the receptive field, the calculation speed is accelerated. Figure 4 As shown, the attention module mechanism.

[0082] Step 4, with sparse CT as input, original CT as label, and perceptual loss as loss function, the feature fusion residual network is trained, and the upper and lower losses are calculated according to the perceptual difference between the sparse CT and original CT images and the features of the corresponding convolutional layer. As the background loss changes, the image value of the reconstructed CT image will also be updated. The present invention uses an iterative optimization method to correct the sparse CT image. According to the result of removing artifacts, the context loss function, network parameters and the ratio of feature sample size are adjusted when necessary, and the network training is repeated until the artifacts are effectively removed.

[0083] Image problems can be viewed as image transformation tasks, which accept an input image and then output the transformed image. For example, in image processing, such as denoising, super-resolution reconstruction, and image colorization, the input image is a degraded low-quality image (noise, low resolution, grayscale), and the output is a color, high-resolution, high-quality image.

[0084] The traditional loss function based on convolutional neural networks is defined based on pixel-based loss. When comparing two images based on pixels, if the two images have the same perception but different pixels, the pixel loss will cause a large deviation. Perceptual loss implements perceptual comparison based on advanced convolutional neural network features without pixel-based loss.

[0085] The perceptual loss function is defined as follows:

[0086]

[0087] in and are the features of the sparse CT and original CT images extracted by the j-layer VGG19 network, where W j is the weight coefficient of the jth layer, C j and H j They represent the length and width of the j-th layer network respectively.

[0088] The absolute value loss mainly measures the difference between the pixel values ​​between the neural network output image and the reference image. Its formula is as follows:

[0089]

[0090] M and N are the number of rows and columns of the image. This formula calculates the expectation of the absolute value of the difference between the pixels of the convolutional neural network output image I and the reference image R. Where I(k, l) and R(k, l) represent the input image pixel and output image pixel of the kth row and lth column, respectively. The result of the neural network using MAE loss has better image details.

[0091] The loss function in the network is:

[0092] L loss =L mae +λL feat

[0093] The loss function in the network combines absolute value loss and perceptual loss, where λ is the weight coefficient of perceptual loss.

[0094] This embodiment uses the Pytorch framework in the Python environment on a GeForce GTX 1080Ti processor. For network optimization, the present invention uses the Adam optimizer and the nonlinear activation function ReLU. During network training, the number of iterations of the present invention can be set to 100, and the learning rate is set to an adaptive setting. In order to achieve accurate convergence, the step size is set to 2.

[0095] Step (5) uses the sparse CT image of the same part as input for testing, obtains an image with artifacts removed, and calculates the peak signal-to-noise ratio and structural similarity between the output CT image and the labeled CT image. The peak signal-to-noise ratio (PSNR) is defined as follows:

[0096]

[0097]

[0098] M and N represent the length and width of the image, L is the maximum grayscale value in the image, im is the image to be measured, and ref is the reference image. PSNR calculates the difference between the measured image im and the reference image ref. The larger the PSNR value, the smaller the difference between the measured image and the reference image, that is, the smaller the noise in the measured image im.

[0099] The definition of structural similarity index measurement (SSIM) is as follows:

[0100] SSIM(im,ref)=L(im,ref)×C(im,ref)×S(im,ref)

[0101]

[0102]

[0103]

[0104] u im and u ref denote the mean of the measured image and the reference image, σ im and σ refdenote the standard deviation of the measured image and the reference image, σ im,ref Represents the covariance between the measured image and the reference image, C1, C2 and C3 are constants. Structural similarity is used to compare the similarity between the measured image im and the reference image ref. It calculates the similarity between images from three aspects: image brightness, contrast and structure. The value range of structural similarity is between 0 and 1. The higher the structural similarity, the more similar the measured image is to the reference image.

[0105] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been discussed above with a preferred embodiment, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for removing artifacts from sparse angle CT, characterized in that: The specific steps are as follows: Step 1, preprocessing the sparse angular CT data with streak artifacts and the original CT data into .dcm format images of a specified size; Step 2, the preprocessed sparse angular CT data is passed through the VGG19 network to extract image features for subsequent training; Step 3: construct a feature fusion attention residual network; Step 4: Take the sparse CT image as input, the original CT image as the label, and the perceptual loss as the loss function for training to obtain a trained feature fusion attention residual network model; Step 5: Use the sparse CT image of the same part as input for testing to obtain an image with artifacts removed, and calculate the peak signal-to-noise ratio and structural similarity between the output CT image and the labeled CT image; The specific operation of constructing the feature fusion attention residual network in step 3 is as follows: connect between several dense residual skip modules, fuse the local features in each RSDB, and pass them to the subsequent residual block. The fth RSDB directly introduces the local features in the (f-1)th RSDB or RB and the F f,2 Perform feature fusion, where if f = 1, it is the model adaptation residual block RB, and the local feature fusion LFF is expressed as: where F f,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 It represents the result of shallow feature extraction after the f-th RSDB input passes through two Conv layers, expressed as: Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 Indicates the result of the f-1th RSDB output; After local features are fused in each RSDB, they are passed to the attention module. The feature map obtained by the attention module is passed to the next RSDB for the same operation. The specified layer features in different basic blocks are stacked together to achieve spaced jump connections and effective feature extraction. The feature map stacking is represented as: Among them, F0 is the feature map of the Conv layer in the residual block RB of the first RSDB pre-model adaptation. It will be mentioned in the subsequent FFRN network structure that in order to adapt to the RSDB structure, an RB block will be introduced in RSDB first; finally, local residual learning is introduced to further improve the information flow, F f,3 Indicates that the conversion layer function uses a convolutional layer with a kernel size of axa; Assuming the input of the main road is x, the output is T(x), the mask output is M(x), and the output of the attention module is: H i,c (x)=M i,c (x)*T i,c (x) Where i represents the spatial position of the feature, and c represents the feature channel; When mask is used as a gradient update filter during back propagation: When back-propagating the gradient update, the parameters θ of the mask part are not updated, but only the parameters of the main road are updated. Where θ is the parameter of the mask branch, and φ is the parameter of the trunk branch; Since the value of the mask is [0,1], referring to the residual learning structure of Resnet, the output of this module is a residual structure: H i,c (x)=(1+M i,c (x))*T i,c (x); The perceptual loss in step 4 is expressed as: in and are the features of the sparse CT and original CT images extracted by the j-layer VGG19 network, where W j is the weight coefficient of the jth layer, C j and H j Respectively represent the length and width of the j-th layer network; The absolute value loss mainly measures the difference between the pixel values ​​between the neural network output image and the reference image. Its formula is as follows: M and N are the number of rows and columns of the image. This formula calculates the expectation of the absolute value of the difference between the pixels of the convolutional neural network output image I and the reference image R, where I(k,l) and R(k,l) represent the input image pixel and output image pixel of the kth row and lth column, respectively. The loss function in the network combines absolute value loss and perceptual loss, expressed as follows: THE loss =L mae +λL feat Where λ is the weight coefficient of perceptual loss.

2. The method for removing artifacts from sparse angle CT according to claim 1, characterized in that: Step 1 is as follows: Step 1.1, modify the size of the head sparse CT images and original CT images in the dataset so that the resolution of each image is m×n, where m and n are the length and height of the image respectively, and the sparse CT images and original CT images are in .dcm format; Step 1.2: Convert the image of the new size in step 1.1 into a Tensor data type suitable for the Pytorch framework.

3. The method for removing artifacts from sparse angle CT according to claim 1, characterized in that: In step 2, three identical images are put together to form three channels to adapt to the VGG19 network, and the preprocessed sparse CT images and the original CT images are extracted features by the trained VGG19 model. The former is used for artifact removal training, and the latter is used for prediction training with stripe artifacts.

Images