A method for removing artifacts from CBCT images

Abstract

The present invention discloses a method for removing artifacts from CBCT images. The method is based on contextual loss and feature fusion residual network. First, feature extraction is performed on input CBCT images with artifacts and CT images to obtain feature maps. The obtained feature maps are trained by an improved feature fusion residual network with contextual loss as the loss function to learn the structural similarity between CBCT and CT images. The CT images are used as labels so that the CBCT images can remove artifacts but keep their structures unchanged. Finally, the CBCT images are used as input and the trained network model is used to remove artifacts. The method utilizes contextual loss to allow misalignment between input images, and can be well applied to the feature that CBCT and CT medical images cannot be strictly aligned, and finally achieves a fast and effective artifact removal effect.

Description

Technical Field

[0001] The invention relates to a CBCT image artifact removal method, in particular to a CBCT image artifact removal method based on contextual loss and feature fusion residual network, and belongs to the technical field of image processing. Background Art

[0002] Cone-beam computed tomography (CBCT) has the advantages of short scanning time and low X-ray dose, and is widely used in the diagnosis of head and neck diseases. However, compared with traditional fan-beam computed tomography (CT), the quality of CBCT images is reduced due to X-ray scattering and projection truncation. These problems lead to large scattering artifacts, which affect the application prospects of CBCT in many fields. Therefore, the main challenge facing CBCT is scattering artifacts.

[0003] In recent years, many researchers have studied CBCT scatter correction. In the published literature, the main correction methods can be divided into two categories according to their processing methods. One is the hardware processing method, which reduces the number of scattered X-rays reaching the detector by considering the geometric conditions of X-ray imaging, thereby suppressing scatter. This method can prevent the scattered light generated during the attenuation process from reaching the detector to the greatest extent. Common methods include air gap method, collimator method, filter method, anti-scatter grating method and method using modulator. These methods can prevent scattered light from reaching the detector, however, the hardware-based correction method adds hardware equipment for filtering out scatter on the basis of the original CBCT system, which introduces operational difficulties in the CBCT system and increases the cost of the entire process. Therefore, more and more people are studying software-based methods.

[0004] Another method is to use convolutional neural networks to deal with scattering artifacts, and achieve satisfactory results. With the powerful function of deep learning, end-to-end deep supervised learning and convolutional neural networks have made great progress in natural image problems. Methods based on deep learning have been widely used in CBCT image artifact removal.

[0005] Compared with conventional fan-beam CT images, CBCT image quality is reduced due to x-ray scattering and truncated projections, and has the disadvantages of low soft tissue contrast, noise, and artifacts, which hinder the effective use of CBCT in many potential applications. Therefore, it is crucial to improve the image quality of CBCT so that cone-beam CT can be better used in clinical medicine. Summary of the invention

[0006] In view of the above-mentioned problems in the prior art and the selection of materials, the present invention provides a fast, concise and robust method for removing artifacts in CBCT images.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows: a CBCT image artifact removal method, the method is based on contextual loss and feature fusion residual network, and includes the following steps:

[0008] Step 1: preprocess the CBCT and CT data into 512*512 size .raw format images;

[0009] Step 2: The preprocessed CBCT and CT data are passed through the VGG19 network to extract image features for subsequent training;

[0010] Step 3, calculating the structural similarity between the obtained CBCT and CT feature maps;

[0011] Step 4: Construct a feature fusion residual network. The residual blocks in the network are connected between blocks, and the local features of each residual block are fused and finally passed to the subsequent residual blocks. The feature fusion result of the previous module is used as the input of the next feature fusion module, and then the features after local feature fusion are stacked, and residual learning is used to integrate feature information to form the basic network architecture.

[0012] Step 5: Take the CBCT image as input, the CT image as label, and the contextual loss as loss function for training to obtain a trained feature fusion residual network model;

[0013] Step 6: Use the CBCT image of the same part as input for testing to obtain an image with artifacts removed.

[0014] Furthermore, the image preprocessing in step 1 includes the following sub-steps:

[0015] Step 1.1, cut the chest CBCT images and CT images in the dataset into the same size, so that the resolution of each image is m×n, where m and n are the length and height of the image respectively; the CBCT images and CT images are in .dcm format;

[0016] Step 1.2, convert the image captured in the previous step into .raw format suitable for the web.

[0017] Furthermore, the specific implementation steps of feature extraction in step 2 are as follows:

[0018] Since the input of the VGG network is a three-channel RGB image, and the medical image is a single-channel grayscale image, three identical images are put together to form a three-channel (512, 512, 3) to adapt to the network, and then the processed images are passed through the trained VGG19 model for feature extraction.

[0019] Furthermore, in step 3, the specific steps of calculating feature similarity are as follows:

[0020] Step 3.1, calculate feature similarity:

[0021] The feature similarity is defined using contextual loss based on cosine distance, assuming s i and t j are the features extracted by VGG19 from the input CBCT image S and the label CT image T, respectively. ij is the cosine distance between features, and its calculation formula is:

[0022] in

[0023] Among them, N represents the number of features extracted by VGG19, μ t is the normalization of the label image features; for any k≠j, d ij < <d ik , then s i and t j It is similar;

[0024] Step 3.2, calculate image similarity:

[0025] Based on the similarity calculation formula between features, the similarity calculation formula between images is further defined as:

[0026]

[0027] CX represents the similarity of features. ij Representative features i and t j The similarity between them; using an image to calculate the similarity with itself, we can see that the similarity is 1, that is, CX ii =1, so CX(s,s)=1; on the contrary, using an image to compare with a completely different image, the similarity Then there is

[0028] The loss function is defined as follows:

[0029] L CX (s,t,l)=-log(CX(φ l (s),φl (t)))

[0030] where φ l (s) and φ l (t) are the features of CBCT and CT images extracted by the l-layer VGG19 network.

[0031] Furthermore, in step 4, the method for constructing a feature fusion residual network is as follows:

[0032] The dense residual block (RDB) extracts multi-level features from the original image and further improves the network performance. It utilizes all layers through local dense connections, and then adaptively retains the accumulated features through local feature fusion (LFF). RDB combines shallow features and deep features for global feature fusion (GFF), and obtains global dense features through global residual learning.

[0033] The blocks are connected to each other, the local features in each RSDB are fused, and finally passed to the subsequent residual block; the f-th RSDB directly introduces the local features in the (f-1)-th RSDB or RB for feature fusion. If f = 1, the model adapts to the residual block RB, and the local feature fusion LFF is expressed as:

[0034]

[0035] where F f,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 The result of shallow feature extraction after the f-th RSDB input passes through two Conv layers can be expressed as:

[0036]

[0037] Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 Indicates the result of the f-1th RSDB output;

[0038] After local feature fusion in each RSDB, no processing is performed and the stacked feature map is directly passed to the next RSDB for the same operation. This method stacks the specified layer features in different basic blocks together to achieve spaced jump connections and effective feature extraction.

[0039] Furthermore, the implementation of the method includes at least one computing device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that when the computer program is loaded into the processor, the dynamic expression recognition method based on the aforementioned spatiotemporal graph convolutional neural network is implemented.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention is based on contextual loss and does not require strict alignment of input and label. It only requires CBCT and CT data of the same part, which solves the difficulty of CBCT data alignment.

[0042] 2. The present invention uses a similarity measurement method to learn the spatial structure of a CT image without artifacts, which well preserves the spatial information of the original data and is not prone to deformation;

[0043] 3. The present invention adds a small module to improve the resolution at the end of the network, so that the final result has a higher resolution and it is easier to observe the effectiveness of the experimental results;

[0044] 4. Compared with the traditional residual network, the network of the present invention makes more full use of local features and fully integrates the features of the previous and next modules to learn more detailed information;

[0045] 5. The present invention does not change the size of the image during the training process, and perfectly preserves the details and edge information of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is an overall flow chart of the method of an embodiment of the present invention.

[0047] Figure 2 It is the RSDB module of the present invention.

[0048] Figure 3 It is a network model diagram constructed by the present invention.

[0049] Figure 4 It is the overall structural diagram of the present invention. DETAILED DESCRIPTION

[0050] In order to better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0051] Example 1

[0052] like Figure 1 As shown, the present embodiment discloses a CBCT image artifact removal method based on contextual loss and feature fusion residual network, which specifically includes the following steps:

[0053] Step (1) preprocesses each image in the CBCT and CT datasets so that each image has the same size and is in a format suitable for network input. This includes the following sub-steps:

[0054] (1.1) The CBCT and CT data (.dcm format) obtained from the hospital were cut into the same size, with a pixel size of m*n, where m and n are the length and height of the image respectively. In the experiment, m=n=512 was taken;

[0055] (1.2) Convert the resized .dcm data into .raw format data suitable for the network.

[0056] The data used in this embodiment comes from a hospital, and then we make our own data sets for different parts, including chest, head and pelvis. In practice, it can also be adapted to other parts. The data is preprocessed, and each image is cut into 512*512 pixels, and its format is changed to .raw format suitable for network training.

[0057] Step (2) extracts features from the preprocessed image. The specific operation method is as follows:

[0058] The preprocessed data are all single-channel medical images, and the input of the selected VGG19 network is a three-channel color image, so the input image needs to be increased in dimension to adapt to the network. We select three identical images to form a three-channel (512, 512, 3) input and perform feature extraction through the VGG19 network.

[0059] For the universality of the experiment, the open source trained VGG19 model imagenet-vgg-verydeep-19 version is used to extract features in this embodiment.

[0060] Step (3) calculates feature similarity, and the specific operation is as follows:

[0061] We use a contextual loss based on cosine distance to define feature similarity, because this loss function does not require strict alignment of the input and label images. Let s i and t j are the features extracted by VGG19 from the input CBCT image S and the label CT image T, respectively. ij is the cosine distance between features, N is the number of features extracted by VGG19, μ t It is the normalization of the label image features, and its calculation formula is:

[0062] in

[0063] If for any k≠j, there is d ij < <d ik , then s i and t j is similar. The above formula can be simplified as follows:

[0064]

[0065] Here we set ε to 10 -5 We transform the distance formula into a similarity formula through exponential transformation, which is defined as follows:

[0066]

[0067] Here h is the bandwidth parameter, set to 0.5. Finally, we use the scale-invariant version of the normalized similarity to define the similarity CX between features ij :

[0068]

[0069] (3.2) Image similarity

[0070] Based on the similarity calculation formula between features, the similarity calculation formula between images can be further defined as:

[0071]

[0072] The loss function is defined as follows:

[0073] L CX (s,t,l)=-log(CX(φ l (s),φ l (t)))

[0074] where φ l (s) and φ l (t) are the features of CBCT and CT images extracted by the l-layer VGG19 network.

[0075] This example trains a network to obtain a mapping from a source image s to an output image G(s). Here, for the network G, we use a feature fusion residual network, and the source images include CBCT images with artifacts in the chest and head regions. The loss function we used in the experiment can be expressed as:

[0076] L(G)=λ·L CX (G(S),t,l t )+L CX (G(S),s,l s )

[0077] The first loss on the right side of the equal sign in the above formula represents the similarity between the generated image and the target image, and the second one is used to measure the similarity between the generated image and the source image. λ is a constant used to adjust the ratio of the two loss functions, which is set to 5 in our experiment.

[0078] Step (4) constructs a feature fusion residual network, the steps are as follows:

[0079] Dense residual block (RDB) can extract multi-level features in the original image and further improve the network performance. It utilizes all layers through local dense connections and then adaptively retains the accumulated features through local feature fusion (LFF). RDB combines shallow features and deep features for global feature fusion (GFF) to obtain global dense features through global residual learning.

[0080] We choose to connect between blocks, fuse the local features in each RSDB, and finally pass them to the subsequent residual block. We directly introduce the local features in the (f-1)th RSDB or RB (if f=1, the model adapts to the residual block RB) in the fth RSDB and perform feature fusion. The local feature fusion (LFF) can be expressed as:

[0081]

[0082] where F f,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 The result of shallow feature extraction after the f-th RSDB input passes through two Conv layers can be expressed as:

[0083]

[0084] Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 It represents the result of the f-1th RSDB output. After the local features are fused in each RSDB, no processing is done and the stacked feature map is directly passed to the next RSDB for the same operation. This method stacks the specified layer features in different basic blocks together to achieve spaced jump connections and effective feature extraction.

[0085] The convolutional neural network in this example is Figure 2As shown in the figure, the residual block (RB) structure is as follows: the first layer is a 3*3 convolution layer, followed by a ReLU layer, then a 3*3 convolution layer, a 3*1 convolution layer, and a 1*3 convolution layer. While increasing the receptive field, it speeds up the calculation. The dense residual skip module (RSDB) is shown in the figure. Figure 1 shown.

[0086] Step (5) takes CBCT as input, CT as label, and contextual loss as loss function to train the feature fusion residual network. During the network training process, the contextual loss is calculated based on the difference pixel values ​​between the CBCT and CT images and the features of the corresponding convolutional layer. As the background loss changes, the image value of the reconstructed CT image will also be updated. During the experiment, we used an iterative optimization method to correct the CBCT image. According to the results of artifact removal, the contextual loss function, network parameters, and the ratio of feature sample size are adjusted when necessary, and the network training is repeated until the artifacts are effectively removed.

[0087] This example uses the Tensorflow library in the Python environment on a GeForce GTX 1080Ti processor. For network optimization, we used the Adam optimizer and the nonlinear activation function ReLU in our experiments. During network training, we set the number of epochs to 200 and the learning rate to . In order to achieve accurate convergence, the step size was set to 2.

[0088] Step (6), taking the CBCT image of the same part as input for testing, and obtaining an image with artifacts removed.

[0089] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A CBCT image artifact removal method, characterized in that: The method is based on contextual loss and feature fusion residual network, and includes the following steps: Step 1: preprocess the CBCT and CT data into 512*512 size .raw format images; Step 2: The preprocessed CBCT and CT data are passed through the VGG19 network to extract image features for subsequent training; Step 3, calculating the structural similarity between the obtained CBCT and CT feature maps; the specific contents are as follows: Step 3.1, calculate feature similarity: The feature similarity is defined using contextual loss based on cosine distance, assuming s i and t j are the features extracted by VGG19 from the input CBCT image S and the label CT image T, respectively. ij is the cosine distance between features, and its calculation formula is: in ; Among them, N represents the number of features extracted by VGG19, μ t is the normalization of the label image features; for any k≠j, d ij < <d ik , then s i and t j It is similar; Step 3.2, calculate image similarity: Based on the similarity calculation formula between features, the similarity calculation formula between images is further defined as: ； CX represents the similarity of features, and CXij represents the similarity between features si and tj. When an image is used to calculate the similarity with itself, its similarity is 1, that is, CXii = 1, so CX(s, s) = 1. On the contrary, when an image is used to compare with a completely different image, its similarity is Then there is ; The loss function is defined as follows: L CX (s,t ,l)=-log(CX(φ l (s),φ l (t))) where φ l (s) and φ l (t) are the features of CBCT and CT images extracted by the VGG19 network with l layers; Step 4: Construct a feature fusion residual network. The residual blocks in the network are connected between blocks, and the local features of each residual block are fused and finally passed to the subsequent residual blocks. The feature fusion result of the previous module is used as the input of the next feature fusion module, and then the features after local feature fusion are stacked, and residual learning is used to integrate feature information to form the basic network architecture. The method for constructing a feature fusion residual network is as follows: The dense residual block (RDB) extracts multi-level features from the original image and further improves the network performance. It utilizes all layers through local dense connections, and then adaptively retains the accumulated features through local feature fusion (LFF). RDB combines shallow features and deep features for global feature fusion (GFF), and obtains global dense features through global residual learning. The blocks are connected to each other, the local features in each RSDB are fused, and finally passed to the subsequent residual block; the f-th RSDB directly introduces the local features in the (f-1)-th RSDB or RB for feature fusion. If f = 1, the model adapts to the residual block RB, and the local feature fusion LFF is expressed as: ; Among them, F f，,LF represents the f-th RSDB local feature fusion result, Represents the Concat feature map stacking function, F f-1,LF represents the (f-1)th RSDB local feature fusion result, F f,2 It represents the result of shallow feature extraction after the f-th RSDB input passes through two Conv layers, expressed as: ; Where δ represents the ReLU activation function, W f,2 and W f,1 They represent the weights of the second and first convolutional layers, b2 and b1 represent the biases of the second and first convolutional layers, respectively. f-1 Indicates the result of the f-1th RSDB output; After local feature fusion in each RSDB, no processing is done and the stacked feature map is directly passed to the next RSDB for the same operation. This method stacks the specified layer features in different basic blocks together to achieve spaced skip connections and effective feature extraction. Step 5: Take the CBCT image as input, the CT image as label, and the contextual loss as loss function for training to obtain a trained feature fusion residual network model; Step 6: Use the CBCT image of the same part as input for testing to obtain an image with artifacts removed.

2. The CBCT image artifact removal method according to claim 1, characterized in that: The image preprocessing in step 1 includes the following sub-steps: Step 1.1, cut the chest CBCT images and CT images in the dataset into the same size, so that the resolution of each image is m×n, where m and n are the length and height of the image respectively; the CBCT images and CT images are in .dcm format; Step 1.2, convert the image captured in the previous step into .raw format suitable for the web.

3. The CBCT image artifact removal method according to claim 1, characterized in that: The specific implementation steps of feature extraction in step 2 are as follows: Since the input of the VGG network is a three-channel RGB image, and the medical image is a single-channel grayscale image, three identical images are put together to form a three-channel (512, 512, 3) to adapt to the network, and then the processed images are passed through the trained VGG19 model for feature extraction.

Images