Method for removing blocking effect of capsule endoscopy image with low compression ratio

Abstract

The invention discloses a deblocking effect method for a low-compression-ratio capsule endoscopy image. The method comprises the following steps: constructing a data set used for deblocking effect of a capsule endoscopy compressed image; constructing a capsule endoscopy deblocking effect network model which comprises a backbone network consisting of a codec and a coordinate attention module and an auxiliary branch network consisting of a quality factor predictor and a flexible controller, a quality factor is introduced as prior information to guide a capsule endoscopy deblocking effect network model to restore a compressed capsule endoscopy image, so that the purpose of deblocking effect is achieved; constructing a loss function; training the constructed deblocking effect network model by using a data set and a loss function; and inputting the capsule endoscopy compressed image of which the blocking effect is to be removed into the trained deblocking effect network model, so that a restored image of which the blocking effect is removed can be output. According to the method, the problems of incomplete block artifact removal, image edge blurring, detail loss, partial color cast and the like in the prior art are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of medical image processing methods, and in particular relates to a method for removing blocking effects of low-compression ratio capsule endoscopy images. Background Art

[0002] Capsule endoscopy realizes non-invasive examination. Patients only need to swallow an endoscope the size of a capsule to complete the whole process of examination, avoiding the discomfort of traditional endoscopic examination. Capsule endoscopy can fully observe the inside of the digestive tract and provide doctors with timely and accurate diagnosis and treatment guidance through real-time transmission of image data. In particular, it has the characteristics of comfort and non-destructiveness, and has great application value.

[0003] The capsule endoscopy system will generate tens of thousands of images during the imaging process. These images need to be transmitted to the server through the data transmission module. However, the system is limited by the network bandwidth and cannot complete the transmission of large amounts of data in a short period of time. Therefore, the system will compress a large number of images through the JPEG compression algorithm and then transmit them through the network. However, the JPEG compression algorithm is a lossy compression method. Especially under low compression ratios, the compressed capsule endoscopy images lose a lot of information in terms of details and colors, which manifests as a serious block effect in the image, affecting the doctor's judgment of the lesions.

[0004] In recent years, deep learning technology has been gradually introduced into the field of compressed image deblocking technology, and has achieved good restoration effects. However, the following problems still exist: First, it is impossible to restore the detailed information in capsule endoscopy images well, and it is easy to cause a certain degree of color cast; second, the relevant network model does not take into account the loss of quantization of color and details of the image caused by the JPEG compression algorithm in feature extraction. Summary of the invention

[0005] The purpose of the present invention is to provide a method for deblocking capsule endoscopy images with a low compression ratio, which solves the problems of incomplete removal of block artifacts, blurred image edges, loss of details and partial color cast in the prior art.

[0006] The technical solution adopted by the present invention is a method for removing blocking effects of low-compression ratio capsule endoscopy images, which is specifically implemented according to the following steps:

[0007] Step 1, constructing a data set for deblocking of capsule endoscopy compressed images;

[0008] Step 2, constructing a capsule endoscopy deblocking network model, which includes a backbone network consisting of a codec and a coordinate attention module and an auxiliary branch network consisting of a quality factor predictor and a flexible controller, and introducing the quality factor as prior information to guide the capsule endoscopy deblocking network model to restore the compressed capsule endoscopy image to achieve the purpose of deblocking;

[0009] Step 3: Construct loss function;

[0010] Step 4: Use the data set of step 1 and the loss function of step 3 to train the deblocking network model constructed in step 2;

[0011] Step 5: For the capsule endoscopy compressed image with the blocking effect to be removed, input it into the deblocking effect network model trained in step 4, and then the capsule endoscopy restored image with the blocking effect removed can be output.

[0012] The present invention is also characterized in that:

[0013] Step 1 is as follows:

[0014] Capsule endoscopy images of different organs are collected, and invalid capsule endoscopy images are removed using a deep learning network. Then, the valid capsule endoscopy images are randomly cropped and compressed using the JPEG compression algorithm to obtain compressed capsule endoscopy images, thereby constructing a dataset for deblocking capsule endoscopy compressed images.

[0015] Step 1 is implemented as follows:

[0016] Step 1.1, collect capsule endoscopy images of different organs, then randomly select some images from the collected images, and manually mark them into two categories: valid images that can be used for medical analysis and invalid images that cannot be used for medical analysis, to form a capsule endoscopy image classification dataset;

[0017] Step 1.2: Establish a capsule endoscopy image classification network, including 4 convolutional layers and 4 fully connected layers, and train the network with the capsule endoscopy image classification dataset formed in step 1.1;

[0018] Step 1.3, using the image classification network established in step 1.2 to classify the collected capsule endoscopy images, and removing invalid capsule endoscopy images that cannot be used for medical analysis according to the classification results;

[0019] Step 1.4: For effective capsule endoscopy images that can be used for medical analysis, a random number generation algorithm is used to determine the position of the random cropping area. Then, according to the selected cropping area size and position, the corresponding image blocks are cropped from the original image as the uncompressed capsule endoscopy image dataset. The uncompressed capsule endoscopy image is compressed using the JPEG compression algorithm to obtain the compressed capsule endoscopy image, thereby constructing a dataset for deblocking the capsule endoscopy compressed image.

[0020] Step 2: In the backbone network composed of the encoder-decoder and the coordinate attention module, the encoder is used to extract deep features and decouple the potential quality factor from the input compressed image. The encoder includes four feature extraction layers of different scales, and the number of channels of each scale is 64, 128, 256, and 512 respectively. Four residual blocks are used for each scale, and each residual block contains two 3×3 convolutional layers. The ReLU activation function is used between these two convolutional layers, and each scale is connected to the decoder through a jump connection; the coordinate attention module is located between the encoder and the decoder; the input of the decoder is the output feature map of the attention module and the embedding factors γ and β output by the flexible controller, and its output is a restored image. The decoder consists of three spatial feature transformation blocks, each of which includes a convolutional layer, a ReLU function activation layer, another convolutional layer, and an affine transformation layer composed of embedding factors γ and β to adaptively adjust the input features.

[0021] In step 2, the input of the quality factor predictor is the deep features of the decoder's last scale output. Through a 3-layer perceptron, the quality factor of the compressed image is predicted as supervision information.

[0022] In step 2, the input of the flexible controller is the quality factor predicted by the quality factor predictor, which is then passed through a 4-layer perceptron to learn the embedding factors γ and β.

[0023] The loss function in step 3 consists of the restored image, the uncompressed image and the quality factor, namely:

[0024]

[0025] Among them, I rec represents the restored image output by the capsule endoscopy deblocking network, I gt represents the corresponding uncompressed capsule endoscopy image, QF est represents the value of the quality factor predictor output, QF gt represents the quality factor during compression, and N represents the number of batch samples.

[0026] The beneficial effects of the present invention are:

[0027] The method of the present invention first removes invalid images from the collected capsule endoscopy images through a classification network; then the valid capsule endoscopy images are randomly cropped and compressed through a JPEG compression algorithm to construct a capsule endoscopy image deblocking effect data set; further a capsule endoscopy image deblocking effect network model guided by a quality factor prior and combined with a spatial feature transformation is designed, and the network model is trained; finally, the trained capsule endoscopy deblocking effect network model is applied to remove the block effect of the input JPEG compressed capsule endoscopy image to obtain a restored image. According to the characteristics of quantization of image color information and detail information in the encoding and decoding process of JPEG compressed images, the method of the present invention specifically designs a network structure for low compression ratio capsule endoscopy image restoration, introduces a quality factor as a priori information in the network structure, optimizes the features by embedding factors and combining spatial feature transformation, so that the restored image block artifacts and color cast are eliminated and the edge details are clear. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 is a capsule endoscopy image effect diagram before and after processing in Example 1 of the present invention;

[0030] Figure 3 is a capsule endoscopy image effect diagram before and after processing in Example 2 of the present invention;

[0031] Figure 4 It is a capsule endoscopy image effect diagram before and after processing in Example 3 of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] The method for removing blocking effect of low compression ratio capsule endoscopy image of the present invention has the following process: Figure 1 As shown, the specific implementation steps are as follows:

[0034] Step 1: Collect capsule endoscopy images of different organs, and use a deep learning network to remove invalid capsule endoscopy images, such as images affected by secretions, food residues, etc., then randomly crop the valid capsule endoscopy images and compress them using the JPEG compression algorithm to obtain compressed capsule endoscopy images, thereby constructing a data set for deblocking the capsule endoscopy compressed images; specifically:

[0035] Step 1.1, collect capsule endoscopy images of different organs, then randomly select some images from the collected images, and manually mark them into two categories: valid images that can be used for medical analysis and invalid images that cannot be used for medical analysis, to form a capsule endoscopy image classification dataset;

[0036] Step 1.2: Establish a capsule endoscopy image classification network, including 4 convolutional layers and 4 fully connected layers, and train the network with the capsule endoscopy image classification dataset formed in step 1.1;

[0037] Step 1.3, using the image classification network established in step 1.2 to classify the collected capsule endoscopy images, and removing invalid capsule endoscopy images that cannot be used for medical analysis according to the classification results;

[0038] Step 1.4: For effective capsule endoscopy images that can be used for medical analysis, a random number generation algorithm is used to determine the position of the random cropping area. Then, according to the selected cropping area size and position, the corresponding image blocks are cropped from the original image as the uncompressed capsule endoscopy image dataset. The uncompressed capsule endoscopy image is compressed using the JPEG compression algorithm to obtain the compressed capsule endoscopy image, thereby constructing a dataset for deblocking the capsule endoscopy compressed image.

[0039] Step 2, constructing a capsule endoscopy deblocking network model, which includes a backbone network consisting of a codec and a coordinate attention module and an auxiliary branch network consisting of a quality factor predictor and a flexible controller, and introducing the quality factor as prior information to guide the capsule endoscopy deblocking network model to restore the compressed capsule endoscopy image to achieve the purpose of deblocking;

[0040] Among them, in the backbone network composed of the encoder-decoder and the coordinate attention module, the encoder is used to extract deep features and decouple the potential quality factor from the input compressed image. The encoder includes 4 feature extraction layers of different scales, and the number of channels of each scale is 64, 128, 256, and 512 respectively. Four residual blocks are used for each scale, and each residual block contains two 3×3 convolutional layers. The ReLU activation function is used between these two convolutional layers, and each scale is connected to the decoder through a jump connection; the coordinate attention module is located between the encoder and the decoder; the input of the decoder is the output feature map of the attention module and the embedding factors γ and β output by the flexible controller, and its output is the restored image. The decoder consists of 3 spatial feature transformation blocks, each of which includes a convolutional layer, a ReLU function activation layer, another convolutional layer and an affine transformation layer composed of embedding factors γ and β to adaptively adjust the input features.

[0041] In addition, the input of the quality factor predictor is the deep features of the last scale output of the decoder. Through a 3-layer perceptron, the quality factor of the compressed image is predicted as supervision information.

[0042] The input of the flexible controller is the quality factor predicted by the quality factor predictor, which is then passed through a 4-layer perceptron to learn the embedding factors γ and β.

[0043] Step 3: Construct a loss function consisting of the restored image, the uncompressed image and the quality factor, that is

[0044]

[0045] Among them, I rec represents the restored image output by the capsule endoscopy deblocking network, I gt represents the corresponding uncompressed capsule endoscopy image, QF est represents the value of the quality factor predictor output, QF gt represents the quality factor during compression, and N represents the number of batch samples.

[0046] Step 4: Use the data set of step 1 and the loss function of step 3 to train the capsule endoscopy deblocking network model constructed in step 2;

[0047] Step 5: For the capsule endoscopy compressed image with the blocking effect to be removed, input it into the capsule endoscopy deblocking network model trained in step 4, and then output the capsule endoscopy restored image with the blocking effect removed.

[0048] Example 1

[0049] The above method is used to restore the capsule endoscopy image compressed by JPEG. The results are as follows: Figure 2 As shown, the image on the left is the original uncompressed capsule endoscopy image, the image in the middle is the capsule endoscopy image compressed by JPEG, and the image on the right is the image restored after removing the block effect of the image in the middle using the method of the present invention.

[0050] Example 2

[0051] The above method is used to restore the capsule endoscopy image compressed by JPEG. The results are as follows: Figure 3 As shown, the image on the left is the original uncompressed capsule endoscopy image, the image in the middle is the capsule endoscopy image compressed by JPEG, and the image on the right is the image restored after removing the block effect of the image in the middle using the method of the present invention.

[0052] Example 3

[0053] The above method is used to restore the capsule endoscopy image compressed by JPEG. The results are as follows: Figure 4 As shown, the image on the left is the original uncompressed capsule endoscopy image, the image in the middle is the capsule endoscopy image compressed by JPEG, and the image on the right is the image restored after removing the block effect of the image in the middle using the method of the present invention.

[0054] from Figure 2 , Figure 3 and Figure 4 It can be seen that the capsule endoscopy image after JPEG compression has block artifacts, blurred edges, unclear details and certain color cast. The block artifacts of the capsule endoscopy image restored by the method of the present invention are well removed, the edges and details are clear, the color cast is overcome, and it can be very realistic and close to the original uncompressed image.

Claims

1. A method for removing blocking effects of low compression ratio capsule endoscopy images, characterized in that: Follow the steps below to implement it: Step 1, constructing a data set for deblocking capsule endoscopy compressed images; Step 2, constructing a capsule endoscopy deblocking network model, which includes a backbone network consisting of a codec and a coordinate attention module and an auxiliary branch network consisting of a quality factor predictor and a flexible controller, and introducing the quality factor as prior information to guide the capsule endoscopy deblocking network model to restore the compressed capsule endoscopy image to achieve the purpose of deblocking; Step 3: Construct loss function; Step 4: Use the data set of step 1 and the loss function of step 3 to train the deblocking network model constructed in step 2; Step 5: For the capsule endoscopy compressed image with the blocking effect to be removed, input it into the deblocking effect network model trained in step 4, and then the capsule endoscopy restored image with the blocking effect removed can be output.

2. The method for deblocking low-compression ratio capsule endoscopy images according to claim 1, characterized in that: Step 1 is as follows: Capsule endoscopy images of different organs are collected, and invalid capsule endoscopy images are removed using a deep learning network. Then, the valid capsule endoscopy images are randomly cropped and compressed using the JPEG compression algorithm to obtain compressed capsule endoscopy images, thereby constructing a dataset for deblocking capsule endoscopy compressed images.

3. The method for deblocking low-compression ratio capsule endoscopy images according to claim 2, characterized in that: Step 1 is implemented according to the following steps: Step 1.1, collect capsule endoscopy images of different organs, then randomly select some images from the collected images, and manually mark them into two categories: valid images that can be used for medical analysis and invalid images that cannot be used for medical analysis, to form a capsule endoscopy image classification dataset; Step 1.2: Establish a capsule endoscopy image classification network, including 4 convolutional layers and 4 fully connected layers, and train the network with the capsule endoscopy image classification dataset formed in step 1.1; Step 1.3, using the image classification network established in step 1.2 to classify the collected capsule endoscopy images, and removing invalid capsule endoscopy images that cannot be used for medical analysis according to the classification results; Step 1.4: For effective capsule endoscopy images that can be used for medical analysis, a random number generation algorithm is used to determine the position of the random cropping area. Then, according to the selected cropping area size and position, the corresponding image blocks are cropped from the original image as the uncompressed capsule endoscopy image dataset. The uncompressed capsule endoscopy image is compressed using the JPEG compression algorithm to obtain the compressed capsule endoscopy image, thereby constructing a dataset for deblocking the capsule endoscopy compressed image.

4. The method for deblocking low-compression ratio capsule endoscopy images according to claim 1, characterized in that: Step 2: In the backbone network composed of the encoder-decoder and the coordinate attention module, the encoder is used to extract deep features and decouple the potential quality factor from the input compressed image. The encoder includes four feature extraction layers of different scales, and the number of channels of each scale is 64, 128, 256, and 512 respectively. Four residual blocks are used for each scale, and each residual block contains two 3×3 convolutional layers. The ReLU activation function is used between these two convolutional layers, and each scale is connected to the decoder through a jump connection; the coordinate attention module is located between the encoder and the decoder; the input of the decoder is the output feature map of the attention module and the embedding factors γ and β output by the flexible controller, and its output is a restored image. The decoder consists of three spatial feature transformation blocks, each of which includes a convolutional layer, a ReLU function activation layer, another convolutional layer, and an affine transformation layer composed of embedding factors γ and β to adaptively adjust the input features.

5. The method for deblocking low-compression-ratio capsule endoscopy images according to claim 4, characterized in that: In step 2, the input of the quality factor predictor is the deep features of the last scale output of the decoder. Through a 3-layer perceptron, the quality factor of the compressed image is predicted as supervision information.

6. The method for deblocking low-compression-ratio capsule endoscopy images according to claim 5, characterized in that: In step 2, the input of the flexible controller is the quality factor predicted by the quality factor predictor, and then the embedding factors γ and β are learned through a 4-layer perceptron.

7. The method for deblocking low-compression capsule endoscopy images according to claim 1, characterized in that: The loss function in step 3 consists of the restored image, the uncompressed image and the quality factor, namely: Among them, I rec represents the restored image output by the capsule endoscopy deblocking network, I gt represents the corresponding uncompressed capsule endoscopy image, QF est represents the value of the quality factor predictor output, QF gt represents the quality factor during compression, and N represents the number of batch samples.

Images