An image fusion-based endoscope image enhancement method

Abstract

The present application relates to a kind of based on image fusion enteroscopy image enhancement method, which is designed for endoscope image illumination uneven, contrast is low, texture detail information loss a kind of enhancement method, it includes four steps: (1) create dataset;(2) image is inputed into encoder-decoder multi-attention U-Net network, carries out depth network model training to obtain global enhancement image;(3) image is inputed into segmentation network HarDNet-MSEG, carries out depth network model training to obtain polyp segmentation mask image;(4) judge whether it contains polyp, if it contains, carry out image fusion, otherwise output global enhancement image;(5) the global enhancement image obtained by encoder-decoder multi-attention U-Net network and the mask image obtained by HarDNet-MSEG are fused to output final local enhancement image.The enteroscopy image enhancement method based on image fusion proposed in the present application can attract people's attention to the lesion area more than other image enhancement algorithms, and has medical value applicable to clinical diagnosis.

Description

Technical Field

[0001] This invention relates to the fields of medical image enhancement and computer network technology, and in particular to a method for enhancing intestinal endoscopy images based on image fusion. Background Technology

[0002] Intestinal polyps are a leading cause of cancer. Endoscopy is a standard visual diagnostic procedure for detecting colon polyps. However, intestinal images obtained directly by endoscopy are prone to problems such as uneven lighting, low contrast, and loss of texture detail, leading to decreased accuracy of manual examination. This affects the accuracy of physicians' disease analysis and diagnosis. Intestinal image enhancement can provide detailed edge and texture information for clinical analysis. Therefore, research on intestinal image enhancement is crucial to helping colonoscopists reduce misdiagnosis and missed diagnosis rates.

[0003] Researchers have proposed several classic image enhancement methods, but applying them to intestinal endoscopic images with low contrast and non-uniform illumination often leads to problems such as over-enhancement of local background areas and insufficient enhancement of lesion areas. In recent years, deep learning methods have performed well in the field of natural image enhancement. However, the lack of publicly available paired datasets for intestinal image enhancement has hindered the development of deep learning in intestinal endoscopic image enhancement. Unsupervised neural network frameworks applied to intestinal endoscopic image enhancement tasks often fail to yield high-quality image enhancement results. Furthermore, deep learning-based image enhancement methods aim for global enhancement; due to the unique characteristics of the in vivo environment, natural image enhancement algorithms are not entirely suitable for intestinal endoscopic images. Summary of the Invention

[0004] The purpose of this invention is to address the lack of suitable datasets by constructing the first dataset applicable to image enhancement and segmentation. To overcome the shortcomings of existing technologies, it provides an image enhancement method for intestinal endoscopy based on image fusion. This method, using Python and the PyTorch framework, enhances intestinal images and performs both global and local enhancement tasks. The proposed EDMAU method preserves more image details in non-polyp images and adjusts global image illumination. Combining deep learning-based polyp segmentation and image enhancement techniques significantly improves image information efficiency, resulting in richer details in lesion areas within polyp images.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An image enhancement method for intestinal endoscopy based on image fusion, the steps of which are as follows:

[0007] Step 1: Create a dataset that can be used for enhanced intestinal endoscopy and polyp segmentation;

[0008] Step 2: Input the low-quality / high-quality images from the dataset into the encoder-decoder multi-attention U-Net network, train the deep network model, and make predictions to obtain globally enhanced images;

[0009] Step 3: Input the low-quality / mask images from the dataset into the segmentation network HarDNet-MSEG, train and predict the deep network model to obtain polyp segmentation mask images;

[0010] Step 4: Determine if polyps are present. If present, perform image fusion; otherwise, output a globally enhanced image.

[0011] Step 5: Fuse the global augmented image obtained by the encoder-decoder multi-attention U-Net network and the mask image obtained by HarDNet-MSEG to output the final local augmented image.

[0012] Preferably, the dataset creation in step 1 includes the following operations:

[0013] 1-1: Collect real intestinal images and corresponding polyp segmentation masks from some publicly available open-source medical imaging datasets related to gastrointestinal polyps;

[0014] 1-2: Adjust the resolution of the collected images to 256×256 pixels;

[0015] 1-3: Use Adobe Lightroom to adjust real intestinal images to obtain low-quality images and high-quality reference images;

[0016] 1-4: Invite volunteers to subjectively evaluate the images obtained in 1-3 in order to obtain the best image output.

[0017] 1-5: Divide the obtained dataset into training and test sets in a 2:1 ratio.

[0018] Preferably, the encoder-decoder multi-attention U-Net network model training and prediction in step 2 includes the following operations:

[0019] 2-1: The Encoder-Decoder Multi-Attention U-Net (EDMAU) network is an image augmentation network consisting of three parts: input, encoding attention, decoding attention, and output; multi-attention refers to detail attention map and brightness attention map; low-quality images and high-quality reference images from the dataset are input into EDMAU for training;

[0020] 2-2: The objective function uses a combination of four loss functions: absolute error loss, mean squared error loss, structural similarity loss, and perceptual loss.

[0021] 2-3: The network model uses an optimization algorithm with a learning rate of 0.0002 and is trained for a total of 200 epochs. After training, the corresponding global augmentation network model training parameters are obtained.

[0022] 2-4: Input the test set into the trained global augmentation network model to obtain the globally augmented image I predicted by the model. eg .

[0023] Preferably, step 3, the training and prediction of the HarDNet-MSEG network model, includes the following operations:

[0024] 3-1: HarDNet-MSEG is a segmentation network for intestinal polyps. It is trained by inputting low-quality images from the dataset and the corresponding polyp segmentation masks into EDMAU.

[0025] 3-2: The network model uses an optimization algorithm with a learning rate of 0.0001 and is trained for a total of 100 epochs. After training, the corresponding polyp segmentation network model training parameters are obtained.

[0026] 3-3: Input the test set into the trained HarDNet-MSEG network model to obtain the model's predicted precise segmentation mask image of the polyp location. mask .

[0027] Preferably, the determination of whether a polyp is present in step 4 is based on the mask image to determine whether a polyp is present in the image, thus distinguishing between global enhancement and further local enhancement; if the image does not contain polyps, the final globally enhanced image I is obtained. eg If polyps are present, proceed to step 5 to remove I. mask and I eg The final locally enhanced image I is obtained by fusion. el .

[0028] The image fusion in step 5 includes the following operations:

[0029] 5-1: For I mask Extracted I eg The lesion area retains the EDMAU enhancement effect;

[0030] 5-2: For I mask Extracted I eg Gamma correction was used on non-lesion areas to achieve local enhancement of polyp images. el .

[0031] Compared with the prior art, the present invention has the following obvious and prominent substantive features and significant advantages:

[0032] 1. The first synthetic dataset of paired low-light / normal-light and low-light / corresponding mask intestinal endoscopy images was constructed.

[0033] 2. A novel end-to-end global image enhancement network architecture, EDMAU, is proposed to address the problem of low-light, non-uniform illumination image enhancement. Specifically, a detail attention map is added to the network's decoder to highlight texture detail features, and a brightness attention map is introduced into the network's encoder to improve global image illumination.

[0034] 3. A novel image fusion strategy is proposed, which highlights the details of the region of interest by weakening the background region.

[0035] 4. Compared with several existing methods, this invention achieves advanced results. EDMAU can effectively eliminate non-uniform illumination effects and improve image texture details. Locally enhanced images can help doctors quickly and clearly observe regional texture details, and have certain clinical applicability. Attached Figure Description

[0036] Figure 1 This is a flowchart of the method of the present invention.

[0037] Figure 2 This is an overall flowchart of the method of the present invention.

[0038] Figure 3 This invention relates to the global image enhancement network EDMAU architecture.

[0039] Figure 4 This is a comparison chart of the enhanced prediction results of the EDMAU network model trained by the method of this invention on the constructed dataset and the results of several existing methods.

[0040] Figure 5 This is a comparison diagram showing the significant changes in the final locally enhanced image obtained by the proposed fusion method of this invention compared to the original input image without enhancement. Detailed Implementation

[0041] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0042] Example 1:

[0043] See Figure 1 An image enhancement method for intestinal endoscopy based on image fusion includes the following steps:

[0044] Step 1: Create a dataset that can be used for intestinal endoscopy image enhancement and polyp segmentation;

[0045] Step 2: Input the low-quality images and high-quality reference images from the dataset into the encoder-decoder multi-attention U-Net network, train the deep network model, and make predictions to obtain globally augmented images;

[0046] Step 3: Input the low-quality images and corresponding polyp segmentation mask images from the dataset into the segmentation network HarDNet-MSEG, train and predict the deep network model to obtain polyp segmentation mask images.

[0047] Step 4: Determine if polyps are present. If present, perform image fusion; otherwise, output a globally enhanced image.

[0048] Step 5: Fuse the global augmented image obtained by the encoder-decoder multi-attention U-Net network and the mask image obtained by HarDNet-MSEG to output the final local augmented image.

[0049] The method in this embodiment can use the constructed dataset to train and test the network model, accurately enhancing the detailed information of the region of interest to the doctor.

[0050] Example 2:

[0051] This embodiment is basically the same as Embodiment 1, with the following differences:

[0052] The dataset creation in step 1 includes the following operations:

[0053] 1-1: 2108 real intestinal images and their corresponding polyp segmentation masks were collected from several publicly available open-source medical imaging datasets related to gastrointestinal polyps, including Kvasir-SEG dataset, ETIS-Larib Polyp DB, and CVC-EndoSceneStill.

[0054] 1-2: Adjust the resolution of the collected images to 256×256 pixels;

[0055] 1-3: Some well-lit images were degraded using Lightroom to obtain low-quality images with uneven lighting. All collected images were manually retouched using Lightroom to adjust the real intestinal endoscopy images to obtain high-quality reference images.

[0056] 1-4: Volunteers are invited to subjectively evaluate the images obtained in 1-3 to produce satisfactory output. For sequences that do not produce satisfactory output, the images will be re-edited to select the best output for each sequence.

[0057] 1-5: Divide the obtained dataset into training and test sets in a 2:1 ratio.

[0058] Step 2, which involves acquiring the multi-attention map and training and predicting the encoder-decoder multi-attention U-Net network model, includes the following operations:

[0059] 2-1: Multiple attention maps refer to detail attention maps and luminance attention maps. This invention proposes a novel detail attention map and incorporates it into the decoding and encoding stage of the U-Net network to highlight detailed features and reduce the loss of some information during image enhancement. Secondly, a luminance attention map is introduced into the encoding stage of the U-Net network to improve the global image illumination.

[0060] The formula for calculating the detail attention map is as follows:

[0061]

[0062] Among them, I low This indicates the original, low-quality image. f canny This indicates that the Canny operator is used to process the image. Mp This indicates that Max-pooling is applied to the edge image. D1, ..., D8 constitute a detail attention map.

[0063] The brightness attention map is a low-quality image of the original RGB color space. low Convert to single-channel grayscale image I g Normalize it to [0,1]. Then calculate 1-I. g This serves as our brightness attention map.

[0064] 2-2: The Encoder-Decoder Multi-Attention U-Net (EDMAU) network is an image augmentation network consisting of three parts: input, encoding attention, decoding attention, and output. Low-quality images from the training set of the constructed dataset and high-quality reference images are input into EDMAU for training, resulting in a globally augmented image I. eg A complete network structure is as follows Figure 3 ;

[0065] The following is based on Figure 3 A detailed explanation follows:

[0066] The EDMAU network model proposed in this invention uses the U-Net network as the basic network for feature extraction, and mainly consists of four parts: input, encoding attention, decoding attention, and output.

[0067] The input module takes in a feature map with 4 channels and a size of 256×256, which is obtained by stitching together a 3-channel RGB low-quality color image and a single-channel luminance attention map.

[0068] The encoding attention module comprises eight downsampling layers. During the encoding phase, these eight downsampling convolutional layers use 4×4 convolutional kernels and the Leaky LeRU activation function to extract multi-scale features from the input image. The output layer numbers are 64, 128, 256, 512, 512, 512, 512, and 512, respectively. The brightness attention map is multiplied by the feature map output from the downsampling process, and max-pooling is applied to the brightness attention map to ensure its size matches the size of the corresponding convolutional layer feature map.

[0069] The decoding attention module contains eight upsampling layers. During the decoding stage, the first seven upsampling convolutional layers use 4×4 convolutional kernels and the Leaky LeRU activation function. The final deconvolutional layer uses the Tanh activation function to produce a globally enhanced image with more complete detail preservation. The output layer numbers are 512, 512, 512, 512, 512, 256, 128, 64, and 3, respectively. The detail attention map is concatenated with the feature map output from the upsampling process. Max-pooling is applied to the detail attention map to make its size consistent with the size of the corresponding convolutional layer feature map.

[0070] 2-3: The objective function uses a combination of four loss functions: absolute error loss, mean squared error loss, structural similarity loss, and perceptual loss. The total loss function can be expressed as:

[0071]

[0072] In the formula, L1 represents the absolute error loss, L2 represents the mean square error loss, and L... Structure L represents the structural similarity loss. Perceptual This represents the perceptual loss. λ1, λ2, λ3, and λ4 represent constants, set to 1, 1000, 1, and 1, respectively. eg and I gt These are the enhanced image output by EDMAU and the corresponding Ground Truth, respectively. l represents brightness information, c represents contrast information, s represents structural similarity, i represents pixel coordinates, m represents the total number of pixels, and α, β, and θ are parameters that adjust the importance of the three components l, c, and s. C, W, and H represent the channels, height, and width of the feature map, respectively. Φ(x) is the feature map generated by VGG-16.

[0073] 2-4: The network model uses an optimization algorithm with a learning rate of 0.0002 and is trained for a total of 200 epochs. After 200 epochs of training, the corresponding global augmentation network model training parameters are obtained.

[0074] 2-5: Input the test set of the constructed dataset into the trained global augmentation network model to obtain the global augmentation map I predicted by the model. eg .

[0075] Step 3, the training and prediction of the segmentation network HarDNet-MSEG model, includes the following operations:

[0076] 3-1: HarDNet-MSEG is a segmentation network for intestinal polyps. It takes low-quality images from the training set of the constructed dataset and the corresponding polyp segmentation masks as inputs to EDMAU for training.

[0077] 3-2: The network model uses an optimization algorithm with a learning rate of 0.0001 and is trained for a total of 100 epochs. After 100 epochs of training, the corresponding polyp segmentation network model training parameters are obtained.

[0078] 3-3: Input the test set of the constructed dataset into the trained HarDNet-MSEG network model to obtain the model's predicted precise segmentation mask image of polyp locations. mask .

[0079] Step 4 determines whether a polyp is present based on the mask image. If the mask image has no white areas, it indicates that the image does not contain polyps, and the final global enhancement image I is obtained. eg If the mask image contains white areas, it indicates that the image contains polyps. Then proceed to step 5 to remove the polyps. mask and I eg The final locally enhanced image I is obtained by fusion. el .

[0080] The image fusion in step 5 includes the following operations:

[0081] 5-1: For I mask Extracted I eg The lesion area retains the EDMAU enhancement effect;

[0082] 5-2: For I mask Extracted I eg Gamma correction was used on non-lesion areas to achieve local enhancement of polyp images. el .

[0083] The specific expression for the image fusion strategy is: In the formula, f gamma (I eg ) for I eg The image generated after gamma correction.

[0084] The method in this embodiment performs qualitative and quantitative analysis on the LHI intestinal polyp image dataset constructed in this invention. The method of this invention can simultaneously achieve two different tasks: global detail enhancement and local detail enhancement.

[0085] The EDMAU method of this invention is used to perform global enhancement processing on intestinal endoscopy images, and its enhancement effect is compared with four traditional image enhancement methods: Limit Contrast Adaptive Histogram Equalization (CLAHE), Structure-revealing Low Light Enhancement Method (RRM) based on Rubost Retinex Decomposition Model, Adaptive Gamma Correction Weighted Distribution (AGCWD), and an endoscopy image enhancement method proposed by Wang et al. (EndpIMLE), as well as four deep learning-based image enhancement methods: RetinexNet, KinD, DeepUPE, and Zero-DCE++. Figure 4 This is a detailed comparison of the prediction results of the EDMAU method of this invention and existing methods. Figure 4 The comparison results show that the method of the present invention can successfully remove shadows from dark areas when processing intestinal endoscopy images, while still maintaining richer texture details in the brighter parts of the original image, and the contrast is also improved, making it more consistent with the original colors of the objects represented in the image.

[0086] The method of this invention is used to perform local enhancement processing on intestinal polyp images using a fusion strategy, and then compared with the original input image without enhancement. Figure 5 This compares the changes in image saliency between the unenhanced original input image and the final locally enhanced image obtained using the method of this invention. Figure 5 The comparative results show that after local enhancement using our proposed fusion strategy, the image of intestinal polyps allows for greater focus on the lesion area. Therefore, the image enhancement method proposed in this paper holds promise for assisting doctors in making accurate diagnoses.

[0087] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A method for enhancing intestinal endoscopic images based on image fusion, characterized in that, The operation steps are as follows: Step 1: Create a dataset; Step 2: Input the low-quality images and high-quality reference images from the dataset into the encoder-decoder multi-attention U-Net network to train the deep network model and make predictions to obtain globally augmented images; Multiple attention maps are obtained, which refer to detail attention maps and brightness attention maps. The detail attention map is added to the decoding stage of the U-Net network to highlight detailed features and reduce the loss of some information during image enhancement. The brightness attention map is introduced into the encoding stage of the U-Net network to improve the global image illumination. The encoder-decoder multi-attention U-Net network is an image enhancement network that consists of three parts: input, encoding attention, decoding attention, and output. The low-quality images and high-quality reference images from the training set of the constructed dataset are input into the encoder-decoder multi-attention U-Net network EDMAU for training, resulting in globally enhanced images. ; Step 3: Input the images from the dataset into the segmentation network HarDNet-MSEG to train and predict the deep network model to obtain polyp segmentation mask images; Step 4: Determine if polyps are present. If present, perform image fusion; otherwise, output a globally enhanced image. Step 5: Fuse the global augmented image obtained by the encoder-decoder multi-attention U-Net network and the mask image obtained by HarDNet-MSEG to output the final local augmented image.

2. The method for enhancing intestinal endoscopic images based on image fusion according to claim 1, characterized in that, Step 1, creating the dataset, includes the following specific steps: 1-1: Collect real intestinal images and corresponding polyp segmentation masks from some publicly available open-source medical imaging datasets related to the gastrointestinal tract; 1-2: Adjust the resolution of the collected images to 256×256 pixels; 1-3: Use Adobe Lightroom to adjust real intestinal polyp images to obtain low-quality images and high-quality reference images; 1-4: Invite volunteers to subjectively evaluate the images obtained in 1-3 to obtain the best image output; 1-5: Divide the obtained dataset into training and test sets in a 2:1 ratio.

3. The method for enhancing intestinal endoscopic images based on image fusion according to claim 1, characterized in that, Step 2, the encoder-decoder multi-attention U-Net network model training and prediction, includes the following specific steps: 2-1: Encoder-Decoder Multi-Attention U-Net Network EDMAU is an image augmentation network consisting of three parts: input, encoding attention, decoding attention, and output; multi-attention refers to detail attention maps and brightness attention maps; low-quality images and high-quality reference images from the training set are input into EDMAU for training; 2-2: The objective function uses a combination of four loss functions: absolute error loss, mean squared error loss, structural similarity loss, and perceptual loss. 2-3: The network model uses an optimization algorithm with a learning rate of 0.0002 and is trained for a total of 200 epochs; 2-4: Input the test set into the trained EDMAU network model to obtain the globally enhanced image predicted by the model.

4. The method for enhancing intestinal endoscopic images based on image fusion according to claim 1, characterized in that, The segmentation network HarDNet-MSEG in step 3 includes the following specific operation steps: 3-1: HarDNet-MSEG is a segmentation network for intestinal polyps. It is trained by inputting low-quality images in the training set and the corresponding polyp segmentation masks into EDMAU. 3-2: The network model uses an optimization algorithm with a learning rate of 0.0001 and is trained for a total of 100 epochs; 3-3: Input the test set into the trained HarDNet-MSEG network model to obtain the mask image predicted by the model.

5. The method for enhancing intestinal endoscopic images based on image fusion according to claim 1, characterized in that, The determination of whether a polyp is present in step 4 is based on the mask image predicted by HarDNet-MSEG to distinguish whether the image is to be enhanced globally or further enhanced locally; if there are no polyps in the image, the final global enhancement image is obtained; if there are polyps, step 5 is executed.

6. The method for enhancing intestinal endoscopic images based on image fusion according to claim 1, characterized in that, The image fusion in step 5 includes the following specific steps: 5-1: The EDMAU enhancement effect is preserved in the lesion area of ​​the EDMAU image extracted from the mask image; 5-2: Gamma correction is used on the non-lesion areas of the EDMAU image extracted from the mask image to weaken the background and achieve local enhancement of the polyp image.

Images