Method for optimizing exposure effect of endoscope based on color features

Abstract

The invention provides a method for optimizing the exposure effect of an endoscope based on color features, and the method comprises the steps: embedding a multi-scale color feature pyramid in an encoder-decoder structure to distinguish and represent different exposure regions, and applying different convolution kernels to the color features of different scales, according to the method, the region recognition capability of an encoder-decoder structure is improved, efficient multi-scale attention is introduced, and the calculation overhead is reduced while the feature extraction quality is improved. In order to adaptively restore the exposure level of the image, the method adopts a Retinex algorithm to perform double-branch enhancement on the image. According to the method, on a laparoscope overexposure data set, the PNSR index reaches 21.06; the NIQE index on a real laryngoscope data set reaches 41.79, which is superior to that of other existing methods. The method provides a new thought for the development of a low-quality endoscope image enhancement method.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and image enhancement, and in particular relates to a method for optimizing endoscope exposure effect based on color features. Background Art

[0002] Electronic endoscopes provide cameras and lighting systems to directly observe the tissue morphology and pathological conditions of the human body cavity and organ cavity, and use high-definition display systems to observe the patient's internal conditions in real time. However, in actual clinical applications, due to the complexity of the internal structure of the human lumen, the endoscope probe light source, as the only light source in the cavity, is often affected by the non-uniform reflection source caused by complex structures such as folds, folds, and torsion of the cavity wall, resulting in uneven illumination of the cavity image, overexposure or underexposure, etc., which seriously affects the image quality and poses a challenge to the doctor's visual perception. Overexposure may conceal subtle cases, and underexposure may make important pathological features unclear, reducing the doctor's screening and diagnostic efficiency of the disease. Therefore, developing an efficient image enhancement algorithm to effectively solve the problem of uneven exposure (overexposure and underexposure) of endoscopic images under low illumination conditions and significantly improve image quality is crucial to improving the early and accurate screening and diagnosis of intracavitary lesions.

[0003] Low-light image enhancement has always been a hot topic in the field of image processing. However, due to the difficulty in obtaining medical datasets and the special nature of medical images, there are few studies focusing on medical image enhancement, especially endoscopic image enhancement. Although the enhancement methods of natural scene images have developed rapidly, they are not suitable for complex endoscopic imaging environments and often have problems such as color distortion, loss of details, and abnormal exposure. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a method for optimizing the endoscope exposure effect based on color features, so as to solve the overexposure and underexposure problems of the endoscope image while avoiding color distortion and detail loss.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for optimizing endoscope exposure effect based on color features comprises the following steps:

[0007] Step 1: Unify the size of the original endoscope image to obtain image I low , and generate the corresponding inverted image I l ′ ow ;

[0008] Step 2: Define image I low and the corresponding inverted image I l′ ow For low quality image I l , as the input of the encoder-decoder, the encoder-decoder is used to estimate the overexposed and underexposed illumination components, and then the illumination components are enhanced by the exposure correction module to obtain enhanced overexposed images and underexposed images. Finally, the enhanced overexposed images and underexposed images are compared with the image I low Feature fusion is performed to obtain the final result; wherein a multi-scale color feature pyramid is embedded in the encoder-decoder, and under the guidance of the multi-scale color feature pyramid, learning is performed to divide different exposure areas and estimate corresponding lighting components.

[0009] The present invention embeds a multi-scale color feature pyramid in the encoder-decoder structure to distinguish and represent different exposure areas. By applying different convolution kernels to color features of different scales, the regional recognition ability of the encoder-decoder structure is improved, and efficient multi-scale attention is introduced to improve the quality of feature extraction while reducing computational overhead. Furthermore, in order to adaptively restore the exposure level of the image, the present invention also uses the Retinex algorithm to perform dual-branch enhancement on the image.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. Compared with other methods, the present invention can accurately estimate the illumination components of different exposure areas in the image through the encoder-decoder structure under the guidance of the multi-scale color feature pyramid, use the dual-branch Retinex algorithm to enhance the illumination components, and fuse the enhanced components with the original image, so as to maintain the consistency of image color while optimizing the endoscopic exposure effect, effectively improve the exposure effect of the endoscopic image, and meet the high-fidelity requirements of medical images.

[0012] 2. The method of the present invention has greatly reduced parameters and has a significant lightweight advantage compared with other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The figure is a flowchart of the use of an embodiment of the present invention.

[0014] Figure 2 This is the process of data preprocessing in an embodiment of the present invention.

[0015] Figure 3 4 is a structural diagram of a network model in an embodiment of the present invention.

[0016] Figure 4 2 is a structural diagram of an encoder-decoder in an embodiment of the present invention.

[0017] Figure 5The figure is a comparison of the test results of various methods in the embodiments of the present invention on a real laryngoscope dataset. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0019] Due to the special imaging environment of the endoscope, the image often contains both overexposure and underexposure. However, most existing enhancement methods focus on enhancing the dark areas in the image or the overall brightness of the image, which often destroys the normally exposed area or aggravates the exposure of the overexposed area, accompanied by color distortion. In order to solve this problem, the present invention proposes a method for optimizing the exposure effect of the endoscope based on color features. The method uses color features as prior information to guide the network model to identify and divide overexposed and underexposed areas, and uses the Retinex algorithm to adaptively restore the exposure level of the image, thereby improving the visual quality of the endoscopic image while avoiding color distortion.

[0020] The method of optimizing the endoscope exposure effect based on color characteristics of the present invention has the following main processes: Figure 1 As shown, including:

[0021] Step 1: Data preprocessing.

[0022] In this step, the original endoscope image needs to be resized to obtain image I low , a typical size of 512×512, and then calculate its RGB inversion map, which is the same as image I low The corresponding inverted image I l ′ ow , for subsequent image enhancement tasks, the process can refer to Figure 2 .

[0023] Step 2: Construction of the overall network model.

[0024] refer to Figure 3 , the preprocessed image I low and its corresponding inverted image I l ′ ow , will be used as the input of the network model to process the underexposed and overexposed areas respectively. For the convenience of description, the present invention will image I low and the corresponding inverted image I l ′ ow Defined as low quality image I lThe network model of the present invention adopts an encoder-decoder architecture, uses an encoder-decoder to estimate the overexposed and underexposed lighting components, and then enhances the lighting components through an exposure correction module to obtain enhanced overexposed images and underexposed images. Finally, the enhanced overexposed images and underexposed images are compared with image I low Perform feature fusion to obtain the final result I output The overall process can be formulated as:

[0025] L under =ED(I low )

[0026] L over =ED(I′ low )

[0027]

[0028]

[0029]

[0030] Among them, ED is the encoder-decoder operation, L under and L over is the underexposed and overexposed lighting component estimated by the encoder-decoder, Concat is the feature fusion operation, Enhance is the enhancement network in the exposure correction module, and It is the enhanced underexposed and overexposed image, and EFM is the exposure fusion module.

[0031] The specific implementation of the network model of the present invention is as follows:

[0032] Step 2.1: Encoder-Decoder.

[0033] Based on the features of endoscopic images, the present invention adopts an encoder-decoder structure to extract image features and divide the exposure area to guide subsequent enhancement operations. Figure 4 The encoder-decoder of the present invention mainly includes 8 components: input layer, multi-scale color feature pyramid, convolution block, maximum pooling, efficient multi-scale attention module, adaptive convolution kernel prediction module, bilinear interpolation upsampling and output layer.

[0034] The encoder-decoder structure uses an N-layer architecture. Figure 4The embodiment shown is a four-layer architecture, which functionally includes an input layer, an encoding layer, a decoding layer, and an output layer, wherein the encoding layer and the decoding layer are both N layers, and a jump connection is used between each encoding layer and the decoding layer. The multi-scale color feature pyramid is N layers, each embedded in a decoding layer. The encoder-decoder uses the input layer to process the input image (I low and I l ′ ow ) performs shallow feature extraction and constructs a multi-scale feature pyramid to count the color distribution of different regions. The encoder part extracts input features, and the decoder upsamples the features extracted by the encoder part and enhances them under the guidance of color features. Finally, the output layer outputs overexposed and underexposed lighting components.

[0035] Specifically, the input layer receives an input image of size 512×512×3, and then uses the multi-scale color feature pyramid to calculate its color histograms at different scales to construct a four-layer color feature pyramid.

[0036] The efficient multi-scale attention module of the present invention divides the feature map output by the previous layer into multiple sub-features along the channel dimension, and uses two convolution branches of 1×1 and 3×3 in parallel without channel dimensionality reduction to capture the information between channels and multi-scale spatial features respectively, and generates a spatial attention map through matrix dot product. This feature grouping and cross-space learning method enables the module to effectively capture short-term and long-term pairwise dependencies at the pixel level with limited computing resources, thereby enhancing the feature representation capability.

[0037] The adaptive convolution kernel prediction module of the present invention realizes convolution kernel generation and mask generation respectively through two branches, namely, a convolution kernel generation module and a mask generation module. The convolution kernel generation module receives a feature map input by a previous layer of convolution blocks, and learns to generate multiple convolution kernels of different sizes; the mask generation module receives a color feature map generated by a layer of the multi-scale color feature pyramid, and generates a multi-value mask under the guidance of the color feature map to divide different areas. By assigning different convolution kernels to different areas, the multi-scale color feature pyramid guides the network to distinguish different exposure areas.

[0038] Each encoding layer of the present invention sequentially comprises a maximum pooling, a convolution block and an efficient multi-scale attention module; each decoding layer sequentially comprises a bilinear interpolation upsampling, an adaptive convolution kernel prediction module and an efficient multi-scale attention module. A layer of multi-scale color feature pyramid is embedded between the convolution block and the adaptive convolution kernel prediction module; finally, the output layer outputs a feature map through a convolution block.

[0039] The multi-scale color feature pyramid is an important module of the present invention, which is embedded in the encoder-decoder and guides the learning to divide different exposure areas and estimate the corresponding lighting components. Specifically, the present invention uses color information to help the model learn the illumination level of the area. The input of the multi-scale color feature pyramid is a low-quality image I l Then, according to the size of the scale, the color histogram features of different scales are calculated as the output of each layer of the pyramid. The output of each layer is connected to the adaptive convolution kernel prediction module in each layer of the decoder.

[0040] The steps of calculating the multi-scale color feature pyramid of the present invention are as follows: a low-quality image I with a size of H×W is l Divide into M patches of size [H*C]·[W / C], where [·] means taking the integer closest to the calculation result. Then, calculate the color histogram of the patch size C×C and define it as the color feature, where the color features of different scales are each layer of the feature pyramid. The adaptive convolution kernel prediction module in each decoding layer applies different convolution kernels to the color features of different scales according to the output of each layer of the pyramid. Finally, a bilateral grid μ is constructed by pixel histogram voting. The bilateral grid μ is a data structure that can combine the spatial dimension and color dimension of the image to form a four-dimensional representation. According to the bilateral grid μ, the present invention uses the four-dimensional color histogram feature F of x, y, n, and z. C To represent the distribution of scale C. C The calculation formula is:

[0041]

[0042] Where x, y, n represent the horizontal index, vertical index and channel index of the image, z is the index of the histogram, i, j represent the horizontal and vertical index of the pixel in the patch, respectively. The value of is the index of the pixel (x, y) in the patch to which it belongs.

[0043] The F obtained in this step C Represents the lighting and color distribution characteristics of different areas. The model can distinguish overexposed and underexposed areas by analyzing these characteristics, and also help reduce color inconsistency problems caused by local processing.

[0044] 2.2: Exposure correction module.

[0045] In order to adaptively correct the exposure of the endoscopic image, the present invention constructs an exposure correction module based on the Retinex theory. The Retinex theory divides the low-light observation image into a reflectivity component and an illumination component, and the calculation formula is:

[0046] I=R·L

[0047] Wherein, I is the observed image, i.e., the original low-quality image input, R and L are the reflectance component and illumination component corresponding to the observed image, respectively, and · is the Hadamard product. The present invention uses the reflectance image as the enhanced image, and transforms the above formula to obtain the calculation formula:

[0048] R=I·L -1

[0049] In order to simultaneously process overexposed and underexposed areas in a single image, the present invention regards the overexposed areas in the low-light image as underexposed areas in its inverted image. Therefore, the input image and its inverted image are passed through the encoder-decoder structure to generate the corresponding underexposed feature map and overexposed feature map, which are the illumination map components, and then the corresponding reflectivity components are calculated respectively to implement the exposure correction operation and obtain the enhanced image. The specific calculation formula is:

[0050]

[0051]

[0052]

[0053]

[0054] Among them, R under is the reflectance component of the underexposed illumination component, i.e. the enhanced underexposed image R over is the reflectance component of the overexposed illumination component, which is then inverted to obtain the enhanced overexposed image and The inverse of the two illumination components estimated by the encoder-decoder.

[0055] 2.3: Exposure fusion module.

[0056] In order to reduce the impact of uneven exposure on image enhancement, the present invention uses an exposure fusion module to fuse the enhanced underexposed image, overexposed image and original image. It consists of a four-layer structure. The first and third layers contain a convolution block and a CBAM, and the second and fourth layers contain a convolution block, a CBAM and a non-local block. Among them, CBAM acts on the convolution feature map. By sequentially executing the channel attention and spatial attention mechanisms, the network can focus on the exposure area in the input feature map. The non-local block can capture long-distance pixel dependencies by calculating the correlation between position features. The fused exposure module combines the convolution block, CBAM and non-local block, and uses the three-channel weight feature map to fuse the three inputs to generate the final enhancement result. It can be expressed as follows:

[0057]

[0058] Among them, f represents the exposure fusion module, which realizes the fusion operation of the input image, the overexposed image and the underexposed image.

[0059] Step 3: Loss function setting.

[0060] In order to correct the color distortion of the unevenly exposed area while preserving the color information of the well-exposed area, the present invention uses the cosine similarity loss L cos The calculation formula is:

[0061]

[0062] Where A·B represents the dot product of vector A and vector B, and ||A|| and ||B|| are the L2 norms of vectors A and B.

[0063] In order to measure the reconstruction loss, the present invention uses the mean square error loss L mse The calculation formula is:

[0064]

[0065] Where N is the number of samples, y i is the true value, is the predicted value.

[0066] In order to allow the network to better estimate the illumination component of the image, the present invention uses a local smoothness loss L tv The calculation formula is:

[0067]

[0068] Where S is the predicted lighting component, p is the pixel, c is the channel, and Is the pixel p in the water

[0069] The smoothing weights in the horizontal and vertical directions, and are the partial derivatives in the horizontal and vertical directions in the image space, and are the rates of change of the illumination value of pixel p in the horizontal and vertical directions, respectively.

[0070] The overall loss function is:

[0071] L total =α1L cos +α2L mse +α3L tv1 +α4L tv2

[0072] Among them, L tv1 and L tv2 are the local smoothing losses corresponding to the overexposed and underexposed illumination components, respectively. α1, α2, α3 and α4 are the corresponding weights, which are set to α1 = 0.5, α2 = 1.0, α3 = α4 = 0.01, respectively.

[0073] Step 4: Output processing.

[0074] The trained model weights are used for testing and the test images are enhanced. Different evaluation indicators are used depending on whether the test data contains GroundTruth images.

[0075] Continue to refer Figure 1 In a specific embodiment of the present invention, the input original image data is in jpg, jpeg or png format, which can be taken from the laparoscopic dataset and the laryngoscopic dataset for different experiments, wherein the laparoscopic dataset is further divided into the laparoscopic over-exposure dataset and the laparoscopic under-exposure dataset. Then, the size of the original image is adjusted to 512×512, and its RGB inverted image is calculated, such as Figure 2 ; Use the original image and the reversed image as model input and start training the model, such as Figure 3 ; Finally, use the trained model weights for testing and evaluation.

[0076] The comparison results of the method in this embodiment and other methods on RTX4090 GPU are shown in Table 1 and Figure 5 .

[0077] Table 1 is a comparison of the performance indicators of various methods in the embodiments of the present invention.

[0078]

[0079] As can be seen from Table 1, the proposed method has the best PNSR index on the laparoscopic overexposure dataset, reaching 21.06, and the best PIQE index on the laryngoscopy dataset, reaching 41.79, which is better than other existing methods. Compared with other methods, the number of parameters of the proposed method is only 0.646M; Figure 5 It can be seen that the method of the present invention can simultaneously improve the overexposure and underexposure conditions in the endoscope, and can maintain the color consistency with the original image, providing a new idea for the development of low-quality endoscopic image enhancement methods.

Claims

1. A method for optimizing endoscope exposure effect based on color features, characterized in that: The steps include: Step 1: Unify the size of the original endoscope image to obtain image I low , and generate the corresponding inverted image I l ′ ow ; Step 2: Define image I low and the corresponding inverted image I l ′ ow For low quality image I l , as the input of the encoder-decoder, the encoder-decoder is used to estimate the overexposed and underexposed illumination components, and then the illumination components are enhanced by the exposure correction module to obtain enhanced overexposed images and underexposed images. Finally, the enhanced overexposed images and underexposed images are compared with the image I low Feature fusion is performed to obtain the final result; wherein a multi-scale color feature pyramid is embedded in the encoder-decoder, and under the guidance of the multi-scale color feature pyramid, learning is performed to divide different exposure areas and estimate corresponding lighting components.

2. The method for optimizing endoscope exposure effect based on color features according to claim 1, characterized in that: The encoder-decoder uses an N-layer architecture, including an input layer, an encoding layer, a decoding layer and an output layer, wherein the encoding layer and the decoding layer are both N layers, and a jump connection is used between each encoding layer and the decoding layer. The multi-scale color feature pyramid is N layers, each embedded in a decoding layer; the input layer receives an input image, and then uses the multi-scale color feature pyramid to calculate the color histogram of different scales of the input image, and the output layer outputs a feature map through a convolution block.

3. The method for optimizing endoscope exposure effect based on color features according to claim 2, characterized in that: One of the encoding layers sequentially comprises a maximum pooling, a convolution block and an efficient multi-scale attention module; one of the decoding layers sequentially comprises a bilinear interpolation upsampling, an adaptive convolution kernel prediction module and an efficient multi-scale attention module, and a layer of the multi-scale color feature pyramid is embedded between the convolution block and the adaptive convolution kernel prediction module.

4. The method for optimizing endoscope exposure effect based on color features according to claim 3, characterized in that: The efficient multi-scale attention module divides the feature map output by the previous layer into multiple sub-features along the channel dimension, uses two convolution branches in parallel to capture the information between channels and multi-scale spatial features respectively, and generates a spatial attention map through matrix dot product.

5. The method for optimizing endoscope exposure effect based on color features according to claim 3, characterized in that: The adaptive convolution kernel prediction module realizes convolution kernel generation and mask generation respectively through two branches, namely, a convolution kernel generation module and a mask generation module. The convolution kernel generation module receives the feature map input by the previous layer of convolution block, and learns to generate multiple convolution kernels of different sizes; the mask generation module receives a layer of color feature map generated by the multi-scale color feature pyramid, and generates a multi-value mask under the guidance of the color feature map to divide different areas. By assigning different convolution kernels to different areas, the multi-scale color feature pyramid guides the network to distinguish different exposure areas.

6. The method for optimizing endoscope exposure effect based on color features according to claim 3, characterized in that: The calculation steps of the multi-scale color feature pyramid are as follows: a low-quality image I with a size of H×W is l The image is divided into M patches of size [H / C]·[W / C], where [·] means taking the integer closest to the calculated result; then, the color histogram of the patch of size C×C is calculated and defined as the color feature; finally, a bilateral grid μ is constructed by pixel histogram voting, and the four-dimensional color histogram feature F is used. C represents the distribution of scale C; F C The calculation formula is: Where x, y, n represent the horizontal index, vertical index and channel index of the image, z is the index of the histogram, and i, j represent the horizontal and vertical index of the pixels in the patch, respectively. The value of is the index of the pixel (x, y) in the patch to which it belongs.

7. The method for optimizing endoscope exposure effect based on color features according to claim 1, characterized in that: The N=4, the encoder-decoder, uses the input layer to process the input image (I low and I l ′ ow ) performs shallow feature extraction and constructs a multi-scale feature pyramid to count the color distribution of different regions. The encoder part extracts input features, and the decoder upsamples the features extracted by the encoder part and enhances them under the guidance of color features. Finally, the output layer outputs overexposed and underexposed lighting components.

8. The method for optimizing endoscope exposure effect based on color features according to claim 1, characterized in that: The exposure correction module calculates the reflectivity component corresponding to the illumination component based on the Retinex theory to obtain an enhanced image. The calculation process is as follows: Among them, Runde r is the reflectance component of the underexposed illumination component, i.e. the enhanced underexposed image Rover is the reflectivity component of the overexposed illumination component, and its inversion is the enhanced overexposed image. and The inverse of the two illumination components estimated for the encoder-decoder.

9. The method for optimizing endoscope exposure effect based on color features according to claim 8, characterized in that: The exposure correction module divides the observed image into a reflectance component and an illumination component, and uses the reflectance image as the enhanced image, which is expressed as: RI·L-1 Where I is the observed image, R and L are the reflectance component and illumination component corresponding to the observed image, respectively, and · is the Hadamard product.

10. The method for optimizing endoscope exposure effect based on color features according to claim 1, 8 or 9, characterized in that: The exposure fusion module consists of a four-layer structure. The first and third layers contain a convolutional block and a CBAM, and the second and fourth layers contain a convolutional block, a CBAM, and a non-local block. The CBAM acts on the convolutional feature map and sequentially executes the channel attention and spatial attention mechanisms to enable the network to focus on the exposure area in the input feature map. The non-local block captures long-distance pixel dependencies by calculating the correlation between position features. The exposure fusion module combines the convolutional block, CBAM, and non-local block, and uses the three-channel weight feature map to fuse the three inputs to generate the final result, which is expressed as follows: Among them, f represents the exposure fusion module, which realizes the fusion operation of the input image, the overexposed image and the underexposed image.

Images