An endoscope image enhancement method against exposure inconsistency

Abstract

The application discloses an endoscope image enhancement method for exposure inconsistency, relates to the technical field of computer vision and image enhancement, and comprises the following steps: constructing an image enhancement overall network model, setting a denoising network and a color correction module in the reverse process of a T-shaped diffusion model, adding additive frequency domain attention to the denoising network, and dynamically inserting noise time information into the denoising network and the color correction module through a noise embedding module; setting a loss function; training the image enhancement overall network model; and using the trained image enhancement overall network model to perform actual endoscope image enhancement. The application reduces the calculation cost, eliminates detail loss and color distortion, and improves the recovery precision of the endoscope image by combining the T-shaped diffusion model, the denoising network, the color correction module, the additive frequency domain attention and the noise embedding module.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image enhancement technology, and in particular to a method for enhancing endoscopic images with inconsistent exposure. Background Technology

[0002] Endoscopic imaging has long been plagued by a major challenge: uneven image exposure caused by inconsistent illumination. This is primarily due to its low-light imaging environment. Specifically, endoscopes need to penetrate deep into human cavities, which are narrow, complex, and subject to a single light source that constantly changes distance from the target. This often results in undesirable images where some areas are overexposed or underexposed, severely impairing image quality and negatively impacting diagnostic efficiency and the performance of computer-aided detection and diagnosis technologies. Image enhancement in low-light environments has been a hot topic in computer vision. While techniques for enhancing natural images are becoming increasingly mature, they are not suitable for endoscopic images due to the unique imaging environment. Although researchers have proposed targeted methods for enhancing endoscopic images, research in this field is still in its early stages, facing challenges such as detail loss, color distortion, high computational costs, and low image restoration accuracy.

[0003] In the prior art, Chinese patent CN114387191A discloses an endoscopic image enhancement method and endoscopic device, comprising: acquiring an original endoscopic image and extracting its edge information to obtain an edge image; enhancing the brightness of the low-light areas of the original endoscopic image to obtain a first endoscopic image; flipping the first endoscopic image and performing color equalization processing to obtain a second endoscopic image; performing Gaussian blur processing on the second endoscopic image to obtain a third endoscopic image; performing color equalization processing on the third endoscopic image to obtain a fourth endoscopic image; fusing the fourth endoscopic image with the original endoscopic image to obtain a fifth endoscopic image; and sharpening the edge image and the fifth endoscopic image to obtain an enhanced endoscopic image.

[0004] However, the aforementioned existing technologies have not studied or solved the above problems, have high computational costs, suffer from severe loss of detail and color distortion, and have low accuracy in restoring endoscopic images. Summary of the Invention

[0005] This application provides a method for enhancing endoscopic images with inconsistent exposure, which solves the problems of high computational cost, severe loss of detail and color distortion, and low accuracy of endoscopic image restoration in existing endoscopic image enhancement techniques.

[0006] On the one hand, this application provides a method for enhancing endoscopic images with inconsistent exposure, comprising the following steps: Step 1: Construct an overall image enhancement network model. The overall image enhancement network model is based on the T-shaped diffusion model. The reverse process of the T-shaped diffusion model includes a denoising network and a color correction module. Additive frequency domain attention is added to each layer of the decoder of the denoising network. The denoising network and the color correction module dynamically insert noise time information through a noise embedding module.

[0007] Step 2: Set the loss function for the overall image enhancement network model.

[0008] Step 3: Construct a training set using endoscopic labeled images and corresponding low-quality images, and train the overall image enhancement network model to obtain the trained overall image enhancement network model.

[0009] Step four: Use the trained image enhancement network model to perform actual endoscopic image enhancement.

[0010] In one possible implementation, step one, the construction process of the T-shaped diffusion model includes: Based on the denoising diffusion probability model, the noise sampling strategy is adjusted to T-shape to obtain the T-shaped diffusion model, which is used to learn the exposure distribution of endoscopic images in Gaussian space.

[0011] In one possible implementation, step one, the construction process of the denoising network includes: A multi-layer encoder-decoder architecture is constructed to obtain the denoising network, which is used to learn the noise distribution of the endoscopic image during the reverse process of the T-shaped diffusion model in order to recover a high-quality image.

[0012] In one possible implementation, step one, the construction process of the additive frequency domain attention includes: The additive frequency domain attention is obtained by combining Fourier transform with additive attention. The Fourier transform is used to capture the global dependencies of the endoscopic image, and the additive attention is used to enhance the local detail features of the endoscopic image.

[0013] In one possible implementation, step one, the construction process of the color correction module includes: The color correction module is constructed sequentially according to the structure of normalization, convolution, scale transformation, channel attention, scale transformation, convolution, and scale transformation.

[0014] In one possible implementation, step one, the construction process of the noise embedding module includes: The noise embedding module is constructed based on the KAN network. The noise embedding module is used to dynamically insert noise time information into the denoising network and the color correction module.

[0015] In one possible implementation, in step two, the loss function of the overall image enhancement network model adopts... function.

[0016] In one possible implementation, in step three, the trained image augmentation network model is tested using test data to evaluate its performance.

[0017] In one possible implementation, different objective image evaluation metrics are used for cases where endoscopic labeled images are present and cases where endoscopic labeled images are not present in the test data.

[0018] The endoscopic image enhancement method for inconsistent exposure described in this application has the following advantages: By combining a T-shaped diffusion model, a denoising network, a color correction module, additive frequency domain attention, and a noise embedding module, the following approach is employed: The T-shaped diffusion model utilizes the Markov principle to progressively add noise to the endoscopic image during the forward pass, mapping the endoscopic image features to Gaussian space. During the reverse pass, the denoising network progressively recovers the exposure distribution of the endoscopic image from Gaussian space. The color correction module then performs color correction on the recovered endoscopic image. Additive frequency domain attention balances global dependencies and local details in the image with relatively low computational complexity. The noise embedding module dynamically inserts noise temporal information into the denoising network and color correction module. These combined approaches reduce computational costs, eliminate detail loss and color distortion, and improve the accuracy of endoscopic image recovery. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic flowchart illustrating an endoscopic image enhancement method for inconsistent exposure provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the T-type diffusion model provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the denoising network provided in the embodiments of this application; Figure 4 This is a schematic diagram of the additive frequency domain attention structure provided in an embodiment of this application; Figure 5This is a schematic diagram of the structure of the color correction module provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the noise embedding module provided in an embodiment of this application; Figure 7 The performance metrics of the endoscopic image enhancement method provided in this application embodiment are compared with those of other existing methods on publicly available laparoscopic datasets and gastrointestinal endoscopy datasets. Figure 8 Performance metrics comparison between the endoscopic image enhancement method provided in this application embodiment and other existing methods on a self-built nasopharyngeal endoscopy dataset; Figure 9 Visual comparison results of the endoscopic image enhancement method provided in this application embodiment with other existing methods on a publicly available laparoscopic dataset; Figure 10 Visual comparison results of the endoscopic image enhancement method provided in this application embodiment with other existing methods on a publicly available digestive tract endoscopy dataset; Figure 11 Visual comparison results of the endoscopic image enhancement method provided in this application embodiment with other existing methods on a self-built nasopharyngeal endoscopy dataset. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] like Figure 1 As shown in the figure, this application provides a method for enhancing endoscopic images with inconsistent exposure, including the following steps: Step 1: Construct an overall image enhancement network model. The overall image enhancement network model is based on the T-shaped diffusion model. The reverse process of the T-shaped diffusion model includes a denoising network and a color correction module. Additive frequency domain attention is added to each layer of the decoder of the denoising network. The denoising network and the color correction module dynamically insert noise time information through a noise embedding module.

[0023] Step 2: Set the loss function for the overall image enhancement network model.

[0024] Step 3: Construct a training set using endoscopic labeled images and corresponding low-quality images, and train the overall image enhancement network model to obtain the trained overall image enhancement network model.

[0025] Step four: Use the trained image enhancement network model to perform actual endoscopic image enhancement.

[0026] Specifically, in this embodiment, the input to the overall image enhancement network model is the endoscopic annotation image (Ground Truth image). and the corresponding low-quality image In the forward phase of the T-type diffusion model, a maximum time step is set. Using variance scheduling By controlling the time step, the model will As the starting point of diffusion, continuously moving towards Add Gaussian noise Until the image is completely corrupted into a noisy image. In the reverse phase of the T-shaped diffusion model, the denoising network will... and The stitched result is used as input to progressively predict noise in the image. The color correction module is used to adjust the colors of the denoised image from the current step, and finally the reconstructed image is output. The entire process can be formalized as follows:

[0027]

[0028] in, This represents the forward process of the T-shaped diffusion model. This is the reverse process of the T-type diffusion model. For color correction module, This is for splicing operations.

[0029] For example, in step one, the construction process of the T-shaped diffusion model includes: Based on the denoising diffusion probability model, the noise sampling strategy is adjusted to T-shape to obtain the T-shaped diffusion model, which is used to learn the exposure distribution of endoscopic images in Gaussian space.

[0030] like Figure 2 The diagram shows the structure of the T-type diffusion model, where Forward Diffusion represents forward diffusion, Reverse Denoising represents reverse diffusion, and Concatenation represents the concatenation operation.

[0031] Specifically, in this embodiment, the T-shaped diffusion model performs diffusion at a non-constant resolution. During forward diffusion to... At this point, the image resolution is reduced to half, and the forward process continues. The forward process can be parameterized as follows:

[0032] in, , Indicates time step as Image resolution at that time, Indicates a Gaussian distribution. This indicates that downsampling adjustments are made based on changes in resolution. This is the variance term.

[0033] In reverse diffusion to Then, the image resolution is restored, followed by the reverse process. The reverse process can be parameterized as follows:

[0034] in, , For noise reduction networks, This is the variance term.

[0035] For example, in step one, the construction process of the denoising network includes: A multi-layer encoder-decoder architecture is constructed to obtain the denoising network, which is used to learn the noise distribution of the endoscopic image during the reverse process of the T-shaped diffusion model in order to recover a high-quality image.

[0036] like Figure 3 The diagram shows the structure of a denoising network, where Embedding represents the operation of embedding noise temporal information.

[0037] Specifically, in this embodiment, the denoising network adopts a three-layer encoder-decoder architecture, and noise temporal information is introduced in each encoder-decoder layer through a noise embedding module. The construction of the noise embedding module is described in the following process.

[0038] In a denoising network, firstly, the input image is processed through a... The convolution expands the number of channels to 64. After entering the encoder, it uses multi-level residual blocks and convolutional blocks for feature extraction. Each stage expands the number of channels through downsampling, extracting image features at different scales. A single-head self-attention mechanism is used to enhance features at the bottleneck layer of the network. Subsequently, the decoder integrates the features extracted by the encoder. Its composition is similar to the encoding part, consisting of multi-level residual blocks and convolutional blocks. Each stage gradually recovers the feature map through upsampling and uses skip connections to fuse encoder information at each scale. To further enhance decoding capabilities, additive frequency domain attention is added to each decoder layer to preserve global dependencies and local detail features. Finally, the feature map is processed by a... The convolution restores the number of channels to Output the results.

[0039] For example, in step one, the construction process of the additive frequency domain attention includes: The additive frequency domain attention is obtained by combining Fourier transform with additive attention. The Fourier transform is used to capture the global dependencies of the endoscopic image, and the additive attention is used to enhance the local detail features of the endoscopic image.

[0040] like Figure 4 The diagram shows the structure of additive frequency domain attention.

[0041] Specifically, to address the global and local degradation issues in images generated by backdiffusion, this embodiment combines Fourier transform with additive attention, proposing an additive frequency domain attention mechanism. The additive frequency domain attention receives the output of the convolutional block. First, go through one layer 1 1 convolution and one layer of 3 3. Depthwise separable convolution generates queries ( Query, ),key( Key ) and value ( Value ), and then and Divided into Local blocks of varying sizes are obtained By using a two-dimensional Fast Fourier Transform (FFT), and Mapping to the frequency domain, the correlation between the two is calculated to capture global dependencies. The formula for calculating the frequency domain attention weights is:

[0042]

[0043] in, and For the frequency domain representation of the query and key, For two-dimensional fast Fourier transform, For frequency domain attention weights, To take the real part of a complex number, This indicates element-wise multiplication. The frequency domain representation of the key is the conjugate transpose. Then, the inverse Fourier transform (IFFT) is used to map the frequency domain weights back to the spatial domain and normalize them to obtain the frequency domain attention features. The calculation formula is:

[0044] To further enhance the local feature representation, this invention initializes a learnable additive weight matrix. Used for values Perform channel weighting. First, Broadcast to The additive attention weights are obtained by summing all spatial locations along the channel dimension. The calculation formula is:

[0045] in, For the channel, Indicates the channel scaling factor. (For...) After normalizing the spatial dimensions, it is broadcast to all channels, along with the value. Additive attention features can be obtained by multiplying element by element. The calculation formula is:

[0046] Ultimately, additive attention features will be used. Frequency domain attention features Added through residual connections, and then through a layer of 1 1. Convolution adjusts the channel dimensions to obtain the output. The calculation formula is:

[0047] in, This represents the output of the convolution block.

[0048] For example, in step one, the construction process of the color correction module includes: The color correction module is constructed sequentially according to the structure of normalization, convolution, scale transformation, channel attention, scale transformation, convolution, and scale transformation.

[0049] like Figure 5 The diagram shown is a structural schematic of the color correction module.

[0050] Specifically, to address the color deviation issue in images generated during the later stages of diffusion, this embodiment incorporates a color correction module. The color correction module receives the image generated during the reverse process. As input, after introducing noisy temporal information, the data is normalized. Then, the color channels are adjusted through convolution and scaling transformation. Channel attention is further refined by combining channel attention and scaling transformation. Finally, after convolution and scaling transformation, the color-corrected image is output. .

[0051] For example, in step one, the construction process of the noise embedding module includes: The noise embedding module is constructed based on the KAN network. The noise embedding module is used to dynamically insert noise time information into the denoising network and the color correction module.

[0052] like Figure 6 The diagram shows the structure of the noise embedding module, where KANLinear represents the KANLinear network layer and Swish represents the Swish activation function.

[0053] Specifically, to further improve the model's recovery accuracy, this invention designs a noise embedding module based on the Kolmogorov-Arnold Network (KAN network), dynamically introducing noise temporal information into the denoising network and color correction module. For the input noise... First, based on the current feature dimensions Positional encoding is then applied to this model, introducing relative positional information about the noise level. The formula for calculating the positional encoding is:

[0054] in, Subsequently, a KANLinear layer is used to increase the dimensionality of the encoded noise features, and the Swish activation function is used to perform a nonlinear transformation on the high-dimensional noise features. Finally, another KANLinear layer is used to map the noise features back to their original dimension. 3D space serves as the temporal information for noise that can be dynamically inserted into denoising networks and color correction modules. The calculation formula is:

[0055] The formula for calculating Swish is as follows:

[0056] in, For Swish input, for function, This is the output of Swish.

[0057] For example, in step two, the loss function of the overall image enhancement network model adopts... function.

[0058] Specifically, in this embodiment, using The function is used to optimize the loss between the noise predicted by the model during the backward process and the Gaussian noise added during the forward process. The formula for calculating the function is:

[0059] in, It is a predicted value. Compared with the true value difference.

[0060] For example, in step three, the trained image enhancement network model is tested using test data to evaluate its performance.

[0061] For example, different objective image evaluation metrics are used for cases where endoscopic labeled images are present and cases where endoscopic labeled images are not present in the test data.

[0062] Specifically, in this embodiment, when there are endoscopic labeled images in the test data, one or more of the PSNR evaluation index, SSIM evaluation index, and LPIPS evaluation index are used as objective image evaluation indexes; when there are no endoscopic labeled images in the test data, one or more of the BRISUQE evaluation index, PIQE evaluation index, and NIQE evaluation index are used as objective image evaluation indexes.

[0063] In one possible embodiment, the endoscopic image enhancement method of this application is compared with other existing methods in terms of performance metrics and visualization.

[0064] like Figure 7 The table shows the performance metrics of the endoscopic image enhancement method of this application (ours represents the endoscopic image enhancement method of this application) compared with other existing methods (including ZeroDEC image enhancement method, EnlightenGAN image enhancement method, LLCaps image enhancement method, LCDPNet image enhancement method, PyDiff image enhancement method, EndoUIC image enhancement method, Retinexformer image enhancement method, LightTDiff image enhancement method, and HVI-CIDNet image enhancement method) on publicly available laparoscopic datasets and gastrointestinal endoscopy datasets (including gastrointestinal endoscopy overexposure datasets and gastrointestinal endoscopy underexposure datasets); PSNR, SSIM, and LPIPS evaluation metrics are used.

[0065] like Figure 8 The figure shows the performance comparison results of the endoscopic image enhancement method provided in this application embodiment and other existing methods on a self-built nasopharyngeal endoscopy dataset; the BRISUQE evaluation index, PIQE evaluation index, and NIQE evaluation index are used.

[0066] Figure 9The visualization comparison results of the endoscopic image enhancement method provided in the embodiments of this application with other existing methods on a publicly available laparoscopic dataset; wherein, the scale of the heatmap from blue to red indicates that the error with the Ground Truth image is getting larger and larger.

[0067] Figure 10 The visualization comparison results of the endoscopic image enhancement method provided in the embodiments of this application with other existing methods on publicly available gastrointestinal endoscopy datasets (including gastrointestinal endoscopy overexposure datasets and gastrointestinal endoscopy underexposure datasets); wherein, the scale of the heatmap from blue to red indicates that the error with the Ground Truth image is getting larger and larger.

[0068] Figure 11 Visual comparison results of the endoscopic image enhancement method provided in this application embodiment with other existing methods on a self-built nasopharyngeal endoscopy dataset.

[0069] Depend on Figures 7 to 11 As can be seen, the endoscopic image enhancement method of this application achieves a PSNR of 33.27 on the publicly available laparoscopic dataset, 29.03 and 29.09 on the overexposed and underexposed digestive tract endoscopy datasets, respectively, and a BRISUQE score of 39.17 on the self-built nasopharyngeal endoscopy dataset (a real nasopharyngeal endoscopy dataset), all of which are superior to other existing methods. The endoscopic image enhancement method of this application provides a new approach for the development of endoscopic image exposure correction methods.

[0070] This application's embodiments combine a T-shaped diffusion model, a denoising network, a color correction module, additive frequency domain attention, and a noise embedding module. The T-shaped diffusion model utilizes the Markov principle to progressively add noise to the endoscopic image during the forward pass, mapping the endoscopic image features to Gaussian space. During the reverse pass, the denoising network progressively recovers the exposure distribution of the endoscopic image from Gaussian space. The color correction module then performs color correction on the recovered endoscopic image. Additive frequency domain attention balances global dependencies and local details in the image with relatively low computational complexity. The noise embedding module dynamically inserts noise temporal information into the denoising network and color correction module. These combined approaches reduce computational costs, eliminate detail loss and color distortion, and improve the accuracy of endoscopic image recovery.

[0071] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0072] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for enhancing endoscopic images with inconsistent exposure, characterized in that, Includes the following steps: Step 1: Construct an overall image enhancement network model. The overall image enhancement network model is based on the T-shaped diffusion model. The reverse process of the T-shaped diffusion model includes a denoising network and a color correction module. Additive frequency domain attention is added to each layer of the decoder of the denoising network. The denoising network and the color correction module dynamically insert noise time information through a noise embedding module. Step 2: Set the loss function for the overall image enhancement network model; Step 3: Construct a training set using endoscopic labeled images and corresponding low-quality images, and train the overall image enhancement network model to obtain the trained overall image enhancement network model; Step four: Use the trained image enhancement network model to perform actual endoscopic image enhancement; In step one, the construction process of the additive frequency domain attention includes: The additive frequency domain attention is obtained by combining Fourier transform with additive attention. The Fourier transform is used to capture the global dependencies of the endoscopic image, and the additive attention is used to enhance the local detail features of the endoscopic image. In step one, the construction process of the noise embedding module includes: The noise embedding module is constructed based on the KAN network. The noise embedding module is used to dynamically insert noise temporal information into the denoising network and the color correction module. Step one, the construction process of the T-shaped diffusion model includes: Based on the denoising diffusion probability model, the noise sampling strategy is adjusted to T-shape to obtain the T-shaped diffusion model, which is used to learn the exposure distribution of endoscopic images in Gaussian space. The decoder integrates the features extracted by the encoder and consists of multi-level residual blocks and convolutional blocks; additive frequency domain attention is added in each layer of the decoder to preserve global dependencies and local detail features. The output of the additive frequency domain attention receiving convolution block First, go through one layer 1 1 convolution and one layer of 3 3. Depthwise separable convolution generates queries ( Query, ),key( Key ) and value ( Value ), and then and Divided into Local blocks of varying sizes are obtained Through two-dimensional fast Fourier transform, and Mapping to the frequency domain, the correlation between the two is calculated to capture global dependencies; the formula for calculating the frequency domain attention weight is: ， ， in, and For the frequency domain representation of the query and key, For two-dimensional fast Fourier transform, For frequency domain attention weights, This indicates taking the real part of a complex number. This indicates element-wise multiplication. The frequency domain representation of the key is the conjugate transpose; its frequency domain weights are mapped back to the spatial domain using the inverse Fourier transform and then normalized to obtain the frequency domain attention features. The calculation formula is: ； Initialize a learnable additive weight matrix Used for values Perform channel weighting; Broadcast to The additive attention weights are obtained by summing all spatial locations along the channel dimension. The calculation formula is: ， in, For the channel, Indicates the channel scaling factor; for After normalizing the spatial dimensions, it is broadcast to all channels, along with the value. Additive attention features can be obtained by multiplying element by element. The calculation formula is: ； Additive attention features Frequency domain attention features Added through residual connections, and then through a layer of 1 1. Convolution adjusts the channel dimensions to obtain the output. The calculation formula is: ， in, This represents the output of the convolution block.

2. The method for enhancing endoscopic images with inconsistent exposure according to claim 1, characterized in that, In step one, the construction process of the denoising network includes: A multi-layer encoder-decoder architecture is constructed to obtain the denoising network, which is used to learn the noise distribution of the endoscopic image during the reverse process of the T-shaped diffusion model in order to recover a high-quality image.

3. The method for enhancing endoscopic images with inconsistent exposure according to claim 1, characterized in that, In step one, the construction process of the color correction module includes: The color correction module is constructed sequentially according to the structure of normalization, convolution, scale transformation, channel attention, scale transformation, convolution, and scale transformation.

4. The method for enhancing endoscopic images with inconsistent exposure according to claim 1, characterized in that, In step two, the loss function of the overall image enhancement network model is adopted. function.

5. The method for enhancing endoscopic images with inconsistent exposure according to claim 1, characterized in that, In step three, the trained image enhancement network model is tested using test data to evaluate its performance.

6. The method for enhancing endoscopic images with inconsistent exposure according to claim 5, characterized in that, Different objective image evaluation metrics are used for cases where endoscopic labeled images are present and cases where endoscopic labeled images are not present in the test data.

Images