Endoscope image enhancement method and endoscope image enhancement equipment

Abstract

The invention provides an endoscope image enhancement method and endoscope image enhancement equipment, and relates to the technical field of image processing. According to the invention, the adaptive color constancy loss function calculated based on the first brightness adaptive weight map in the Lab color space is designed, and the adaptive exposure control loss function calculated based on the second brightness adaptive weight map is designed; the image enhancement network is trained based on the self-adaptive color constancy loss function and the self-adaptive exposure control loss function, so that the trained image enhancement network provided by the embodiment of the invention can be based on the to-be-enhanced endoscope image; adaptive learning is carried out to generate a multi-channel curve parameter graph containing a group of pixel-level brightness enhancement curve parameters, and the group of pixel-level brightness enhancement curve parameters corresponding to each pixel point in the multi-channel curve parameter graph is applied to the endoscope image to be enhanced. And the brightness contrast level is maintained while the color fidelity of the enhanced endoscope image to be enhanced is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an endoscopic image enhancement method and an endoscopic image enhancement device. Background Technology

[0002] Endoscopy is a core technique for diagnosing and treating digestive tract diseases such as gastritis, gastric ulcers, esophageal cancer, and colon cancer. In clinical practice, doctors insert a camera into the patient's digestive tract using an endoscope to acquire real-time images of the internal structure, observe the mucosal condition, identify lesions, and perform pathological analysis. Due to the complex internal environment of the digestive tract (e.g., uneven mucosal reflection, intestinal folds obscuring the view, interference from blood or secretions), and the limitations of the endoscope's light source due to equipment power and physical structure, the acquired images generally suffer from insufficient brightness, low contrast, and color distortion. These problems significantly reduce the visual clarity of the images, affecting the efficiency and accuracy of doctors in identifying minute lesions (such as early-stage cancer and submucosal lesions), and may even lead to missed or misdiagnosed cases. Therefore, how to improve the visual quality of endoscopic images through image enhancement technology is a critical issue that urgently needs to be addressed in the field of medical image processing.

[0003] In related technologies, unsupervised learning models such as the Zero-DCE series models are used to enhance the images of the endoscope to be enhanced. Specifically, the Zero-DCE++ model corrects the color of the endoscope image by optimizing loss functions such as the color constancy loss function and the exposure control loss function, thereby achieving the enhancement of the endoscope image.

[0004] However, the loss functions for color and brightness in the aforementioned Zero-DCE++ model are mainly designed for natural images such as landscapes. Furthermore, the loss functions for color and brightness are based on global non-adaptive constraints to enhance the image, which results in color distortion and low brightness contrast when used to enhance endoscopic images. Summary of the Invention

[0005] The endoscopic image enhancement method and endoscopic image enhancement device provided in this application are used to solve the problems of color distortion and low brightness contrast when performing image enhancement processing on endoscopic images using existing models.

[0006] In a first aspect, this application provides an endoscopic image enhancement method, comprising: acquiring an endoscope image to be enhanced; inputting the endoscope image to be enhanced into an image enhancement network to generate a multi-channel curve parameter map, wherein the image enhancement network is trained using a no-reference loss function, the no-reference loss function including an adaptive color constancy loss function and an adaptive exposure control loss function, the adaptive color constancy loss function being calculated based on a first brightness adaptive weight map generated in the Lab color space, the first brightness adaptive weight map being used to apply differentiated color constraints to different brightness regions in the first endoscope image, the adaptive exposure control loss function being calculated based on a generated second brightness adaptive weight map, the second brightness adaptive weight map being used to apply differentiated exposure correction constraints to different brightness regions in the first endoscope image, the first endoscope image being the input image used to optimize the parameters of the image enhancement network; and enhancing the endoscope image to be enhanced according to a preset brightness enhancement curve and the multi-channel curve parameter map to obtain a target endoscope image.

[0007] In one possible implementation, the adaptive color constancy loss function is calculated as follows: A first endoscope image and a second endoscope image are acquired, wherein the second endoscope image is an endoscope image obtained by enhancing the first endoscope image; the first and second endoscope images are downsampled respectively to obtain a downsampled first endoscope image and a downsampled second endoscope image; the downsampled first and second endoscope images are color space converted respectively to obtain a third endoscope image corresponding to the downsampled first endoscope image in Lab color space, and a fourth endoscope image corresponding to the downsampled second endoscope image; the adaptive color constancy loss function is calculated based on the third and fourth endoscope images.

[0008] In one possible implementation, the adaptive color constancy loss function is calculated based on the third endoscopic image and the fourth endoscopic image, including: generating a first brightness adaptive weight map based on the first brightness value corresponding to each pixel in the third endoscopic image; and calculating the adaptive color constancy loss function based on the first brightness adaptive weight map, the third endoscopic image, and the fourth endoscopic image.

[0009] In one possible implementation, the adaptive exposure control loss function is calculated as follows: color and brightness conversion processing is performed on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain the fifth endoscope image corresponding to the downsampled first endoscope image and the sixth endoscope image corresponding to the downsampled second endoscope image; the adaptive exposure control loss function is calculated based on the fifth endoscope image and the sixth endoscope image.

[0010] In one possible implementation, the adaptive exposure control loss function is calculated based on the fifth and sixth endoscopic images, including: generating a second brightness adaptive weight map based on the second brightness value corresponding to each pixel in the fifth endoscopic image; and calculating the adaptive exposure control loss function based on the second brightness adaptive weight map and the third brightness value corresponding to each pixel in the sixth endoscopic image.

[0011] In one possible implementation, the no-reference loss function also includes a spatial consistency loss function and a curve parametric graph smoothness loss function; the no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function, the adaptive exposure control loss function, the spatial consistency loss function, and the curve parametric graph smoothness loss function.

[0012] In one possible implementation, the image enhancement network comprises multiple convolutional layers, each of which uses element-wise addition for skip connections.

[0013] In one possible implementation, the input convolutional layer in the multiple convolutional layers has 3 input channels and 8 output channels; the output convolutional layer in the multiple convolutional layers has 8 input channels and 3 output channels; and the intermediate convolutional layer in the multiple convolutional layers has 8 input channels and 8 output channels.

[0014] In one possible implementation, before inputting the endoscope image to be enhanced into the image enhancement network, the endoscope image enhancement method provided in this application further includes: normalizing the endoscope image to be enhanced to obtain a normalized endoscope image to be enhanced.

[0015] Secondly, this application provides an endoscopic image enhancement device, comprising:

[0016] The acquisition module is used to acquire images of the endoscope to be enhanced;

[0017] The generation module is used to input the endoscope image to be enhanced into the image enhancement network and generate a multi-channel curve parameter map. The image enhancement network is trained using a no-reference loss function, which includes an adaptive color constancy loss function and an adaptive exposure control loss function. The adaptive color constancy loss function is calculated based on a first brightness adaptive weight map generated in the Lab color space. The first brightness adaptive weight map is used to apply differentiated color constraints to different brightness regions in the first endoscope image. The adaptive exposure control loss function is calculated based on a generated second brightness adaptive weight map. The second brightness adaptive weight map is used to apply differentiated exposure correction constraints to different brightness regions in the first endoscope image. The first endoscope image is the input image used to optimize the parameters of the image enhancement network.

[0018] The enhancement module is used to enhance the endoscope image to be enhanced based on the preset brightness enhancement curve and multi-channel curve parameter map, so as to obtain the target endoscope image.

[0019] In one possible implementation, the adaptive color constancy loss function is calculated as follows: A first endoscope image and a second endoscope image are acquired, wherein the second endoscope image is an endoscope image obtained by enhancing the first endoscope image; the first and second endoscope images are downsampled respectively to obtain a downsampled first endoscope image and a downsampled second endoscope image; the downsampled first and second endoscope images are color space converted respectively to obtain a third endoscope image corresponding to the downsampled first endoscope image in Lab color space, and a fourth endoscope image corresponding to the downsampled second endoscope image; the adaptive color constancy loss function is calculated based on the third and fourth endoscope images.

[0020] In one possible implementation, the adaptive color constancy loss function is specifically calculated as follows: a first brightness adaptive weight map is generated based on the first brightness value corresponding to each pixel in the third endoscopic image; the adaptive color constancy loss function is calculated based on the first brightness adaptive weight map, the third endoscopic image, and the fourth endoscopic image.

[0021] In one possible implementation, the adaptive exposure control loss function is calculated as follows: color and brightness conversion processing is performed on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain the fifth endoscope image corresponding to the downsampled first endoscope image and the sixth endoscope image corresponding to the downsampled second endoscope image; the adaptive exposure control loss function is calculated based on the fifth endoscope image and the sixth endoscope image.

[0022] In one possible implementation, the adaptive exposure control loss function is specifically calculated as follows: a second brightness adaptive weight map is generated based on the second brightness value corresponding to each pixel in the fifth endoscopic image; the adaptive exposure control loss function is calculated based on the second brightness adaptive weight map and the third brightness value corresponding to each pixel in the sixth endoscopic image.

[0023] In one possible implementation, the no-reference loss function also includes a spatial consistency loss function and a curve parametric graph smoothness loss function; the no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function, the adaptive exposure control loss function, the spatial consistency loss function, and the curve parametric graph smoothness loss function.

[0024] In one possible implementation, the image enhancement network comprises multiple convolutional layers, each of which uses element-wise addition for skip connections.

[0025] In one possible implementation, the input convolutional layer in the multiple convolutional layers has 3 input channels and 8 output channels; the output convolutional layer in the multiple convolutional layers has 8 input channels and 3 output channels; and the intermediate convolutional layer in the multiple convolutional layers has 8 input channels and 8 output channels.

[0026] In one possible implementation, the endoscopic image enhancement device provided in this application further includes a normalization module (not shown). Before inputting the endoscopic image to be enhanced into the image enhancement network, the normalization module is used to: normalize the endoscopic image to be enhanced to obtain a normalized endoscopic image to be enhanced.

[0027] Thirdly, this application also provides an endoscopic image enhancement device, including: an input interface and an output interface;

[0028] Input interface for receiving images from the endoscope to be enhanced;

[0029] Output interface, used to output enhanced endoscopic images;

[0030] The endoscopic image enhancement device also includes a processor and a memory communicatively connected to the processor;

[0031] The memory stores instructions that the computer executes;

[0032] The processor executes computer execution instructions stored in memory to implement the endoscopic image enhancement method as provided in the first aspect above.

[0033] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the endoscopic image enhancement method provided in the first aspect above.

[0034] Fifthly, this application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the endoscopic image enhancement method as described in the first aspect above.

[0035] The endoscopic image enhancement method and device provided in this application acquire an endoscope image to be enhanced, input the endoscope image to be enhanced into an image enhancement network, and generate a multi-channel curve parameter map. The parameters of the image enhancement network are obtained by optimization using a no-reference loss function, which includes an adaptive color constancy loss function and an adaptive exposure control loss function. The adaptive color constancy loss function is calculated based on a first brightness adaptive weight map generated in the Lab color space. The first brightness adaptive weight map is used to apply differentiated color constraints to different brightness regions in the first endoscope image. The adaptive exposure control loss function is calculated based on a generated second brightness adaptive weight map. The second brightness adaptive weight map is used to apply differentiated exposure correction constraints to different brightness regions in the first endoscope image. The first endoscope image is the input image used to optimize the parameters of the image enhancement network. According to the preset brightness enhancement curve and the multi-channel curve parameter map, the endoscope image to be enhanced is enhanced to obtain the target endoscope image. In this embodiment, an adaptive color constancy loss function calculated based on a first adaptive brightness weight map in the Lab color space and an adaptive exposure control loss function calculated based on a second adaptive brightness weight map are designed. The image enhancement network is then trained using these adaptive color constancy and adaptive exposure control loss functions. This allows the trained image enhancement network to adaptively learn and generate a multi-channel curve parameter map containing a set of pixel-level brightness enhancement curve parameters based on the endoscope image to be enhanced. Furthermore, the set of pixel-level brightness enhancement curve parameters corresponding to each pixel in the multi-channel curve parameter map is applied to the endoscope image to be enhanced based on a preset brightness enhancement curve. This achieves differentiated color constraints on different brightness regions in the endoscope image to be enhanced, improving the color fidelity of the enhanced endoscope image and applying differentiated exposure correction constraints to different brightness regions in the endoscope image to be enhanced, effectively improving the brightness of dark areas while maintaining the brightness contrast level. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0037] Figure 1 This is an architectural diagram of an endoscopic image enhancement system provided in an embodiment of this application;

[0038] Figure 2 A schematic flowchart of the endoscopic image enhancement method provided in the embodiments of this application;

[0039] Figure 3 This is a schematic diagram of the structure of the image enhancement network provided in the embodiments of this application;

[0040] Figure 4 A flowchart illustrating the calculation method of the adaptive color constancy loss function provided in the embodiments of this application;

[0041] Figure 5 This is a schematic diagram of the Lab color space.

[0042] Figure 6 A flowchart illustrating the calculation method of the adaptive exposure control loss function provided in the embodiments of this application;

[0043] Figure 7 A schematic diagram of a first endoscopic image provided in an embodiment of this application;

[0044] Figure 8 A schematic diagram of a second endoscopic image provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the structure of the endoscopic image enhancement device provided in the embodiments of this application;

[0046] Figure 10 This is a schematic diagram of the structure of the endoscopic image enhancement device provided in the embodiments of this application.

[0047] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0049] In related technologies, methods for enhancing endoscopic images mainly include traditional image enhancement methods and deep learning-based image enhancement methods. Traditional image enhancement methods primarily include Histogram Equalization (HE), Contrast Limited Adaptive Histogram Equalization (CLAHE), tone mapping methods, and methods based on Retinex theory. Retinex theory is a core theory in computer vision and image processing used to explain human visual perception and solve the problem of uneven image illumination. Its core idea is to decompose an image into the inherent reflective properties of the object and the ambient lighting properties, and achieve illumination normalization by separating the two to restore the true visual characteristics of the object.

[0050] In traditional image enhancement methods, the HE method improves global contrast by redistributing pixel gray levels in the endoscopic image. However, the standard HE method tends to over-enhance the background region of the endoscopic image, resulting in severe distortion of the enhanced image. The CLAHE method applies histogram equalization and contrast amplification limits to local regions of the endoscopic image, but this method is highly parameter-dependent; different parameter combinations produce significantly different enhancement effects on different endoscopic images, lacking universality. Furthermore, this method cannot process endoscopic images by understanding their semantic content. The core idea of ​​tone mapping methods is to apply a nonlinear... Tone curves are used to remap pixel brightness values, thereby enhancing dark areas of endoscopic images and avoiding overexposure in bright areas. However, this method also suffers from strong parameter dependence. The Retinex theory assumes that the image observed by the human eye is composed of the reflection component of the object surface and the incident light component of the environment. The goal of image enhancement is to accurately separate the reflection component from the original image to eliminate the influence of uneven lighting. Methods based on this theory include single-scale Retinex and multi-scale Retinex. However, these methods rely on accurate estimation of illumination and have a large computational load, making it difficult to meet the requirements of real-time processing.

[0051] Deep learning-based methods train a deep neural network to learn the complex mapping from low-light to normal-light images, achieving better results than traditional methods. However, many existing deep learning models have complex network structures, a huge number of parameters, and high computational complexity, making them difficult to implement in real-time on resource-constrained endoscopic equipment. Furthermore, mainstream supervised learning methods require a large number of paired contrasting images of low-light and normal-light conditions for training, which is extremely difficult to obtain in endoscopic examination scenarios. Unsupervised learning methods, on the other hand, do not rely on paired data to learn image enhancement. Unsupervised learning models such as Zero-DCE and Zero-DCE++ do not directly learn image-to-image mappings, but instead learn an enhancement curve that dynamically adjusts image brightness based on image content.

[0052] However, on the one hand, the loss functions of these unsupervised learning models are mainly designed for natural images, and the loss functions, such as the color constancy loss function and the exposure control loss function, are non-adaptive. They do not impose differentiated constraints on different regions of the image, but rather impose consistent constraints on different regions of the image. When used to enhance endoscopic images, this can lead to problems such as unrealistic color reproduction and low brightness contrast. On the other hand, the calculation method of the color constancy loss function in these unsupervised learning models is based on the theory of the gray-scale world hypothesis. The average values ​​of the RGB channels in endoscopic images vary greatly, while the gray-scale world hypothesis assumes that the average values ​​of the RGB channels tend to be consistent. Therefore, using this calculation method to process endoscopic images will result in color distortion.

[0053] To address the problems existing in related technologies, this application embodiment designs an adaptive color constancy loss function calculated based on a first brightness adaptive weight map in the Lab color space, and an adaptive exposure control loss function calculated based on a second brightness adaptive weight map. The image enhancement network is then trained based on these adaptive color constancy loss functions and adaptive exposure control loss functions. This allows the trained image enhancement network to adaptively learn and generate a multi-channel curve parameter map containing a set of pixel-level brightness enhancement curve parameters based on the endoscope image to be enhanced. Furthermore, the set of pixel-level brightness enhancement curve parameters corresponding to each pixel in the multi-channel curve parameter map is applied to the endoscope image to be enhanced based on a preset brightness enhancement curve. This achieves differentiated color constraints on different brightness regions in the endoscope image to be enhanced, improving the color fidelity of the enhanced endoscope image and applying differentiated exposure correction constraints to different brightness regions in the endoscope image to be enhanced, effectively improving the brightness of dark areas while maintaining the brightness contrast level.

[0054] The application scenarios of the embodiments of this application will be described below first.

[0055] The endoscopic image enhancement method provided in this application is applicable to the enhancement of endoscopic images, and is also applicable to the enhancement of other images with similar image characteristics to endoscopic images.

[0056] Figure 1 This is an architectural diagram of an endoscopic image enhancement system provided in an embodiment of this application. Figure 1 As shown, the endoscopic image enhancement system includes an image receiving unit 10, an image enhancement network 11, a brightness enhancement module 12, and an image output unit 13.

[0057] The system includes an image receiving unit for receiving an endoscope image to be enhanced; an image enhancement network for generating a multi-channel curve parameter map corresponding to the endoscope image to be enhanced, which includes a set of pixel-level brightness enhancement curve parameters; a brightness enhancement module for applying the brightness enhancement curve parameters to the endoscope image to be enhanced based on a preset brightness enhancement curve, thereby achieving adaptive enhancement of the endoscope image; and an image output unit for outputting the enhanced endoscope image.

[0058] The exemplary image augmentation network is trained using a no-reference loss function.

[0059] like Figure 1 As shown, the no-reference loss function includes the adaptive color constancy loss function. Adaptive exposure control loss function Spatial consistency loss function Smoothness loss function of curve parameter plot .

[0060] When training the image enhancement network, the adaptive color constancy loss function, the adaptive exposure control loss function, and the spatial consistency loss function are calculated based on the input endoscope image and the output endoscope image of the image enhancement network to be trained, which are input to the endoscope image data in the training set; the curve parametric graph smoothness loss function is calculated based on the multi-channel parametric graph output by the image enhancement network to be trained.

[0061] One possible implementation for training the image augmentation network is as follows:

[0062] 1) Based on the endoscopic image data in the training set, the values ​​of adaptive color constancy loss function, adaptive exposure control loss function, and spatial consistency loss function are calculated for the input endoscopic image to the image enhancement network to be trained and the output endoscopic image output by the image output unit, respectively. The value of curve parameter map smoothness loss function is also calculated based on the multi-channel parameter map output by the image enhancement network to be trained.

[0063] 2) The total loss function value corresponding to the no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function value, the adaptive exposure control loss function value, the spatial consistency loss function value, and the curve parametric graph smoothness loss function value;

[0064] 3) Based on the total loss function value, the parameters of the image enhancement network to be trained are continuously optimized through backpropagation algorithm until the parameters of the image enhancement network to be trained converge, thus obtaining the image enhancement network provided in this embodiment of the application.

[0065] The endoscopic image enhancement method provided in this application will be described in detail below with reference to specific embodiments.

[0066] Figure 2 This is a schematic flowchart of an endoscopic image enhancement method provided in an embodiment of this application. Figure 2 As shown, the implementation of this endoscopic image enhancement method may include the following steps:

[0067] S201, Obtain the image of the endoscope to be enhanced.

[0068] For example, the endoscopic image to be enhanced is a low-light endoscopic image.

[0069] For example, the pixel intensity of each pixel in the enhanced endoscopic image is between 0 and 1.

[0070] One possible implementation of this step is to acquire the original endoscopic image output by the clinical endoscopic imaging device, and then perform bilateral filtering on the original endoscopic image to obtain the endoscopic image to be enhanced.

[0071] Understandably, by performing Gaussian filtering on the original endoscopic image, high-frequency noise in the original endoscopic image can be removed, thus avoiding interference from high-frequency noise with subsequent image enhancement.

[0072] S202, input the endoscopic image to be enhanced into the image enhancement network to generate a multi-channel curve parameter map.

[0073] The image enhancement network provided in this application embodiment is trained using a no-reference loss function, which includes an adaptive color constancy loss function and an adaptive exposure control loss function. The adaptive color constancy loss function is calculated based on a first brightness adaptive weight map generated in the Lab color space. This first brightness adaptive weight map is used to apply differentiated color constraints to different brightness regions in the first endoscope image. The adaptive exposure control loss function is calculated based on a generated second brightness adaptive weight map. This second brightness adaptive weight map is used to apply differentiated exposure correction constraints to different brightness regions in the first endoscope image. The first endoscope image is the input image used to optimize the parameters of the image enhancement network.

[0074] The adaptive color constancy loss function and adaptive exposure control loss function provided in this application will be described in detail below with reference to specific embodiments.

[0075] For example, an image enhancement network can be a lightweight deep neural network.

[0076] Figure 3 This is a schematic diagram of the structure of an image enhancement network provided in an embodiment of this application. Figure 3 As shown, the image enhancement network consists of 7 convolutional layers, each with a kernel size of 3x3, a stride of 1, and padding of 1.

[0077] like Figure 3 As shown, starting from the input of the image enhancement network, the output of the first convolutional layer jumps to the input of the sixth convolutional layer, the output of the second convolutional layer jumps to the input of the fifth convolutional layer, and the output of the third convolutional layer jumps to the input of the fourth convolutional layer.

[0078] like Figure 3 As shown, starting from the input of the image enhancement network, each of the first to sixth convolutional layers is followed by a Rectified Linear Unit (ReLU) activation function to enhance the nonlinearity of the image enhancement network and improve its feature extraction capability; the seventh and last convolutional layer is connected to a Tanh activation function to output the enhanced multi-channel curve parameter map.

[0079] In this step, for example, the size of the multi-channel curve parameter plot is the same as the size of the endoscopic image to be enhanced.

[0080] For example, a multi-channel curve parameter plot can be a three-channel curve parameter plot containing RGB channels. Among them, the three-channel curve parameter diagram Each pixel in the image corresponds to a set of brightness enhancement curve parameters used to adjust the brightness of the corresponding pixel in the endoscope image to be enhanced.

[0081] For example, a three-channel curve parameter plot The set of brightness enhancement curve parameters corresponding to each pixel in the image can be represented by the following formula:

[0082]

[0083] in, This represents the brightness enhancement curve parameters corresponding to the i-th pixel in the R channel. This represents the brightness enhancement curve parameters corresponding to the i-th pixel in the G channel. This represents the brightness enhancement curve parameters corresponding to the i-th pixel in channel B.

[0084] In this step, one possible implementation is as follows: the image enhancement network performs a preset downsampling process on the input endoscopic image to be enhanced, and then inputs it into the above-mentioned... Figure 3 The image is processed by multiple convolutional layers, and then the image output by the last convolutional layer is upsampled by a preset factor to generate a multi-channel curve parameter map.

[0085] For example, the preset multiplier can be 12 times. This application does not limit the preset multiplier; it can be determined based on actual application requirements.

[0086] S203, according to the preset brightness enhancement curve and multi-channel curve parameter diagram, perform enhancement processing on the endoscope image to be enhanced to obtain the target endoscope image.

[0087] For example, the preset brightness enhancement curve can be a quadratic curve.

[0088] One possible implementation method for this step is as follows: based on a set of brightness enhancement curve parameters corresponding to each pixel in the preset brightness enhancement curve and multi-channel curve parameter map, perform non-linear adjustment pixel by pixel in the corresponding pixel in the endoscope image to be enhanced to obtain the enhanced pixel, and then obtain the enhanced endoscope image, i.e., the target endoscope image.

[0089] For example, pixels in a target endoscopic image can be represented by the following formula:

[0090]

[0091] in, This represents the c-th channel of the i-th pixel in the target endoscopic image. This represents the c-th channel of the i-th pixel in the enhanced endoscopic image. This represents a set of brightness enhancement curve parameters corresponding to the c-th channel of the i-th pixel in the multi-channel curve parameter diagram.

[0092] In this embodiment of the application, by applying different brightness enhancement curve parameters to each pixel in the image to be enhanced by endoscopy, dynamic brightness enhancement can be achieved in different regions of the image to be enhanced by endoscopy.

[0093] In this embodiment, an adaptive color constancy loss function calculated based on a first adaptive brightness weight map in the Lab color space and an adaptive exposure control loss function calculated based on a second adaptive brightness weight map are designed. The image enhancement network is then trained using these adaptive color constancy and adaptive exposure control loss functions. This allows the trained image enhancement network to adaptively learn and generate a multi-channel curve parameter map containing a set of pixel-level brightness enhancement curve parameters based on the endoscope image to be enhanced. Furthermore, the set of pixel-level brightness enhancement curve parameters corresponding to each pixel in the multi-channel curve parameter map is applied to the endoscope image to be enhanced based on a preset brightness enhancement curve. This achieves differentiated color constraints on different brightness regions in the endoscope image to be enhanced, improving the color fidelity of the enhanced endoscope image and applying differentiated exposure correction constraints to different brightness regions in the endoscope image to be enhanced, effectively improving the brightness of dark areas while maintaining the brightness contrast level.

[0094] The following is combined with Figure 4 The calculation method of the adaptive color constancy loss function provided in the embodiments of this application will be described in detail.

[0095] Figure 4 This is a flowchart illustrating the calculation method of the adaptive color constancy loss function provided in the embodiments of this application. Figure 4 As shown, the implementation of the adaptive color constancy loss function calculation method may include the following steps:

[0096] S401, acquire a first endoscope image and a second endoscope image, wherein the second endoscope image is an endoscope image obtained by enhancing the first endoscope image.

[0097] For example, the first endoscopic image may be a frame of an endoscopic image from an endoscopic image training dataset. This endoscopic image training dataset is used to train the image enhancement network.

[0098] For example, the first endoscopic image may be a low-light endoscopic image.

[0099] For example, the second endoscopic image can be an endoscopic image after enhancing the first endoscopic image by using a multi-channel curve parameter map corresponding to the first endoscopic image output by the image enhancement network to be trained.

[0100] One possible implementation of this step is as follows: randomly select an endoscope image from historical endoscope images acquired by a clinical endoscope imaging device as the first endoscope image; perform Gaussian filtering on the first endoscope image; input the filtered first endoscope image into the image enhancement network to be trained; generate a multi-channel curve parameter map corresponding to the filtered first endoscope image; and perform enhancement processing on the filtered first endoscope image based on the multi-channel curve parameter map to obtain the second endoscope image.

[0101] S402, the first endoscope image and the second endoscope image are downsampled respectively to obtain the downsampled first endoscope image and the downsampled second endoscope image.

[0102] In one possible implementation, the first endoscope image is downsampled by 4 times to obtain the downsampled first endoscope image, and the second endoscope image is downsampled by 4 times to obtain the downsampled second endoscope image.

[0103] In this embodiment of the application, the first endoscope image and the second endoscope image are downsampled respectively to reduce the amount of computation and speed up the training of the network for the image enhancement to be trained.

[0104] It should be noted that, in this embodiment of the application, the first endoscope image and the second endoscope image are downsampled by the same factor to ensure that the size of the first endoscope image and the second endoscope image after downsampling are the same.

[0105] S403, perform color space conversion processing on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain the third endoscope image corresponding to the downsampled first endoscope image in Lab color space, and the fourth endoscope image corresponding to the downsampled second endoscope image.

[0106] First, let's explain the Lab color space.

[0107] The Lab color space, also known as the CIE Lab color space, is a color model that is more in line with human visual perception. The Lab color space separates the brightness (L) of an image from its chromaticity (a and b).

[0108] Figure 5 This is a schematic diagram of the Lab color space. Figure 5As shown, in this Lab color space, the L component represents the image's luminance, ranging from white to black; the a component represents the image's chroma, ranging from green to red; and the b channel represents the image's chroma, ranging from blue to yellow. The values ​​of L can range from 0 to 100, a from -127 to 128, and b from -127 to 128.

[0109] In this step, one possible implementation is to convert the color space of the downsampled first endoscope image from RGB color space to Lab color space to obtain the third endoscope image corresponding to the first endoscope image after downsampling in Lab color space; and to convert the color space of the downsampled second endoscope image from RGB color space to Lab color space to obtain the fourth endoscope image corresponding to the second endoscope image after downsampling in Lab color space.

[0110] S404, Calculate the adaptive color constancy loss function based on the third and fourth endoscopic images.

[0111] For example, the color parameter corresponding to each pixel in the third endoscopic image can be represented as: .in, This represents the brightness value corresponding to a pixel. This represents the chromaticity value of the 'a' component corresponding to the pixel. This represents the chromaticity value of the b component corresponding to a pixel.

[0112] For example, the color parameter corresponding to each pixel in the fourth endoscopic image can be represented as: .in, This represents the brightness value corresponding to a pixel. This represents the chromaticity value of the 'a' component corresponding to the pixel. This represents the chromaticity value of the b component corresponding to a pixel.

[0113] In one possible implementation, a first brightness adaptive weight map is generated based on the first brightness value corresponding to each pixel in the third endoscopic image; and an adaptive color constancy loss function is calculated based on the first brightness adaptive weight map, the third endoscopic image, and the fourth endoscopic image.

[0114] For example, the size of the first brightness adaptive weight map is the same as the size of the third and fourth endoscopic images.

[0115] For example, the weight corresponding to each pixel in the first brightness adaptive weight map can be represented by the following formula:

[0116]

[0117] in, This represents the weight corresponding to the i-th pixel in the first brightness adaptive weight map. This represents the first brightness value corresponding to the i-th pixel in the third endoscopic image. This is an adjustable parameter, and its value can be 2.

[0118] For example, the adaptive color constancy loss function can be expressed by the following formula:

[0119]

[0120] in, This represents the adaptive color constancy loss function. This represents the chromaticity value of the a component corresponding to the i-th pixel in the fourth endoscopic image. This represents the chromaticity value of the a component corresponding to the i-th pixel in the third endoscopic image. This represents the chromaticity value of the b-component corresponding to the i-th pixel in the fourth endoscopic image. M represents the chromaticity value of the b component corresponding to the i-th pixel in the third endoscopic image, and M represents the total number of pixels in the first endoscopic image after downsampling, or M represents the total number of pixels in the second endoscopic image after downsampling.

[0121] It is understandable that the total number of pixels in the first endoscopic image after sampling is the same as the total number of pixels in the second endoscopic image after downsampling.

[0122] Compared to the color constancy loss function used in related technologies, which typically assumes that the average intensity of each channel of an image should be approximately equal, this does not hold true for endoscopic images that are predominantly reddish. Therefore, using the color constancy loss function to enhance the brightness of endoscopic images can lead to severe color distortion. This application provides an adaptive color constancy loss function calculated in the Lab color space. Specifically, by using the brightness value corresponding to each pixel in the input endoscopic image (i.e., the first endoscopic image) of the image enhancement network to be trained in the Lab color space, a first brightness adaptive weight map corresponding to the first endoscopic image is generated. Furthermore, based on the weight corresponding to each pixel in the first brightness adaptive weight map, dynamic color constraints are applied to the chromaticity values ​​corresponding to each pixel in the first and second endoscopic images, thereby improving the color fidelity of the enhanced endoscopic image.

[0123] The following is combined with Figure 6 The calculation method of the adaptive exposure control loss function provided in the embodiments of this application will be described in detail.

[0124] Figure 6This is a flowchart illustrating the calculation method of the adaptive exposure control loss function provided in an embodiment of this application. Figure 6 As shown, the implementation of the adaptive exposure control loss function calculation method may include the following steps:

[0125] S601, acquire a first endoscope image and a second endoscope image, wherein the second endoscope image is an image obtained by enhancing the first endoscope image.

[0126] The specific implementation method is similar to the above embodiments, and will not be repeated here.

[0127] S602, the first endoscope image and the second endoscope image are downsampled respectively to obtain the downsampled first endoscope image and the downsampled second endoscope image.

[0128] The specific implementation method is similar to the above embodiments, and will not be repeated here.

[0129] It should be noted that when downsampling the first and second endoscopic images, the downsampling factor used to calculate the adaptive color constancy loss function can be the same as or different from the downsampling factor used to calculate the exposure control loss function. The specific factor can be determined according to the actual application requirements, and this application embodiment does not limit this.

[0130] S603, color and brightness conversion processing is performed on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain the fifth endoscope image corresponding to the downsampled first endoscope image and the sixth endoscope image corresponding to the downsampled second endoscope image.

[0131] In this step, one possible implementation is to convert the downsampled first endoscope image into a grayscale image to obtain the fifth endoscope image corresponding to the downsampled first endoscope image; and to convert the downsampled second endoscope image into a grayscale image to obtain the sixth endoscope image corresponding to the downsampled second endoscope image.

[0132] S604, calculate the adaptive exposure control loss function based on the fifth and sixth endoscopic images.

[0133] For example, the brightness value corresponding to each pixel in the fifth endoscopic image can be represented as: .

[0134] For example, the brightness value corresponding to each pixel in the sixth endoscopic image can be represented as: .

[0135] It should be noted that the brightness value of each pixel in the fifth and sixth endoscopic images is a value between 0 and 1.

[0136] In one possible implementation, a second brightness adaptive weight map is generated based on the second brightness value corresponding to each pixel in the fifth endoscopic image; and an adaptive exposure control loss function is calculated based on the second brightness adaptive weight map and the third brightness value corresponding to each pixel in the sixth endoscopic image.

[0137] For example, the size of the second brightness adaptive weight map is the same as the size of the fifth and sixth endoscopic images.

[0138] For example, the weight corresponding to each pixel in the second brightness adaptive weight map can be represented by the following formula:

[0139]

[0140] in, This represents the weight corresponding to the i-th pixel in the second brightness adaptive weight map. This represents the second brightness value corresponding to the i-th pixel in the fifth endoscopic image. It is an adjustable parameter, and its value can be 1.5.

[0141] For example, the adaptive exposure control loss function can be expressed by the following formula:

[0142]

[0143] in, This represents the adaptive exposure control loss function. This represents the third brightness value corresponding to the i-th pixel in the sixth endoscopic image. This represents the preset ideal brightness value, which can be 0.6. K represents the total number of pixels in the first endoscopic image after downsampling, or M represents the total number of pixels in the second endoscopic image after downsampling.

[0144] It is understandable that when downsampling the first and second endoscopic images, K and M have the same value when the downsampling factor for calculating the adaptive color constancy loss function is the same as that for calculating the exposure control loss function; when the downsampling factor for calculating the adaptive color constancy loss function is different from that for calculating the exposure control loss function, K and M have different values.

[0145] It is understandable that endoscopic images typically contain both shadow and highlight areas. Compared to the exposure control loss function used in related technologies, which requires the average brightness of the enhanced endoscopic image to reach a fixed value, this approach cannot handle complex brightness distributions. This causes the brightness of both shadow and highlight areas to simultaneously approach this fixed value, thus reducing the brightness contrast of the endoscopic image. This application provides an adaptive exposure control loss function that, by setting a uniform ideal exposure level, applies different penalty weights to the exposure errors of different areas based on the brightness distribution of the fifth endoscopic image. Specifically, a larger penalty weight is applied to the dark areas of the fifth endoscopic image to ensure they are effectively brightened, while a smaller penalty weight is applied to the bright areas to allow them to maintain higher brightness, thereby maintaining good brightness contrast.

[0146] Optionally, the no-reference loss function for training the image enhancement network provided in this application embodiment further includes a spatial consistency loss function and a curve parametric graph smoothness loss function. The no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function, the adaptive exposure control loss function, the spatial consistency loss function, and the curve parametric graph smoothness loss function.

[0147] For example, the no-reference loss function can be expressed by the following formula:

[0148]

[0149] in, Indicates the no-reference loss function. This represents the weights corresponding to the adaptive color constancy loss function. This represents the weights corresponding to the adaptive exposure control loss function. Represents the spatial consistency loss function. The weights represent the spatial consistency loss function. This represents the smoothness loss function of the curve parametric plot. This represents the weights corresponding to the curve parametric graph smoothness loss function.

[0150] For example, the spatial consistency loss function is used to constrain the neighborhood differences of the first endoscopic image and the neighborhood differences of the second endoscopic image to remain consistent.

[0151] For example, the spatial consistency loss function can be expressed by the following formula:

[0152]

[0153] Where N is the number of local regions in the first or second endoscopic image. Indicates the first Four adjacent regions of a local region Indicating the second endoscopic image, the first The average brightness of a local area Indicating the second endoscopic image, the first The average brightness value of the j-th neighboring region of a local region. Indicates the first endoscopic image The average brightness of a local area Indicating the second endoscopic image, the first The average brightness value of the j-th neighboring region of a local region.

[0154] Figure 7 This is a schematic diagram of a first endoscopic image provided in an embodiment of this application. Figure 7 As shown, the first endoscopic image is divided into N grids. The grid is defined, with each grid cell representing a local region, and will be compared with the first... The four adjacent local regions of a given local region are defined as the neighborhood of the nth local region.

[0155] Figure 8 This is a schematic diagram of a second endoscopic image provided in an embodiment of this application. Figure 8 As shown, the second endoscopic image is divided into N grids. The grid is defined, with each grid cell representing a local region, and will be compared with the first... The four adjacent local regions of a given local region are defined as the neighborhood of the nth local region.

[0156] For example, the curve parametric map smoothness loss function is used to constrain the total variation loss of the multi-channel curve parametric map output by the image enhancement network to be trained. Specifically, by penalizing the difference between adjacent pixels in the multi-channel curve parametric map, the image enhancement network to be trained is prompted to generate a smooth multi-channel curve parametric map.

[0157] For example, the curve parametric plot smoothness loss function can be expressed by the following formula:

[0158]

[0159] in, This represents the smoothness loss function of the parametric curve plot, where T represents the number of pixels in the multi-channel parametric curve plot. This represents the gradient operator in the horizontal direction. This represents the gradient operator in the vertical direction.

[0160] Optionally, the image enhancement network provided in this application embodiment includes multiple convolutional layers, and the skip connection method of each convolutional layer adopts element-wise addition.

[0161] As mentioned above Figure 3 In the schematic diagram of the image enhancement network shown, the number of convolutional layers can be up to 7.

[0162] Compared to the channel splicing method used in the skip connection method of each convolutional layer in related technologies, the embodiments of this application use element-wise addition in the skip connection method of each convolutional layer in the image enhancement network, which can reduce the number of input channels of the image enhancement network and thus reduce the computational load of the image enhancement network.

[0163] In one possible implementation, the input convolutional layer has 3 input channels and 8 output channels; the output convolutional layer has 8 input channels and 3 output channels; and the intermediate convolutional layer has 8 input channels and 8 output channels.

[0164] The embodiments of this application can significantly reduce the computational load of the image enhancement network by optimizing the number of input and output channels of each convolutional layer in the image enhancement network.

[0165] In summary, it can be understood that the endoscopic image enhancement method provided in this application is a lightweight adaptive endoscopic image enhancement method based on unsupervised learning.

[0166] Optionally, the endoscopic image enhancement method provided in this application embodiment further includes, before inputting the endoscopic image to be enhanced into the image enhancement network, performing normalization processing on the endoscopic image to be enhanced to obtain a normalized endoscopic image to be enhanced.

[0167] For example, the pixel intensity of each pixel in the normalized endoscopic image to be enhanced is between 0 and 1.

[0168] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0169] Figure 9 This is a schematic diagram of the structure of the endoscopic image enhancement device provided in an embodiment of this application. Figure 9 As shown, the endoscopic image enhancement device 90 includes an acquisition module 910, a generation module 920, and an enhancement module 930.

[0170] Among them, the acquisition module 910 is used to acquire the image of the endoscope to be enhanced;

[0171] The generation module 920 is used to input the endoscope image to be enhanced into the image enhancement network and generate a multi-channel curve parameter map. The image enhancement network is trained by a no-reference loss function, which includes an adaptive color constancy loss function and an adaptive exposure control loss function. The adaptive color constancy loss function is calculated based on a first brightness adaptive weight map generated in the Lab color space. The first brightness adaptive weight map is used to apply differentiated color constraints to different brightness regions in the first endoscope image. The adaptive exposure control loss function is calculated based on a generated second brightness adaptive weight map. The second brightness adaptive weight map is used to apply differentiated exposure correction constraints to different brightness regions in the first endoscope image. The first endoscope image is the input image used to optimize the parameters of the image enhancement network.

[0172] The enhancement module 930 is used to enhance the endoscope image to be enhanced according to the preset brightness enhancement curve and multi-channel curve parameter map to obtain the target endoscope image.

[0173] In one possible implementation, the adaptive color constancy loss function is calculated as follows: A first endoscope image and a second endoscope image are acquired, wherein the second endoscope image is an endoscope image obtained by enhancing the first endoscope image; the first and second endoscope images are downsampled respectively to obtain a downsampled first endoscope image and a downsampled second endoscope image; the downsampled first and second endoscope images are color space converted respectively to obtain a third endoscope image corresponding to the downsampled first endoscope image in Lab color space, and a fourth endoscope image corresponding to the downsampled second endoscope image; the adaptive color constancy loss function is calculated based on the third and fourth endoscope images.

[0174] In one possible implementation, the adaptive color constancy loss function is specifically calculated as follows: a first brightness adaptive weight map is generated based on the first brightness value corresponding to each pixel in the third endoscopic image; the adaptive color constancy loss function is calculated based on the first brightness adaptive weight map, the third endoscopic image, and the fourth endoscopic image.

[0175] In one possible implementation, the adaptive exposure control loss function is calculated as follows: color and brightness conversion processing is performed on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain the fifth endoscope image corresponding to the downsampled first endoscope image and the sixth endoscope image corresponding to the downsampled second endoscope image; the adaptive exposure control loss function is calculated based on the fifth endoscope image and the sixth endoscope image.

[0176] In one possible implementation, the adaptive exposure control loss function is specifically calculated as follows: a second brightness adaptive weight map is generated based on the second brightness value corresponding to each pixel in the fifth endoscopic image; the adaptive exposure control loss function is calculated based on the second brightness adaptive weight map and the third brightness value corresponding to each pixel in the sixth endoscopic image.

[0177] In one possible implementation, the no-reference loss function also includes a spatial consistency loss function and a curve parametric graph smoothness loss function; the no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function, the adaptive exposure control loss function, the spatial consistency loss function, and the curve parametric graph smoothness loss function.

[0178] In one possible implementation, the image enhancement network comprises multiple convolutional layers, each of which uses element-wise addition for skip connections.

[0179] In one possible implementation, the input convolutional layer in the multiple convolutional layers has 3 input channels and 8 output channels; the output convolutional layer in the multiple convolutional layers has 8 input channels and 3 output channels; and the intermediate convolutional layer in the multiple convolutional layers has 8 input channels and 8 output channels.

[0180] In one possible implementation, the endoscopic image enhancement device provided in this application further includes a normalization module (not shown). Before inputting the endoscopic image to be enhanced into the image enhancement network, the normalization module is used to: normalize the endoscopic image to be enhanced to obtain a normalized endoscopic image to be enhanced.

[0181] The endoscopic image enhancement device provided in this embodiment can perform the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0182] Figure 10 This is a schematic diagram of the structure of the endoscopic image enhancement device provided in an embodiment of this application. Figure 10 As shown, the endoscopic image enhancement device 11 includes: an input interface 12 and an output interface 13;

[0183] The input interface 12 is used to receive the endoscope image to be enhanced; the output interface 13 is used to output the enhanced endoscope image.

[0184] Optionally, the endoscopic image enhancement device 11 also includes at least one processor 14 and a memory 15.

[0185] Optionally, the endoscopic image enhancement device 110 also includes a communication component 16. The processor 14, memory 15, and communication component 16 are connected via a bus 17.

[0186] In a specific implementation, at least one processor 14 executes computer execution instructions stored in memory 15, causing at least one processor 14 to perform the above-described method.

[0187] The specific implementation process of processor 14 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0188] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Embedded Neural-network Processing Unit (NPU), other general-purpose processors, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0189] The memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk storage device.

[0190] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0191] It should be noted that the endoscopic image enhancement device provided in this application embodiment can be used in an endoscope host or in an artificial intelligence (AI) box such as an edge computing device used in conjunction with an endoscope system.

[0192] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0193] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0194] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0195] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0196] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0199] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0200] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0201] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An endoscope image enhancement method characterized by, The method comprises: obtaining an endoscope image to be enhanced; inputting the endoscope image to be enhanced into an image enhancement network to generate a multi-channel curve parameter map, wherein the image enhancement network is trained by a no-reference loss function, the no-reference loss function comprises an adaptive color constancy loss function and an adaptive exposure control loss function, the adaptive color constancy loss function is calculated based on a first luminance adaptive weight map generated in a Lab color space, the first luminance adaptive weight map is used to apply differentiated color constraints to different luminance regions in a first endoscope image, the adaptive exposure control loss function is calculated based on a second luminance adaptive weight map generated, the second luminance adaptive weight map is used to apply differentiated exposure correction constraints to different luminance regions in the first endoscope image, and the first endoscope image is an input image used to optimize parameters of the image enhancement network; performing enhancement processing on the endoscope image to be enhanced according to a preset luminance enhancement curve and the multi-channel curve parameter map to obtain a target endoscope image.

2. The endoscope image enhancement method of claim 1, wherein, The adaptive color constancy loss function is calculated by the following method: obtaining the first endoscope image and a second endoscope image, wherein the second endoscope image is an endoscope image obtained by performing enhancement processing on the first endoscope image; performing downsampling processing on the first endoscope image and the second endoscope image respectively to obtain a downsampled first endoscope image and a downsampled second endoscope image; performing color space conversion processing on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain a third endoscope image corresponding to the downsampled first endoscope image in the Lab color space and a fourth endoscope image corresponding to the downsampled second endoscope image in the Lab color space; calculating the adaptive color constancy loss function according to the third endoscope image and the fourth endoscope image.

3. The endoscopic image enhancement method of claim 2, wherein, The calculation of the adaptive color constancy loss function according to the third endoscope image and the fourth endoscope image comprises: generating the first luminance adaptive weight map according to a first luminance value corresponding to each pixel point in the third endoscope image; calculating the adaptive color constancy loss function according to the first luminance adaptive weight map, the third endoscope image and the fourth endoscope image.

4. The endoscopic image enhancement method of claim 2, wherein, The adaptive exposure control loss function is calculated by the following method: performing color luminance conversion processing on the downsampled first endoscope image and the downsampled second endoscope image respectively to obtain a fifth endoscope image corresponding to the downsampled first endoscope image and a sixth endoscope image corresponding to the downsampled second endoscope image; calculating the adaptive exposure control loss function according to the fifth endoscope image and the sixth endoscope image.

5. The endoscopic image enhancement method of claim 4, wherein, The calculation of the adaptive exposure control loss function according to the fifth endoscope image and the sixth endoscope image comprises: According to the second brightness value corresponding to each pixel point in the fifth endoscope image, a second brightness adaptive weight map is generated; According to the second brightness adaptive weight map and the third brightness value corresponding to each pixel point in the sixth endoscope image, the adaptive exposure control loss function is calculated.

6. The endoscope image enhancement method according to any one of claims 1 to 5, characterized by, The no-reference loss function further includes a spatial consistency loss function and a curve parameter map smoothness loss function. The no-reference loss function is obtained by weighted summation of the adaptive color constancy loss function, the adaptive exposure control loss function, the spatial consistency loss function, and the curve parameter map smoothness loss function.

7. The endoscope image enhancement method according to any one of claims 1 to 5, characterized by, The image enhancement network includes a plurality of convolutional layers, and the skip connection mode of each convolutional layer in the plurality of convolutional layers adopts element-by-element addition.

8. The endoscopic image enhancement method of claim 7, wherein, The input channel number of the input convolutional layer in the plurality of convolutional layers is 3, and the output channel number is 8; the input channel number of the output convolutional layer in the plurality of convolutional layers is 8, and the output channel number is 3; the input channel number and the output channel number of the intermediate convolutional layer in the plurality of convolutional layers are both 8.

9. The endoscope image enhancement method according to any one of claims 1 to 5, characterized by, Before the to-be-enhanced endoscope image is input into the image enhancement network, the method further includes: normalizing the to-be-enhanced endoscope image to obtain a normalized to-be-enhanced endoscope image.

10. An endoscope image enhancement device, characterized by, It includes: an input interface and an output interface; the input interface is used to receive a to-be-enhanced endoscope image; the output interface is used to output an enhanced endoscope image; The endoscope image enhancement device further includes a processor and a memory in communication with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to realize the endoscope image enhancement method according to any one of claims 1 to 9.

Images