A zero-reference lightweight low-light endoscopic video enhancement method

Abstract

The present invention discloses a zero-reference lightweight low-light endoscopic video enhancement method, which includes the following steps: obtaining an endoscopic video frame image, creating an output video frame folder, inputting the endoscopic video frame image into an improved low-light image enhancement network for processing, obtaining a preliminary brightness-enhanced endoscopic video frame image and correcting it, obtaining a color-corrected endoscopic video frame image, and processing the color-corrected endoscopic video frame image to obtain a final enhanced endoscopic video frame image, placing it in an output video frame folder, and recombining the final enhanced endoscopic video frame image in the output video frame folder into a low-light-enhanced endoscopic video through a video interpolation recombination algorithm. The present invention processes the endoscopic video frame image by improving the low-light image enhancement network, further improving the brightness of the endoscopic video frame image, and improving the overall quality of the endoscopic video frame image.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscope video enhancement processing, and in particular to a zero-reference lightweight low-light endoscope video enhancement method. Background Art

[0002] Endoscopic technology is widely used in medical diagnosis and surgery, especially in the inspection and treatment of cavities such as the digestive tract and respiratory tract. The clarity and color information of endoscopic video are crucial for doctors to make accurate diagnoses and operations. Due to the size limitations of endoscopes, the light source carried by endoscopes provides very limited lighting when inspecting inside the human body. There are many low-light conditions, which make it difficult to clearly distinguish dark details, affecting doctors' accurate diagnosis and operations.

[0003] Current low-light endoscope video enhancement technologies mainly include traditional image enhancement methods such as histogram equalization and contrast-limited adaptive histogram equalization, denoising techniques such as bilateral filtering, and image enhancement models based on deep learning, such as image enhancement networks based on the theory of image processing technology based on the principles of the human visual system. In addition, the generative adversarial network (GAN) technology has also been introduced to improve image brightness, contrast, and detail retention in low-light environments.

[0004] Although existing methods can improve the quality of endoscopic videos to a certain extent, there are still significant deficiencies, including the following problems:

[0005] (1) Traditional image enhancement methods uniformly enhance the entire image, which can easily lead to excessive brightness enhancement of the image, resulting in noise amplification and detail distortion, and obvious artifacts. Especially in low-light scenes, subtle lesion areas may be masked by noise or become difficult to identify.

[0006] (2) Image enhancement models based on deep learning and generative adversarial networks (GANs) have high computational complexity, especially in real-time processing of endoscopic videos, which makes it difficult to meet clinical real-time requirements. They also require high-computing GPU devices for processing, making them difficult to apply to resource-constrained endoscopic equipment, which limits their practical use. In addition, paired reference endoscopic image datasets with ideal lighting are very difficult to obtain, making it difficult to train deep learning image enhancement models with good enhancement effects. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a zero-reference, lightweight, low-light endoscopic video enhancement method to solve the problems raised in the above-mentioned background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a zero-reference lightweight low-light endoscope video enhancement method, comprising the following steps:

[0009] Step S1: obtaining an endoscope video file, extracting the endoscope video file, obtaining an endoscope video frame image, and creating an output video frame folder;

[0010] Step S2: inputting the endoscopic video frame image extracted in step S1 into the improved low-light image enhancement network for processing to obtain a preliminary brightness-enhanced endoscopic video frame image;

[0011] Step S3: Correcting the preliminary brightness-enhanced endoscopic video frame image obtained in step S2 according to the linear color of the endoscopic video frame image to obtain a color-corrected endoscopic video frame image;

[0012] Step S4: The color-corrected endoscopic video frame image obtained in step S3 is processed by a histogram equalization algorithm, image brightening and detail enhancement to generate three sub-images, and the three sub-images are weightedly fused by using local contrast and local entropy to obtain a final enhanced endoscopic video frame image;

[0013] Step S5: placing the final enhanced endoscopic video frame image obtained in step S4 in an output video frame folder, and recombining the final enhanced endoscopic video frame image in the output video frame folder into a low-light enhanced endoscopic video through a video interpolation and recombination algorithm.

[0014] Further, in step S1, an endoscope video frame image is obtained, and the specific process is as follows:

[0015] The endoscopic video collected during the medical endoscopic examination is placed in a folder directory, and the endoscopic video placed in the folder directory is extracted by a video key frame extraction algorithm to obtain an endoscopic video frame image.

[0016] Further, in step S2, a preliminary brightness-enhanced endoscope video frame image is obtained, and the specific process is as follows:

[0017] Step S21: the original code of the low-light image enhancement network uses a standard convolution layer, and the standard convolution layer in the improved low-light image enhancement network is replaced by a depthwise separable convolution layer; the depthwise separable convolution layer includes depthwise convolution and pointwise convolution;

[0018] Step S22: Enhance each input channel in the low-light image network through deep convolution Perform independent spatial convolution to extract channels Spatial feature information; then extract the channel through point 1x1 convolution fusion Spatial feature information, generate a new feature map; for the input endoscope video frame image Dimensions: H x W u × The depthwise separable convolutional layer is processed in two steps; H represents the height of the input endoscope video frame image; W u Represents the width of the input endoscope video frame image;

[0019] Each input channel is processed by depthwise convolution Perform independent spatial convolution, expressed as:

[0020] (1);

[0021] In the formula, Indicates that on each input channel The pixel value of the output image at the index position (i, j) above; and Respectively represent the index of the convolution kernel in the horizontal and vertical directions; Indicates channel The convolution kernel on ; Indicates that the input endoscope video frame image is in channel The pixel value at the index position (i, j) above; Represents the index in the vertical direction of the input endoscope video frame image; Represents the index on the horizontal width of the input image;

[0022] Extract channels by point convolution and degree fusion Spatial feature information generates a new feature map, which is expressed as:

[0023] (2);

[0024] In the formula, Indicates that the new feature map is output at position The value of Represents the weight of 1x1 convolution;

[0025] Step S23: To generate a new feature map, a 1x1 convolutional layer is used to generate weights, which are then normalized by an activation function to calculate the new feature map. The attention weight of the pixel position is obtained by comparing the endoscope video frame image with the new feature map Multiply the attention weights of the pixel positions to obtain the feature map after attention enhancement;

[0026] Calculate the new feature map The attention weight at the pixel position is expressed as:

[0027] (3);

[0028] In the formula, Indicates the new feature map Attention weights at pixel locations; represents the activation function; Represents a 1x1 convolutional layer; Indicates the endoscope video frame image Attention weights at pixel locations;

[0029] The feature map after attention enhancement is obtained, expressed as:

[0030] (4);

[0031] In the formula, Represents the feature map after attention enhancement;

[0032] Use the training dataset of the original low-light image enhancement network to train the lightweight model of the improved low-light image enhancement network;

[0033] The training data set of the original low-light image enhancement network is a natural scene image low-light enhancement data set, which contains low-light images for network training;

[0034] Extract features from the attention-enhanced feature map using the low-light image trained by the network to obtain the enhanced low-light image during training;

[0035] Step S24: Improved spatial consistency loss function of low-light image enhancement network Calculate the gradient change difference between the corresponding adjacent pixels before and after the low-light image trained by the network and the low-light image enhanced during training, and accumulate and sum the squares of the gradient change difference; retain the smoothness of the gradient change between the corresponding front and back pixels, expressed as:

[0036] (5);

[0037] In the formula, represents the spatial consistency loss function; and They represent the gradient directions of the low-light images used for network training and the low-light images enhanced during training respectively; Represents the gradient difference between adjacent pixels on the left side in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the left side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels on the right in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the right side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above the enhanced low-light image during training; Represents the gradient difference between adjacent pixels above and below in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above and below the enhanced low-light image during training;

[0038] Step S25: input the endoscopic video frame image in step S1 into the trained lightweight model, enhance the endoscopic video frame image through the trained lightweight model, and obtain the endoscopic video frame image with preliminary brightness enhancement.

[0039] Further, in step S3, the color-corrected endoscope video frame image is obtained, and the specific process is as follows:

[0040] Step S31: Correct the preliminary brightness-enhanced endoscopic video frame image in step S2 according to the linear color of the endoscopic video frame image. The linear color includes three primary color spaces. The three primary color space information of the endoscopic video frame image and the preliminary brightness-enhanced endoscopic video frame image is counted. The preliminary brightness-enhanced endoscopic video frame image is color-corrected according to the three primary color space information statistics, so that the color of the preliminary brightness-enhanced endoscopic video frame image is closer to the endoscopic video frame image. The linear color correction formula is expressed as:

[0041] (6);

[0042] In the formula, represents the corrected initial brightness-enhanced endoscopic video frame image; represents the pixel value of the initial brightness enhancement endoscope video frame image; represents the mean value of the preliminary brightness enhancement endoscope video frame image; and denote the standard deviation of the preliminary brightness-enhanced endoscopic video frame image and the endoscopic video frame image, respectively; represents the mean value of the endoscope video frame image;

[0043] Step S32: Through linear color transformation, the brightness and color of the corrected preliminary brightness-enhanced endoscopic video frame image are adjusted so that the mean and standard deviation of the corrected preliminary brightness-enhanced endoscopic video frame image are aligned with the endoscopic video frame image to maintain color consistency, thereby obtaining a color-corrected endoscopic video frame image.

[0044] Further, in step S4, the final enhanced endoscope video frame image is obtained, and the specific process is as follows:

[0045] Step S41: Process the color-corrected endoscopic video frame image using a histogram equalization algorithm, image brightening, and detail enhancement to generate three sub-images; the three sub-images include ; Indicates the brightness layer of the sub-image generated by the histogram equalization algorithm; Represents the brightness layer of the sub-image generated by the image brightening method; Represents the brightness layer of the sub-image generated by the detail enhancement method;

[0046] Step S42: converting the three sub-images from the three primary color spaces into the HSI color space, where the HSI color space includes a brightness layer I, a hue layer H, and a saturation layer S;

[0047] Step S43: Generate the brightness layer of the sub-image using the histogram equalization algorithm Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; The brightness layer of the sub-image generated by the brightening image method Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; Generate the brightness layer of the sub-image using the detail enhancement method Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ;

[0048] Step S44: Use guided filtering to enhance the high-frequency components of the sub-images generated by the histogram equalization algorithm, the sub-images generated by the image brightening method, and the sub-images generated by the detail enhancement method in the horizontal direction. , vertical direction and diagonal direction , the process is:

[0049] Step S441: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components and the enhanced diagonal high-frequency components , expressed as:

[0050] + ---> + ---> + ---> (7);

[0051] Step S442: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as:

[0052] + ---> + ---> + ---> (8);

[0053] Step S443: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as:

[0054] + ---> + ---> + ---> (9);

[0055] Step S45: Take the low-frequency components of the three sub-images The maximum value of the middle pixel is the low-frequency component with the largest pixel value in the image, expressed as:

[0056] (10);

[0057] In the formula, Indicates the low-frequency component with the largest pixel value in the image; It means taking the maximum value of pixels in the low-frequency components of the three sub-images;

[0058] Step S46: The enhanced horizontal high-frequency component obtained in step S441 is , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , the enhanced horizontal high-frequency component obtained in step S442 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components and the enhanced horizontal high-frequency component obtained in step S443 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components The local contrast and local entropy are used for weighted fusion, and the process is:

[0059] Step S461: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0060] (11);

[0061] In the formula, Represents the enhanced horizontal high-frequency component The weight of and Parameters representing the weights of local contrast and local entropy respectively; preset as = =0.5;

[0062] Step S462: Optimize the enhanced horizontal high-frequency components The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0063] (12);

[0064] Step S463: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0065] (13);

[0066] Step S464: The obtained enhanced horizontal high-frequency component Weight , enhanced horizontal high-frequency components Weight And the enhanced horizontal high-frequency component Weight Normalization is performed, and then weighted fusion calculation is performed to obtain the final high-frequency component in the horizontal direction. , expressed as:

[0067] (14);

[0068] Step S465: Using the same steps as step S46, the enhanced vertical high-frequency components , enhanced vertical high-frequency components And the enhanced vertical high-frequency component Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final vertical high-frequency component ;

[0069] Step S466: Using the same steps as step S46, the enhanced diagonal high-frequency components , enhanced diagonal high-frequency components And the enhanced diagonal high-frequency components Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final diagonal high-frequency component ;

[0070] Step S47: The low-frequency component with the largest pixel value in the obtained image , the final horizontal high-frequency component , the final vertical high-frequency component And the final diagonal high frequency components Perform inverse wavelet transform to obtain the final enhanced brightness layer , the final enhanced brightness layer It is recombined with the brightness layer I, hue layer H and saturation layer S in the endoscopic video frame image, and converted back to the three original color spaces from the HSI color space to obtain the final enhanced endoscopic video frame image.

[0071] Compared with the existing technology, the present invention has the following beneficial effects:

[0072] (1) The present invention improves the lightweight model of the low-light image enhancement network and uses the low-light images of the network training of the natural scene image low-light enhancement SICE dataset for training. The training result weights are used to improve the overall image brightness of the endoscopic video frame image. Then, according to the linear color correction of the endoscopic video frame image, the color consistency is maintained. Finally, the histogram equalization algorithm (CLAHE), image brightening and detail enhancement, and sub-image generation are used. The high-frequency component and the low-frequency component are decomposed based on the wavelet transform. The edge of the high-frequency component is enhanced by guided filtering, and the dark detail texture information of the endoscopic video frame image is further improved. Based on the proposed maximum fusion rule and the weighted fusion rule of local contrast information and local entropy information, the high-frequency component and the low-frequency component are fused and enhanced, and the brightness of the endoscopic video frame image is further improved, and the overall quality of the endoscopic video frame image is improved.

[0073] (2) The present invention can achieve image enhancement processing of low-light natural scene images with good timeliness and output of endoscopic video through an improved lightweight model and linear color correction of endoscopic video frame images and endoscopic video frame image fusion enhancement technology based on wavelet transform and guided filter decomposition. It can be applied to endoscopic equipment with limited computing resources and meet clinical real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0075] like Figure 1 As shown, the present invention provides a technical solution: a zero-reference lightweight low-light endoscope video enhancement method, comprising the following steps:

[0076] Step S1: obtaining an endoscope video file, extracting the endoscope video file, obtaining an endoscope video frame image, and creating an output video frame folder;

[0077] Step S2: inputting the endoscopic video frame image extracted in step S1 into the improved low-light image enhancement network for processing to obtain a preliminary brightness-enhanced endoscopic video frame image;

[0078] Step S3: Correcting the preliminary brightness-enhanced endoscopic video frame image obtained in step S2 according to the linear color of the endoscopic video frame image to obtain a color-corrected endoscopic video frame image;

[0079] Step S4: The color-corrected endoscopic video frame image obtained in step S3 is processed by a histogram equalization algorithm, image brightening and detail enhancement to generate three sub-images, and the three sub-images are weightedly fused by using local contrast and local entropy to obtain a final enhanced endoscopic video frame image;

[0080] Step S5: placing the final enhanced endoscopic video frame image obtained in step S4 in an output video frame folder, and recombining the final enhanced endoscopic video frame image in the output video frame folder into a low-light enhanced endoscopic video through a video interpolation and recombination algorithm.

[0081] Among them, in step S1, the endoscope video frame image is obtained, and the specific process is:

[0082] The endoscopic video MP4 format collected during medical endoscopic examination is placed in a folder directory, and the endoscopic video file placed in the folder directory is extracted by a video key frame extraction algorithm to obtain an endoscopic video frame image.

[0083] Among them, in step S2, a preliminary brightness enhancement endoscope video frame image is obtained, and the specific process is as follows:

[0084] Step S21: The original code of the low-light image enhancement network (Zero-DCE) uses a standard convolution layer. The standard convolution layer in the improved low-light image enhancement network is replaced by a depthwise separable convolution layer. In the depthwise separable convolution layer, the number of parameters and the amount of calculation are reduced, which improves the efficiency of the low-light image enhancement network. The depthwise separable convolution layer includes depthwise convolution and pointwise convolution.

[0085] Step S22: Deep convolution is used to enhance each input channel of the low-light image network. Perform independent spatial convolution to extract channels Spatial feature information; then extract the channel through pointwise convolution 1x1 convolution fusion Spatial feature information, generate a new feature map; for the input endoscope video frame image (Dimensions are H×W u × ) The depth-wise separable convolutional layer is processed in two steps; H represents the height of the input endoscope video frame image; W u Represents the width of the input endoscope video frame image;

[0086] Each input channel is processed by depthwise convolution Perform independent spatial convolution, expressed as:

[0087] (1);

[0088] In the formula, Indicates that on each input channel The pixel value of the output image at the index position (i, j) above; and Respectively represent the index of the convolution kernel in the horizontal and vertical directions; Indicates channel The convolution kernel on ; Indicates that the input image is in channel The pixel value at the index position (i, j) above; Represents the index in the vertical direction (height) of the input image; Represents the index in the horizontal direction (width) of the input image;

[0089] Extract channels by point convolution and degree fusion Spatial feature information generates a new feature map, which is expressed as:

[0090] (2);

[0091] In the formula, Indicates that the new feature map is output at position The value of Represents the weight of 1x1 convolution;

[0092] Step S23: To generate a new feature map, a 1x1 convolutional layer is used to generate weights, which are then normalized by the activation function Sigmoid to calculate the new feature map. The attention weight of the pixel position is obtained by comparing the endoscope video frame image with the new feature map Multiply the attention weights of the pixel positions to obtain the feature map after attention enhancement;

[0093] Calculate the new feature map The attention weight at the pixel position is expressed as:

[0094] (3);

[0095] In the formula, Indicates the new feature map Attention weights at pixel locations; Represents the activation function Sigmoid; Represents a 1x1 convolutional layer; Indicates the endoscope video frame image Attention weights at pixel locations;

[0096] The feature map after attention enhancement is obtained, expressed as:

[0097] (4);

[0098] In the formula, Represents the feature map after attention enhancement;

[0099] Referring to the above improved low-light image enhancement network, the lightweight model of the improved low-light image enhancement network is trained using the training data set of the original low-light image enhancement network;

[0100] The training dataset of the original low-light image enhancement network is the SICE dataset for low-light enhancement of natural scene images, which contains low-light images for network training;

[0101] Further extract features from the attention-enhanced feature map using the low-light image trained by the network to obtain the enhanced low-light image during training;

[0102] In the original low-light image enhancement network, the color consistency loss function Improvements are made by calculating the local color difference. During the training process, by suppressing the deviation of enhanced colors, the improved low-light image enhancement network can better restore and enhance color information, thereby improving the enhancement effect and visual quality of subsequent images;

[0103] Spatial consistency loss function in the original low-light image enhancement network Improvements are made. During the training process, the gradient changes between the pixels before and after the low-light image trained by the network and the low-light image enhanced during training are calculated, and the gradient change spatial consistency loss function is used for subsequent loss calculation and performance evaluation to optimize the subsequent enhancement effect. The loss function in the low-light image enhancement network The improvements are as follows:

[0104] Step S24: Color consistency loss function of the original low-light image enhancement network In the , the local area division operation of the low-light image trained by the network is added. The low-light image trained by the network is divided into 16x16 local areas. The sum of squares of the differences between the three color channels R, G, and B are calculated in the local area, and finally the local color difference is obtained, which is expressed as:

[0105] ;

[0106] In the formula, Indicates local color differences; Represents the total number of pixels in the local area after division; , , Respectively represent the average values ​​of red, green, and blue channels in the local area;

[0107] Step S25: Spatial consistency loss function of the original low-light image enhancement network In the process, the gradient change between the corresponding pixels before and after the low-light image trained by the network and the low-light image enhanced during training is calculated through the convolution kernel, which increases the gradient change control within the neighborhood of the original low-light image enhancement network; the spatial consistency loss function of the improved low-light image enhancement network The gradient change difference between the corresponding adjacent pixels before and after the low-light image trained by the network and the low-light image enhanced during training is calculated, and the square of the gradient change difference is accumulated and summed; the smoothness of the gradient change between the corresponding front and back pixels is retained, which is expressed as:

[0108] (5);

[0109] In the formula, Represents the spatial consistency loss function, which is used to measure the consistency of the gradient direction before and after image enhancement; and They represent the gradient directions of the low-light images used for network training and the low-light images enhanced during training respectively; Represents the gradient difference between adjacent pixels on the left side in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the left side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels on the right in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the right side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above the enhanced low-light image during training; Represents the gradient difference between adjacent pixels above and below in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above and below the enhanced low-light image during training;

[0110] Step S26: input the endoscopic video frame image in step S1 into the trained lightweight model, enhance the endoscopic video frame image through the trained lightweight model, and obtain the endoscopic video frame image with preliminary brightness enhancement.

[0111] In step S3, the color-corrected endoscope video frame image is obtained, and the specific process is as follows:

[0112] Step S31: The preliminary brightness-enhanced endoscopic video frame image in step S2 is corrected according to the linear color of the endoscopic video frame image. The linear color includes three primary color (RGB) space. The three primary color space information of the endoscopic video frame image and the preliminary brightness-enhanced endoscopic video frame image is counted. The preliminary brightness-enhanced endoscopic video frame image is color-corrected according to the three primary color space information statistics, so that the color of the preliminary brightness-enhanced endoscopic video frame image is closer to the endoscopic video frame image. The linear color correction formula is expressed as:

[0113] (6);

[0114] In the formula, represents the corrected initial brightness-enhanced endoscopic video frame image; represents the pixel value of the initial brightness enhancement endoscope video frame image; represents the mean value of the preliminary brightness enhancement endoscope video frame image; and denote the standard deviation of the preliminary brightness-enhanced endoscopic video frame image and the endoscopic video frame image, respectively; represents the mean value of the endoscope video frame image;

[0115] Step S32: adjusting the brightness and color of the corrected preliminary brightness-enhanced endoscopic video frame image through linear color transformation, so that the mean and standard deviation of the corrected preliminary brightness-enhanced endoscopic video frame image are aligned with the endoscopic video frame image to maintain color consistency; obtaining a color-corrected endoscopic video frame image.

[0116] The final enhanced endoscope video frame image is obtained in step S4, and the specific process is as follows:

[0117] Step S41: Process the color-corrected endoscopic video frame image using a histogram equalization algorithm (CLAHE), image brightening, and detail enhancement to generate three sub-images; the three sub-images include ; Indicates the brightness layer of the sub-image generated by the histogram equalization algorithm; Represents the brightness layer of the sub-image generated by the image brightening method; Represents the brightness layer of the sub-image generated by the detail enhancement method;

[0118] Step S42: converting the three sub-images from the three primary color spaces into the HSI color space, where the HSI color space includes a brightness layer I, a hue layer H, and a saturation layer S; wherein the hue layer H and the saturation layer S remain unchanged;

[0119] Step S43: Generate the brightness layer of the sub-image using the histogram equalization algorithm Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; The brightness layer of the sub-image generated by the brightening image method Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; Generate the brightness layer of the sub-image using the detail enhancement method Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction , the process is:

[0120] Step S44: Use guided filtering to enhance the high-frequency components of the sub-images generated by the histogram equalization algorithm, the sub-images generated by the image brightening method, and the sub-images generated by the detail enhancement method in the horizontal direction. , vertical direction and diagonal direction , to highlight the edges of high-frequency components in three different directions, thereby enhancing the dark detail texture information of the three sub-images;

[0121] Step S441: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components And the enhanced diagonal high-frequency components , expressed as:

[0122] + ---> + ---> + ---> (7);

[0123] Step S442: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as:

[0124] + ---> + ---> + ---> (8);

[0125] Step S443: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as:

[0126] + ---> + ---> + ---> (9);

[0127] Step S45: Take the low-frequency components of the three sub-images The maximum value of the middle pixel is the low-frequency component with the largest pixel value in the image, expressed as:

[0128] (10);

[0129] In the formula, Indicates the low-frequency component with the largest pixel value in the image; It means taking the maximum value of pixels in the low-frequency components of the three sub-images;

[0130] Step S46: The enhanced horizontal high-frequency component obtained in step S441 is , enhanced vertical high-frequency components And the enhanced diagonal high-frequency components , the enhanced horizontal high-frequency component obtained in step S442 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components and the enhanced horizontal high-frequency component obtained in step S443 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components The local contrast and local entropy are used for weighted fusion, and the process is:

[0131] Step S461: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0132] (11);

[0133] In the formula, Represents the enhanced horizontal high-frequency component The weight of and Parameters representing the weights of local contrast and local entropy respectively; preset as = =0.5;

[0134] Step S462: Optimize the enhanced horizontal high-frequency components The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0135] (12);

[0136] Step S463: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as:

[0137] (13);

[0138] Step S464: The obtained enhanced horizontal high-frequency component Weight , enhanced horizontal high-frequency components Weight And the enhanced horizontal high-frequency component Weight Normalization is performed, and then weighted fusion calculation is performed to obtain the final high-frequency component in the horizontal direction. , expressed as:

[0139] (14);

[0140] Step S465: Using the same steps as step S46, the enhanced vertical high-frequency components , enhanced vertical high-frequency components And the enhanced vertical high-frequency component Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final vertical high-frequency component ;

[0141] Step S466: Using the same steps as step S46, the enhanced diagonal high-frequency components , enhanced diagonal high-frequency components and the enhanced diagonal high-frequency components Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final diagonal high-frequency component ;

[0142] Step S47: The low-frequency component with the largest pixel value in the obtained image , the final horizontal high-frequency component , the final vertical high-frequency component And the final diagonal high frequency components Perform inverse wavelet transform to obtain the final enhanced brightness layer , the final enhanced brightness layer It is recombined with the brightness layer I, hue layer H and saturation layer S in the endoscopic video frame image, and converted back to the three original color spaces from the HSI color space to obtain the final enhanced endoscopic video frame image.

[0143] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A zero-reference lightweight low-light endoscopic video enhancement method, characterized in that: The following steps are involved: Step S1: obtaining an endoscope video file, extracting the endoscope video file, obtaining an endoscope video frame image, and creating an output video frame folder; Step S2: inputting the endoscopic video frame image extracted in step S1 into the improved low-light image enhancement network for processing to obtain a preliminary brightness-enhanced endoscopic video frame image; The low-light image enhancement network includes a standard convolutional layer. The standard convolutional layer in the improved low-light image enhancement network is replaced by a depthwise separable convolutional layer. The depthwise separable convolutional layer includes depthwise convolution and pointwise convolution. The low-light image enhancement network is processed through deep convolution to extract channel spatial feature information; Then, the channel space feature information is extracted through point convolution fusion to generate a new feature map; For generating a new feature map, a convolution layer generates weights, which are then normalized by an activation function, and the attention weights of the pixel positions in the new feature map are calculated. The endoscope video frame image is multiplied by the attention weights of the pixel positions in the new feature map to obtain the feature map with enhanced attention. The lightweight model of the improved low-light image enhancement network is trained based on the low-light image enhancement network and the feature map after attention enhancement; Inputting the endoscopic video frame image in step S1 into the trained lightweight model, enhancing the endoscopic video frame image by the trained lightweight model, and obtaining an endoscopic video frame image with preliminary brightness enhancement; Step S3: Correcting the preliminary brightness-enhanced endoscopic video frame image obtained in step S2 according to the linear color of the endoscopic video frame image to obtain a color-corrected endoscopic video frame image; Step S4: The color-corrected endoscopic video frame image obtained in step S3 is processed by a histogram equalization algorithm, image brightening and detail enhancement to generate three sub-images, and the three sub-images are weightedly fused by using local contrast and local entropy to obtain a final enhanced endoscopic video frame image; Step S5: placing the final enhanced endoscopic video frame image obtained in step S4 in an output video frame folder, and recombining the final enhanced endoscopic video frame image in the output video frame folder into a low-light enhanced endoscopic video through a video interpolation and recombination algorithm.

2. The zero-reference lightweight low-light endoscope video enhancement method according to claim 1, characterized in that: In step S1, an endoscope video frame image is obtained, and the specific process is as follows: The endoscopic video collected during the medical endoscopic examination is placed in a folder directory, and the endoscopic video placed in the folder directory is extracted by a video key frame extraction algorithm to obtain an endoscopic video frame image.

3. The zero-reference lightweight low-light endoscopic video enhancement method according to claim 2, characterized in that: In step S2, a preliminary brightness-enhanced endoscope video frame image is obtained, and the specific process is as follows: Step S21: the original code of the low-light image enhancement network uses a standard convolution layer, and the standard convolution layer in the improved low-light image enhancement network is replaced by a depthwise separable convolution layer; the depthwise separable convolution layer includes depthwise convolution and pointwise convolution; Step S22: Enhance each input channel in the low-light image network through deep convolution Perform independent spatial convolution to extract channels Spatial feature information; Then extract the channel through point 1x1 convolution fusion Spatial feature information, generate a new feature map; for the input endoscope video frame image Dimensions: H x W u × The depthwise separable convolutional layer is processed in two steps; H represents the height of the input endoscope video frame image; W u Represents the width of the input endoscope video frame image; Each input channel is processed by depthwise convolution Perform independent spatial convolution, expressed as: （1）； In the formula, Indicates that on each input channel The pixel value of the output image at the index position (i, j) above; and Respectively represent the index of the convolution kernel in the horizontal and vertical directions; Indicates channel The convolution kernel on ; Indicates that the input endoscope video frame image is in channel The pixel value at the index position (i, j) above; Represents the index in the vertical direction of the input endoscope video frame image; Represents the index on the horizontal width of the input image; Extract channels by point convolution and degree fusion Spatial feature information generates a new feature map, which is expressed as: （2）； In the formula, Indicates that the new feature map is output at position The value of Represents the weight of 1x1 convolution; Step S23: To generate a new feature map, a 1x1 convolutional layer is used to generate weights, which are then normalized using an activation function to calculate the new feature map. The attention weight of the pixel position is obtained by comparing the endoscope video frame image with the new feature map Multiply the attention weights of the pixel positions to obtain the feature map after attention enhancement; Calculate the new feature map The attention weight at the pixel position is expressed as: （3）； In the formula, Indicates the new feature map Attention weights at pixel locations; represents the activation function; Represents a 1x1 convolutional layer; Indicates the endoscope video frame image Attention weights at pixel locations; The feature map after attention enhancement is obtained, expressed as: （4）； In the formula, Represents the feature map after attention enhancement; Use the training dataset of the original low-light image enhancement network to train the lightweight model of the improved low-light image enhancement network; The training data set of the original low-light image enhancement network is a natural scene image low-light enhancement data set, which contains low-light images for network training; Extract features from the attention-enhanced feature map using the low-light image trained by the network to obtain the enhanced low-light image during training; Step S24: Improved spatial consistency loss function of low-light image enhancement network Calculate the gradient change difference between the corresponding adjacent pixels before and after the low-light image trained by the network and the low-light image enhanced during training, and accumulate and sum the squares of the gradient change difference; retain the smoothness of the gradient change between the corresponding front and back pixels, expressed as: （5）； In the formula, represents the spatial consistency loss function; and They represent the gradient directions of the low-light images used for network training and the low-light images enhanced during training respectively; Represents the gradient difference between adjacent pixels on the left side in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the left side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels on the right in the horizontal direction of the low-light image for network training; Represents the gradient difference between adjacent pixels on the right side of the enhanced low-light image in training; Represents the gradient difference between adjacent pixels in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above the enhanced low-light image during training; Represents the gradient difference between adjacent pixels above and below in the vertical direction of the low-light image for network training; Represents the gradient difference between adjacent pixels in the vertical direction above and below the enhanced low-light image during training; Step S25: input the endoscopic video frame image in step S1 into the trained lightweight model, enhance the endoscopic video frame image through the trained lightweight model, and obtain the endoscopic video frame image with preliminary brightness enhancement.

4. The zero-reference lightweight low-light endoscopic video enhancement method according to claim 3, characterized in that: In step S3, the color-corrected endoscope video frame image is obtained, and the specific process is as follows: Step S31: Correct the preliminary brightness-enhanced endoscopic video frame image in step S2 according to the linear color of the endoscopic video frame image. The linear color includes three primary color spaces. The three primary color space information of the endoscopic video frame image and the preliminary brightness-enhanced endoscopic video frame image is counted. The preliminary brightness-enhanced endoscopic video frame image is color-corrected according to the three primary color space information statistics, so that the color of the preliminary brightness-enhanced endoscopic video frame image is closer to the endoscopic video frame image. The linear color correction formula is expressed as: （6）； In the formula, represents the corrected initial brightness-enhanced endoscopic video frame image; represents the pixel value of the initial brightness enhancement endoscope video frame image; represents the mean value of the preliminary brightness enhancement endoscope video frame image; and denote the standard deviation of the preliminary brightness-enhanced endoscopic video frame image and the endoscopic video frame image, respectively; represents the mean value of the endoscope video frame image; Step S32: Through linear color transformation, the brightness and color of the corrected preliminary brightness-enhanced endoscopic video frame image are adjusted so that the mean and standard deviation of the corrected preliminary brightness-enhanced endoscopic video frame image are aligned with the endoscopic video frame image to maintain color consistency, thereby obtaining a color-corrected endoscopic video frame image.

5. The zero-reference lightweight low-light endoscopic video enhancement method according to claim 4, characterized in that: In step S4, the final enhanced endoscope video frame image is obtained, and the specific process is as follows: Step S41: Process the color-corrected endoscopic video frame image using a histogram equalization algorithm, image brightening, and detail enhancement to generate three sub-images; the three sub-images include ; Indicates the brightness layer of the sub-image generated by the histogram equalization algorithm; Represents the brightness layer of the sub-image generated by the image brightening method; Represents the brightness layer of the sub-image generated by the detail enhancement method; Step S42: converting the three sub-images from the three primary color spaces into the HSI color space, where the HSI color space includes a brightness layer I, a hue layer H, and a saturation layer S; Step S43: Generate the brightness layer of the sub-image using the histogram equalization algorithm Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; The brightness layer of the sub-image generated by the brightening image method Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; The detail enhancement method generates the brightness layer of the sub-image Perform wavelet transform to decompose into low frequency components and the horizontal direction of the high frequency component , vertical direction and diagonal direction ; Step S44: Use guided filtering to enhance the high-frequency components of the sub-images generated by the histogram equalization algorithm, the sub-images generated by the image brightening method, and the sub-images generated by the detail enhancement method in the horizontal direction. , vertical direction and diagonal direction , the process is: Step S441: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components And the enhanced diagonal high-frequency components , expressed as: + ---> + ---> + ---> （7）； Step S442: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as: + ---> + ---> + ---> （8）； Step S443: Brightness layer As input image, the horizontal , vertical direction and diagonal direction As a guide image, the horizontal , vertical direction and diagonal direction Perform guided filtering enhancement to obtain the enhanced horizontal high-frequency component , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , expressed as: + ---> + ---> + ---> （9）； Step S45: Take the low-frequency components of the three sub-images The maximum value of the middle pixel is the low-frequency component with the largest pixel value in the image, expressed as: （10）； In the formula, Indicates the low-frequency component with the largest pixel value in the image; It means taking the maximum value of pixels in the low-frequency components of the three sub-images; Step S46: The enhanced horizontal high-frequency component obtained in step S441 is , enhanced vertical high-frequency components , enhanced diagonal high-frequency components , the enhanced horizontal high-frequency component obtained in step S442 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components and the enhanced horizontal high-frequency component obtained in step S443 , enhanced vertical high-frequency components , enhanced diagonal high-frequency components The local contrast and local entropy are used for weighted fusion, and the process is: Step S461: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as: （11）； In the formula, Represents the enhanced horizontal high-frequency component The weight of and Parameters representing the weights of local contrast and local entropy respectively; preset as = =0.5; Step S462: Optimize the enhanced horizontal high-frequency components The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as: （12）； Step S463: The enhanced horizontal high-frequency component The local contrast is calculated between each 3×3 neighborhood center cell and the surrounding cells to determine the local contrast information ; At the same time, the enhanced horizontal high-frequency component The local entropy information is determined by calculating the local entropy between the central cell and the surrounding cells in each 3×3 neighborhood in the three sub-images. , by using local contrast information and local entropy information Weighted fusion calculation, we get , expressed as: （13）； Step S464: The obtained enhanced horizontal high-frequency component Weight , enhanced horizontal high-frequency components Weight And the enhanced horizontal high-frequency component Weight Normalization is performed, and then weighted fusion calculation is performed to obtain the final high-frequency component in the horizontal direction. , expressed as: （14）； Step S465: Using the same steps as step S46, the enhanced vertical high-frequency components , enhanced vertical high-frequency components And the enhanced vertical high-frequency component Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final vertical high-frequency component ; Step S466: Using the same steps as step S46, the enhanced diagonal high-frequency components , enhanced diagonal high-frequency components And the enhanced diagonal high-frequency components Perform weighted fusion calculation of local contrast information and local entropy information to obtain the final diagonal high-frequency component ; Step S47: The low-frequency component with the largest pixel value in the obtained image , the final horizontal high-frequency component , the final vertical high-frequency component And the final diagonal high frequency components Perform inverse wavelet transform to obtain the final enhanced brightness layer , the final enhanced brightness layer It is recombined with the brightness layer I, hue layer H and saturation layer S in the endoscopic video frame image, and converted back to the three original color spaces from the HSI color space to obtain the final enhanced endoscopic video frame image.

Images