A low-light video enhancement method and system for endoscopic imaging

Abstract

The application relates to the technical field of video enhancement, and discloses a weak light video enhancement method and system for endoscope imaging, which comprises the following steps: video stream is disassembled into image frames; the image frames are mapped from an RGB space to an HVI space, and decoupled into a color graph and an intensity graph; the color graph and the intensity graph are denoised and enhanced to obtain optimized color features and intensity features; the color features and the intensity features are interacted and fused in a light manner to obtain primary fusion features; the primary fusion features are optimized at a channel level to obtain secondary fusion features; the secondary fusion features are inversely converted from the HVI space to the RGB space to obtain enhanced single-frame images; and the continuous enhanced single-frame images are spliced into an enhanced video stream and output. The method of the application realizes model light weighting while ensuring the quality of weak light video enhancement, and meets the frame rate requirement and quality requirement of real-time video processing of an endoscope.

Description

Technical Field

[0001] This invention relates to the field of video enhancement, and more particularly to a low-light video enhancement method and system for endoscopic imaging. Background Technology

[0002] Endoscopes are commonly used medical devices in clinical practice, capable of acquiring visual information about cavities such as the gastrointestinal tract and abdominal cavity through image sensors. However, in clinical practice, due to insufficient lighting, changes in distance, and interference from body fluids, the acquired videos often suffer from low brightness, poor contrast, and blurred details. The subtle textures of intracavitary tissues and early lesions are easily masked by noise. Furthermore, lens movement and organ peristalsis can cause inter-frame motion blur and uneven brightness, seriously affecting the efficiency and accuracy of doctors' work.

[0003] Existing methods for enhancing low-light endoscopic videos can be broadly categorized into traditional image processing methods and deep learning-based methods. Traditional methods include Retine theory, histogram equalization, and filtering / denoising. While based on manually designed prior rules and possessing simple computational logic, traditional methods struggle to adapt to the complex lighting environment within human cavities, often resulting in overexposure, underexposure, color distortion, and amplified noise. Deep learning-based methods, on the other hand, train neural networks to learn the mapping relationship between low-light and normal-light images. They significantly outperform traditional methods in terms of detail preservation, noise suppression, and color restoration, more accurately restoring the true features of intracavitary tissues and improving the identifiability of lesions.

[0004] However, deep learning-based methods, in pursuit of enhanced video quality, often employ complex Transformer-based network structures. These models have a large number of parameters and high computational complexity, resulting in high hardware requirements. Consequently, they often exhibit low frame rates when processing real-time video, leading to stuttering and unsmooth playback, which fails to meet the real-time requirements of endoscopic clinical procedures. Therefore, there is an urgent need to design a low-light video enhancement method for endoscopic imaging that can guarantee video enhancement quality while achieving lightweight models, ensuring smooth real-time video processing, and balancing the needs of clinical diagnostic accuracy and surgical real-time performance. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a low-light video enhancement method and system for endoscopic imaging, which solves the problem mentioned in the background art that most existing deep learning-based methods adopt complex network structures in order to pursue video enhancement quality, resulting in a large number of model parameters and high computational complexity.

[0006] The present invention solves the above-mentioned technical problems through the following technical means:

[0007] In a first aspect, a low-light video enhancement method for endoscopic imaging is provided, the method comprising:

[0008] The video stream inside the human cavity is acquired in real time through an endoscope, and the video stream is decomposed into independent image frames, which are RGB format data.

[0009] The image frame is mapped from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map;

[0010] The color and intensity maps are denoised and enhanced using a VMamba-based dual-branch U-Net network to obtain optimized color and intensity features.

[0011] The color feature and the intensity feature are fused through a lightweight interaction using an interactive fusion module to obtain a first-level fused feature.

[0012] The primary fusion feature is optimized at the channel level by the squeezing excitation module to obtain the secondary fusion feature;

[0013] The secondary fusion features are inversely transformed from HVI space to RGB space using the HVI transformation module to obtain an enhanced single-frame image.

[0014] The continuous enhanced single-frame images are stitched together to form an enhanced video stream, and the enhanced video stream is output to a display device.

[0015] In one possible implementation, the VMamba-based dual-branch U-Net network includes two independent U-Net branches, which respectively denoise and enhance the color map and the intensity map. Each U-Net branch includes an encoder, a VMamba module, and a decoder, with the VMamba module located between the encoder and the decoder.

[0016] In one possible implementation, the encoder includes several downsampling layers, whereby after a feature passes through the downsampling layers, the number of channels remains unchanged, and the width and height are both 0.5 times that of the original feature.

[0017] The decoder includes several upsampling layers. After a feature passes through the upsampling layers, the number of channels remains unchanged, while the width and height are twice that of the original feature.

[0018] The features output by each downsampling layer are added element-wise to the features output by the upsampling layer at the corresponding scale of the decoder to form skip connections.

[0019] In one possible implementation, the U-Net branch further includes a dimension-upgrading module disposed before the encoder. The dimension-upgrading module is a 3×3 convolutional layer, and the number of channels increases after the feature passes through the dimension-upgrading module.

[0020] In one possible implementation, the interactive fusion module includes a bidirectional interactive layer and a fusion layer;

[0021] Through the bidirectional interaction layer, the color feature is added to the weighted intensity feature in an element-wise addition manner to obtain the color interaction feature, and the intensity feature is added to the weighted color feature in an element-wise addition manner to obtain the intensity interaction feature;

[0022] Through the fusion layer, the color interaction feature and the intensity interaction feature are fused by channel splicing to obtain a first-level fused feature.

[0023] In one possible implementation, the interactive fusion module further includes a channel recovery layer, which is a 3×3 convolutional layer. Through the channel recovery layer, the number of channels of the first-level fusion feature is reduced by half.

[0024] In one possible implementation, the squeeze excitation module includes a feature gating module, a depth-separable convolutional layer, and an optimization layer;

[0025] After the primary fusion feature passes through the feature gating module, the channel-level weights corresponding to the primary fusion feature are extracted.

[0026] The first-level fusion features are passed through the depth-separable convolutional layer to obtain convolutional features;

[0027] Through the optimization layer, the channel-level weights and the convolutional features are multiplied element-wise to obtain the secondary fusion features.

[0028] In one possible implementation, the first-level fusion feature is fused to the second-level fusion feature via residual connections through the optimization layer.

[0029] In one possible implementation, the feature gating module includes an adaptive global average pooling layer, a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence.

[0030] On the other hand, a low-light video enhancement system for endoscopic imaging is provided, comprising:

[0031] The preprocessing unit is used to acquire video streams inside the human cavity in real time through an endoscope and decompose the video stream into independent image frames, wherein the image frames are RGB format data;

[0032] The mapping unit is used to map the image frame from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map.

[0033] The denoising and enhancement unit is used to denoise and enhance the color map and intensity map through a VMamba-based dual-branch U-Net network to obtain optimized color features and intensity features;

[0034] The fusion unit is used to perform lightweight interaction and fusion of the color feature and the intensity feature through the interactive fusion module to obtain a first-level fused feature;

[0035] The optimization unit is used to perform channel-level optimization on the first-level fusion feature through the squeezing excitation module to obtain the second-level fusion feature;

[0036] The inverse transformation unit is used to inversely transform the secondary fusion features from HVI space to RGB space through the HVI transformation module to obtain an enhanced single-frame image.

[0037] The output unit is used to stitch consecutive enhanced single-frame images into an enhanced video stream and output the enhanced video stream to a display device.

[0038] The beneficial effects of this application are:

[0039] This method employs techniques such as RGB-HVI spatial decoupling, a VMamba-based dual-branch U-Net network, lightweight feature interaction fusion, and channel-level attention optimization to achieve lightweight modeling while ensuring the quality of low-light video enhancement, thus meeting the frame rate and quality requirements of real-time endoscopic video processing. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0041] Figure 1 This is a framework diagram of the deep learning network in the embodiments of this application;

[0042] Figure 2 This is a comparison chart of the visualization results in the embodiments of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0044] Furthermore, the use of terms such as "first," "second," etc., in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0045] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0046] like Figure 1 As shown, this application provides a low-light video enhancement method for endoscopic imaging. Addressing the problems of low brightness, poor contrast, blurred details, and the large number of parameters and high computational complexity of existing deep learning models in clinically acquired endoscopic videos, an end-to-end deep learning network is constructed. According to the processing flow of this network, the low-light video enhancement method specifically includes the following steps:

[0047] S100. Real-time acquisition of video stream inside the human cavity via endoscope, and decomposition of the video stream into independent image frames, wherein the image frames are RGB format data;

[0048] S200: The image frame is mapped from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map;

[0049] S300: The color map and intensity map are denoised and enhanced by a VMamba-based dual-branch U-Net network to obtain optimized color features and intensity features;

[0050] S400. The color feature and the intensity feature are lightweighted and fused through the interactive fusion module to obtain a first-level fused feature;

[0051] S500. The first-level fusion feature is optimized at the channel level by the squeezing excitation module to obtain the second-level fusion feature;

[0052] S600: The secondary fusion features are inversely transformed from HVI space to RGB space through the HVI conversion module to obtain an enhanced single-frame image;

[0053] S700: The continuous enhanced single-frame images are stitched together to form an enhanced video stream, and the enhanced video stream is output to the display device.

[0054] This method first acquires real-time video streams of the human gastrointestinal tract, abdominal cavity, and other cavities using the image sensor of an endoscope. Then, it performs frame-by-frame enhancement processing on the video streams and stitches the enhanced single-frame images together to output a real-time enhanced video stream to a display device. In the image processing stage, this method employs techniques such as RGB-HVI spatial decoupling, a VMamba-based dual-branch U-Net network, lightweight feature interactive fusion, and channel-level attention optimization to achieve both enhancement quality and model lightweighting, meeting the frame rate and quality requirements of real-time endoscopic video processing.

[0055] The following is a detailed explanation of each step described above and each module in the network:

[0056] In step S100, the endoscopic device acquires a real-time video stream of the human cavity through an image sensor integrated on the endoscope. This video stream is in RGB format. The video stream is then preprocessed, breaking it down into individual image frames, also in RGB format. To ensure better computation of individual frames by the subsequent network framework, the pixel values ​​of each frame are normalized.

[0057] In step S200, the HVI conversion module maps the single-frame image in RGB format from RGB space to HVI space, thereby decoupling image features and decomposing the image frame into a color map and an intensity map. It should be noted that this application chooses the HVI space instead of other color spaces (such as YUV, HSV, etc.) primarily because the HVI space is a learnable color space, better suited to the characteristics of deep learning networks. Furthermore, deep learning networks using this color space can effectively eliminate red and black artifacts in low-light images. Since low-light videos or images inside human cavities often contain numerous red and black artifacts, this color space is highly suitable for video scenarios captured by endoscopes.

[0058] The conversion from RGB space to HVI space uses HSV space as an intermediate medium. The conversion principle of the HVI conversion module is explained in detail below:

[0059] (1) Calculate the intensity component using the following formula:

[0060]

[0061] Where x represents a single pixel in the image, This represents the pixel value of pixel x in RGB channel c (c=R / G / B). This represents the Max-RGB intensity value of pixel x, with a value range consistent with the input RGB, corresponding to the V component of HSV.

[0062] (2) Calculate the saturation S of the HSV space using the following formula:

[0063]

[0064]

[0065] Where s represents the saturation value of a single pixel, while S represents the saturation value of the entire image. This represents the extreme value difference of the RGB three channels of a pixel. This indicates that the pixel is pure black, and saturation is meaningless.

[0066] (3) Calculate the hue H in the HSV space using the following formula:

[0067]

[0068] Where h represents the original hue value of a single pixel, while H represents the hue value of the entire image. These represent the RGB three-channel pixel values ​​of a pixel.

[0069] (4) The hue H in the HSV space is polarized to suppress red discontinuity noise, as shown in the following formula:

[0070]

[0071]

[0072] Where, h on the left side of the equal sign represents the horizontal hue component after polarization, and h on the right side represents the single hue component.

[0073] The original hue value of each pixel, where v represents the polarized vertical hue component.

[0074] (5) Introduce a learnable parameter k to construct an intensity folding function, compress the pixel radius of the low-light region to 0, and gradually expand the highlight region to 1, thereby realizing pixel clustering in the low-light region. The formula is as follows:

[0075]

[0076] in, represents the intensity folding value of pixel x, with a value range of [0,1]. k represents a learnable positive rational number parameter that controls the density of color points in the low-light region. It is adaptively optimized by network training. ε is 1×10-8 to avoid gradient explosion after the denominator is 0 or the square root is taken.

[0077] (6) Generate the horizontal and vertical color components of the HVI color space using the following formula:

[0078]

[0079] in, Represents the horizontal color components of the HVI color space. Represents the vertical color components of the HVI color space. This indicates element-wise multiplication, and S represents the saturation value of the entire image.

[0080] (7) The three components are combined into an HVI by channel splicing, as shown in the following formula:

[0081]

[0082] In HVI, H and V represent decoupled color information, and I represents decoupled intensity information. The H and V channels correspond to the color map in the network, and I corresponds to the intensity map in the network.

[0083] Through the above decoupling process, the HVI conversion module completely decouples the color and intensity information in the RGB space, reducing the mutual interference between color and brightness during enhancement; at the same time, it introduces a learnable parameter k to adaptively optimize the degree of folding in the low-light region, adapting to low-light scenes with different lighting conditions; it reduces red noise artifacts in low-light scenes through polarization, and reduces black noise artifacts in low-light scenes through the intensity folding function.

[0084] In step S300, the VMamba-based dual-branch U-Net network includes two independent U-Net branches. The two U-Net branches perform denoising and enhancement processing on the color map and intensity map, respectively, to obtain optimized color features and intensity features. Each U-Net branch includes a dimension-upgrading module, an encoder, a VMamba module, and a decoder connected in sequence, with the VMamba module located between the encoder and the decoder.

[0085] The dimensionality-upgrading module is located at the input of the U-Net branch. This module is a 3×3 convolutional layer used to upgrade the channels of the input color map / intensity map, thereby improving the network's feature extraction capabilities. For the color map, the original number of channels is 2; after the dimensionality-upgrading module, the number of channels in the color map increases from 2 to C. For the intensity map, the original number of channels is 1; after the dimensionality-upgrading module, the number of channels in the intensity map increases from 1 to C. It is understood that the aforementioned C is a user-defined parameter, mainly to ensure the network's feature extraction capabilities. To maintain the network's lightweight nature, the value of C does not need to be set too large; the range of C is controlled between 2⁸ and 42, and in this embodiment, 32 is preferred.

[0086] The encoder consists of three downsampling layers connected sequentially. Each downsampling layer includes a 3×3 convolutional layer and a ReLU activation layer. The stride of the 3×3 convolutional layer is 2, and the padding is 1. After the features pass through the 3×3 convolutional layer and the ReLU activation layer, the number of channels remains unchanged, while the width and height are reduced to 0.5 times the size of the original features, thus achieving feature downsampling and spatial feature extraction.

[0087] The decoder consists of three upsampling layers connected sequentially. Each upsampling layer uses the nearest neighbor interpolation method to restore the resolution of the features. After passing through the upsampling layer, the number of channels remains unchanged, while the width and height are doubled compared to the original features, achieving feature upsampling and spatial feature recovery.

[0088] To prevent the loss of detail during feature upsampling and reconstruction, the features output from each downsampling layer are element-wise added to the features output from the corresponding upsampling layer of the decoder, forming skip connections. These skip connections compensate for the spatial information loss during downsampling, improving the ability to restore feature details. It should be noted that the element-wise addition skip connections between the encoder and decoder, compared to the traditional skip connection channel concatenation method, further reduce computational complexity and improve the model's inference speed while preserving feature details, thus meeting the frame rate requirements of real-time endoscopic video processing.

[0089] It should be noted that in conventional U-Net, not only does the resolution change during upsampling or downsampling, but the number of channels also changes; the number of channels doubles during downsampling and decreases during upsampling. However, in this embodiment, to minimize the number of parameters while maximizing the feature processing performance of U-Net, the number of channels remains constant at C during both upsampling and downsampling.

[0090] A VMamba module is embedded between the encoder and decoder. This module is a lightweight feature extraction module based on the Selective State Space Model (SSM). The core of the module achieves feature modeling through four steps: input projection, linear state space encoding, gated activation, and output fusion. After mapping the input feature map into channel-dimensional sequence features, it completes the encoding learning of long-distance spatial dependencies in linear time complexity (O(N)). At the same time, it preserves the fine-grained texture features of the image through local spatial decoupling design. Finally, it restores the encoded sequence features to the original-size feature map, which takes into account both global feature capture and local detail preservation, and its computational efficiency is far superior to that of traditional attention modules.

[0091] Most existing technologies embed a multi-head self-attention module (MHSA) between the encoder and decoder to improve U-Net performance. This module is based on Transformer, has a large number of parameters, and has a computational complexity of quadratic order (O(N²)). In this embodiment, the VMamba module is used instead of the MHSA module, which not only significantly reduces the number of model parameters and inference time, but also improves the processing speed of single-frame images while ensuring the enhancement effect, meeting the clinical requirements of real-time endoscopic imaging. Furthermore, the global attention distribution of MHSA tends to dilute the local lesion texture of endoscopic images and amplify low-light noise, while VMamba can accurately capture long-distance spatial correlation features of images, while effectively preserving local details such as mucosal blood vessels and small polyps. It can also adaptively suppress low-light noise through the selective gating mechanism of state space, which greatly improves the detail restoration and noise suppression effect of endoscopic low-light images.

[0092] In step 400, the interactive fusion module includes a bidirectional interactive layer, a fusion layer and a channel recovery layer connected in sequence. Through the function of each layer, lightweight information interaction and fusion of color features and intensity features are realized.

[0093] It is understandable that color and intensity features in the same image have a strong dependency. While decoupling and processing these two types of information separately can effectively enhance brightness and restore color, the lack of information interaction between the two features during processing may result in unnatural brightness and color restoration. Therefore, this embodiment employs a bidirectional interaction layer to achieve lightweight interaction between color and intensity features.

[0094] Through the bidirectional interaction layer, color features are summed element-wise with weighted intensity features to obtain color interaction features, and intensity features are summed element-wise with weighted color features to obtain intensity interaction features, as shown in the following formulas:

[0095]

[0096]

[0097] in, Represents the original color characteristics, Represents the original intensity characteristics. Indicates color interaction features, This represents the intensity interaction feature, where α and β are weighting coefficients with values ​​ranging from 0.1 to 0.2.

[0098] Through the fusion layer, color interaction features and intensity interaction features are fused using channel concatenation to obtain a first-level fused feature, achieving complete fusion of the two features. The feature dimension after concatenation is the sum of the number of channels, with the size remaining unchanged, as shown in the following formula:

[0099]

[0100] in, This represents a first-level fusion feature, whose number of channels is the sum of the number of channels of the color interaction feature and the intensity interaction feature, i.e., 2C.

[0101] Since the number of channels in the first-level fusion feature is double that of the original color or intensity feature, directly processing the first-level fusion feature with a large number of channels would result in a large number of subsequent parameters in the network, impacting computational performance. Therefore, a channel restoration layer is designed at the output of the interactive fusion module. The channel restoration layer is a 3×3 convolutional layer. This layer reduces the number of channels in the first-level fusion feature by half, restoring it to the same level as the original color or intensity feature, i.e., reducing the number of channels to C. The formula is as follows:

[0102]

[0103] in, This represents the first-level fusion feature after dimensionality reduction.

[0104] In step 500, the squeeze excitation module is a lightweight channel attention module that can perform channel-level attention optimization on the first-level fusion features, adaptively enhance the feature channels that are important for endoscopic imaging, such as tissue texture and lesion feature channels, and suppress ineffective feature channels, such as noise and background interference channels, to obtain the second-level fusion features.

[0105] Specifically, the squeeze activation module includes a feature gating module, a depthwise separable convolutional layer, and an optimization layer. The feature gating module and the depthwise separable convolutional layer are configured in parallel. The feature gating module extracts the channel-level weights of the first-level fusion feature, while the depthwise separable convolutional layer extracts the convolutional features of the first-level fusion feature in a lightweight manner. Finally, the optimization layer multiplies the channel-level weights and convolutional features element-wise to obtain the second-level fusion feature. The first-level fusion feature is then fused to the second-level fusion feature using residual connections (element-wise addition) to compensate for feature information loss during convolution and attention weighting. The formula for the squeeze activation module is as follows:

[0106]

[0107]

[0108] in, This represents the channel-level weights after passing through the feature gating module. This represents the convolutional features after passing through depth-separable convolutional layers. This indicates the secondary fusion characteristics.

[0109] The feature gating module includes an adaptive global average pooling layer, a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. The adaptive global average pooling layer (AdaptiveAvgPool) performs spatial dimensionality global average pooling on the first-level fusion features, mapping the two-dimensional features of each channel to a scalar to obtain pooled features. The first convolutional layer is a 1×1 convolutional layer that performs channel dimensionality reduction on the pooled features, reducing the pooled features of channel C to channel B. The specific value of B is manually calibrated and can be set to 10-20; in this embodiment, 16 is preferred. The first activation layer is a PReLU activation layer, which implements non-linear transformation through the PReLU activation function to alleviate the gradient vanishing problem. The second convolutional layer is also a 1×1 convolutional layer that performs channel dimensionality enhancement on the features, restoring the features of channel B to channel C. The second activation layer is a Sigmoid activation layer, which maps feature values ​​to the [0,1] interval through the Sigmoid activation function to obtain channel-level weights, where the magnitude of the weight value represents the importance of the corresponding channel. Compared to traditional channel attention, which uses a fully connected network to calculate channel-level weights, the feature gating module in this embodiment has fewer parameters and can achieve the same effect, thus making the network lightweight.

[0110] After the excitation module, the secondary fusion feature goes through a 3×3 convolutional layer. The purpose of this convolutional layer is to restore the dimension of the secondary fusion feature, restoring it from C channels to 3 channels, so that it can be reversed to 3-channel RGB format later.

[0111] Furthermore, since the initial HVI features pass through three modules throughout the network, although the features have been optimized, they may have forgotten their original details. Therefore, before the HVI inverse RGB operation, the network adds the initial HVI features to the 3-channel fused features using residual connections (element-wise addition) before performing the inverse transformation operation.

[0112] In step 600, the secondary fusion features are inversely transformed from HVI space to RGB space using the HVI inversion module to obtain the enhanced single-frame image. The HVI space to RGB space conversion also requires HSV space as an intermediate carrier. The specific process is as follows:

[0113] (1) Normalize and decouple the color components to eliminate the mapping effect of the intensity folding function and obtain the intermediate component of polarization HS, as shown in the following formula:

[0114]

[0115]

[0116] in, Represents the horizontal and vertical color components of the HVI color space. This represents the intermediate polarization HS component after decoupling. This represents the intensity folding value in the original HVI conversion.

[0117] (2) Calculate the H, S, and V components in the HSV space using the following formulas:

[0118]

[0119]

[0120]

[0121] in, This represents a custom linear parameter for saturation, used to adjust the color saturation of an image. This represents a custom linear parameter for brightness, used to adjust the overall brightness of the image. For the intensity components in the HVI space.

[0122] (3) Standard HSV-to-RGB conversion to obtain the final RGB pixel values.

[0123] This step is a conventional technical method, and will not be described in detail in this embodiment.

[0124] By following the steps above, the feature map in HVI space is reversed to RGB space to obtain an enhanced image.

[0125] In step S700, the continuous single-frame RGB images enhanced by the above steps are stitched together according to the time sequence of the original video stream to reconstruct a real-time enhanced video stream with a frame rate of more than 30fps to ensure smooth clinical viewing.

[0126] The enhanced video stream is transmitted through the output unit to the endoscope's display device (such as a medical monitor or surgical navigation screen). The display device outputs a clear, flicker-free, and detailed endoscopic image for clinicians to use for diagnosis and surgical procedures.

[0127] The above is the main framework of the network model constructed in this method. The following section introduces the training and validation of this network model. This part follows the conventional training and validation methods and will only be briefly introduced.

[0128] The network framework is trained using the general low-light video dataset SDSD, which consists of strictly frame-level paired low-light / normal-light video pairs containing 150 dynamic video sequences covering multiple scenes, enabling the network to learn general low-light enhancement capabilities.

[0129] In some alternative implementations, after training with SDSD, the network framework can be fine-tuned using a custom-built paired dataset for specific scenarios, such as a paired dataset for the abdominal cavity. This can improve the performance enhancement for specific scenarios.

[0130] The loss function is a weighted combination of three loss functions commonly used in this field: L1 loss function, MSE loss function, and SSIM loss function. The three loss functions are combined in a weighted manner to form the total loss, and the weights of each loss function are determined experimentally.

[0131] After training with SDSD, the model's generalization performance was validated using a low-light subset of the publicly available unpaired endoscopic dataset HyperKvasir (gastrointestinal endoscopy). The test subset consisted of data from selected low-light scenes within HyperKvasir. This dataset includes low-light / unevenly lit samples collected clinically via endoscopy, but lacks ground truth images with normal light. Therefore, NIQE and BRISQUE, two no-reference image quality metrics, were used (lower scores indicate better enhancement). The tests yielded NIQE and BRISQUE scores of 4.36 and 33.58, respectively, which are considered satisfactory and fall within the upper-middle range for low-light video / image enhancement.

[0132] To more intuitively demonstrate the enhancement effect of the model, Figure 2 The results of the visualization comparison test of some single-frame images in the dataset are also shown. It can be clearly seen that compared with the input low-light image, the output enhanced image can not only improve the brightness, but also make the lesions, tissues and blood vessels in the image clearer, proving that the deep learning network model constructed by this method is effective.

[0133] Then, this method uses 1920×1080 resolution images to test the number of parameters and floating-point operations (FLOPs) of the network model. The test results show that the lightweight network model constructed by this method has approximately 1.13M parameters and approximately 4.87G FLOPs, achieving a frame rate greater than 30fps on an endoscope display device (1080P). This frame rate is sufficient to support smooth playback of enhanced video streams on the endoscope display device.

[0134] This application also provides a low-light video enhancement system for endoscopic imaging, including:

[0135] The preprocessing unit is used to acquire video streams inside the human cavity in real time through an endoscope and decompose the video stream into independent image frames, wherein the image frames are RGB format data;

[0136] The mapping unit is used to map the image frame from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map.

[0137] The denoising and enhancement unit is used to denoise and enhance the color map and intensity map through a VMamba-based dual-branch U-Net network to obtain optimized color features and intensity features;

[0138] The fusion unit is used to perform lightweight interaction and fusion of the color feature and the intensity feature through the interactive fusion module to obtain a first-level fused feature;

[0139] The optimization unit is used to perform channel-level optimization on the first-level fusion feature through the squeezing excitation module to obtain the second-level fusion feature;

[0140] The inverse transformation unit is used to inversely transform the secondary fusion features from HVI space to RGB space through the HVI transformation module to obtain an enhanced single-frame image.

[0141] The output unit is used to stitch consecutive enhanced single-frame images into an enhanced video stream and output the enhanced video stream to a display device.

[0142] This system has all the advantages of the methods described above, which will not be elaborated here.

[0143] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0144] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A low-light video enhancement method for endoscopic imaging, characterized in that, The method includes: The video stream inside the human cavity is acquired in real time through an endoscope, and the video stream is decomposed into independent image frames, which are RGB format data. The image frame is mapped from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map; The color and intensity maps are denoised and enhanced using a VMamba-based dual-branch U-Net network to obtain optimized color and intensity features. The color feature and the intensity feature are fused through a lightweight interaction using an interactive fusion module to obtain a first-level fused feature. The primary fusion feature is optimized at the channel level by the squeezing excitation module to obtain the secondary fusion feature; The secondary fusion features are inversely transformed from HVI space to RGB space using the HVI transformation module to obtain an enhanced single-frame image. The continuous enhanced single-frame images are stitched together to form an enhanced video stream, and the enhanced video stream is output to a display device.

2. The low-light video enhancement method according to claim 1, characterized in that, The VMamba-based dual-branch U-Net network includes two independent U-Net branches. The two U-Net branches perform denoising and enhancement processing on the color map and the intensity map, respectively. Each U-Net branch includes an encoder, a VMamba module, and a decoder, with the VMamba module located between the encoder and the decoder.

3. The low-light video enhancement method according to claim 2, characterized in that, The encoder includes several downsampling layers. After a feature passes through the downsampling layer, the number of channels remains unchanged, and the width and height are both 0.5 times that of the original feature. The decoder includes several upsampling layers. After a feature passes through the upsampling layers, the number of channels remains unchanged, while the width and height are twice that of the original feature. The features output by each downsampling layer are added element-wise to the features output by the upsampling layer at the corresponding scale of the decoder to form skip connections.

4. The low-light video enhancement method according to claim 3, characterized in that, The U-Net branch also includes a dimension-upgrading module placed before the encoder. The dimension-upgrading module is a 3×3 convolutional layer, and the number of channels increases after the feature passes through the dimension-upgrading module.

5. The low-light video enhancement method according to claim 1, characterized in that, The interactive fusion module includes a bidirectional interactive layer and a fusion layer; Through the bidirectional interaction layer, the color feature is added to the weighted intensity feature in an element-wise addition manner to obtain the color interaction feature, and the intensity feature is added to the weighted color feature in an element-wise addition manner to obtain the intensity interaction feature; Through the fusion layer, the color interaction feature and the intensity interaction feature are fused by channel splicing to obtain a first-level fused feature.

6. The low-light video enhancement method according to claim 5, characterized in that, The interactive fusion module also includes a channel recovery layer, which is a 3×3 convolutional layer. Through the channel recovery layer, the number of channels of the first-level fusion feature is reduced by half.

7. The low-light video enhancement method according to claim 1, characterized in that, The extrusion excitation module includes a feature gating module, a depthwise separable convolutional layer, and an optimization layer; After the primary fusion feature passes through the feature gating module, the channel-level weights corresponding to the primary fusion feature are extracted. The first-level fusion features are passed through the depth-separable convolutional layer to obtain convolutional features; Through the optimization layer, the channel-level weights and the convolutional features are multiplied element-wise to obtain the secondary fusion features.

8. The low-light video enhancement method according to claim 7, characterized in that, Through the optimization layer, the first-level fusion feature is fused to the second-level fusion feature in a residual connection manner.

9. The low-light video enhancement method according to claim 7, characterized in that, The feature gating module includes an adaptive global average pooling layer, a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence.

10. A low-light video enhancement system for endoscopic imaging, characterized in that, include: The preprocessing unit is used to acquire video streams inside the human cavity in real time through an endoscope and decompose the video stream into independent image frames, wherein the image frames are RGB format data; The mapping unit is used to map the image frame from RGB space to HVI space through the HVI conversion module, so that the image frame is decoupled into a color map and an intensity map. The denoising and enhancement unit is used to denoise and enhance the color map and intensity map through a VMamba-based dual-branch U-Net network to obtain optimized color features and intensity features; The fusion unit is used to perform lightweight interaction and fusion of the color feature and the intensity feature through the interactive fusion module to obtain a first-level fused feature; The optimization unit is used to perform channel-level optimization on the first-level fusion feature through the squeezing excitation module to obtain the second-level fusion feature; The inverse transformation unit is used to inversely transform the secondary fusion features from HVI space to RGB space through the HVI transformation module to obtain an enhanced single-frame image. The output unit is used to stitch consecutive enhanced single-frame images into an enhanced video stream and output the enhanced video stream to a display device.

Images