Endoscope adopting RGB + NIR single sensor and having hemodialysis and fog penetration functions

Abstract

The invention discloses an endoscope system adopting an RGB + NIR single sensor and having a hemodialysis and fog penetration function, and relates to the technical field of medical imaging, the endoscope system comprises a handle, the left end of the handle is connected with an endoscope tube body, and an optical lens, an image sensor and an image processing circuit are installed in the handle or the endoscope tube body; the image sensor adopts a CMOS image sensor of a monolithic integrated visible light (RGB) and near-infrared (NIR) light sensing unit, and a visible light image and a near-infrared image can be obtained at the same time; the installation of the optical lens, the image sensor and the image processing circuit has two implementation structures: 1, in an optical endoscope scheme, the optical lens, the image sensor and the image processing circuit are installed inside the handle, and the endoscope tube body transmits an optical image into the handle; 2, in the scheme of the electronic endoscope, the optical lens and the image sensor are arranged at the front end in the endoscope tube body, direct imaging is carried out in the body, and the image processing circuit is arranged in the handle; the image processing circuit fuses the visible light image and the near-infrared image, a fusion algorithm dynamically adjusts RGB and NIR signal weights according to an operation scene, and high-contrast clear imaging is achieved in a bleeding, smog or liquid medium environment. The endoscope adopting the RGB + NIR single sensor and having the hemodialysis and fog penetration functions is suitable for minimally invasive surgery scenes such as a thoracoscope, a laparoscope and an arthroscope.

Description

Technical Field

[0001] This invention relates to the field of medical devices and optoelectronic imaging technology, specifically to an endoscope that uses RGB+NIR single-sensor fusion imaging, which has the functions of blood penetration, fog penetration and enhanced tissue visibility, and is suitable for minimally invasive surgical scenarios such as thoracoscopy, laparoscopy, and arthroscopy. Background Technology

[0002] Endoscopic examination and surgery are important methods for modern medical diagnosis and treatment, widely used in fields such as thoracoscopy, laparoscopy, arthroscopy, urology, and gastrointestinal endoscopy. During minimally invasive surgeries such as electrocautery and electrocoagulation, the vaporization and burning of tissues produce a large amount of "surgical smoke." These smoke particles severely scatter and absorb light, leading to blurred endoscopic vision and reduced contrast, affecting the surgeon's precision and surgical safety. Furthermore, unavoidable bleeding during surgery, especially the thin layer of blood forming on the tissue surface, strongly absorbs visible light, obscuring underlying blood vessels and tissue structures, posing a significant challenge to the identification of critical tissues.

[0003] To address these issues, existing technologies have explored various approaches. One approach is to utilize near-infrared (NIR) imaging, particularly the 700-900 nm wavelength range, which has stronger penetrating power for blood and smoke. However, traditional endoscopic systems primarily use visible light (RGB) imaging, which cannot acquire NIR information.

[0004] To simultaneously acquire RGB and NIR images, the current mainstream approach is to use a dual-sensor system: a beam splitter (such as a beam splitter prism or filter) separates the incident light into visible and near-infrared light paths, which are received by an RGB sensor and an NIR (usually a mono) sensor, respectively. While this approach allows for simultaneous imaging, it has several inherent drawbacks: 1. Image registration is difficult. The two optical paths and two sensors are physically separated. Any tiny assembly tolerance, thermal deformation or optical distortion will cause spatial misalignment (i.e. registration error) between the two images. In surgical scenarios that require precise fusion, such misalignment is unacceptable. 2. Light flux loss: The beam splitter divides the total light flux in two, resulting in a decrease in the signal-to-noise ratio of each sensor. This is particularly disadvantageous in the deep cavity environment of an endoscope where lighting conditions are already limited. 3. The system is complex and costly. The dual sensors, beam splitter, and precise optical axis alignment structure significantly increase the size, weight, and manufacturing cost of the endoscope lens module.

[0005] Another approach is to use single-sensor time-division imaging, for example, by switching light sources or filters, allowing the same sensor to acquire RGB and NIR images sequentially. While this method avoids registration problems, because the two (or more) images are acquired sequentially, severe motion blur and artifacts will occur when capturing moving scenes (such as a beating heart or moving surgical instruments), which cannot meet the needs of real-time surgery.

[0006] In recent years, a single-chip RGB+NIR image sensor technology has emerged. This sensor integrates a special color filter array (CFA) on the pixel array, for example, containing four pixel channels—R, G, B, and NIR—simultaneously in a 2x2 cell. This design eliminates the need for image registration and beam splitting, enabling synchronous imaging of all pixels and achieving high frame rate synchronous image acquisition. However, this single-chip RGB+NIR solution also brings new technical challenges: 1. Data processing is difficult. The sensor outputs sparse RGBN mosaic raw data. How to reconstruct a high-quality, full-resolution four-channel image through the "demosaicing" algorithm is a complex image processing problem. 2. Severe spectral crosstalk: Due to manufacturing limitations, RGB filters often cannot completely cut off the near-infrared band (especially above 700nm), causing a large amount of NIR light to "leak" into the RGB channels. This crosstalk severely contaminates color information, resulting in color cast, reduced saturation, and color distortion in images.

[0007] Therefore, how to design an endoscope system based on a single-chip RGB+NIR sensor and develop an efficient image processing workflow to solve the two core challenges of de-mosaicing and spectral crosstalk, while simultaneously achieving intelligent blood and fog penetration enhancement fusion, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] The purpose of this invention is to provide an endoscope with blood and fog penetration function using an RGB+NIR single sensor, in order to solve the problems mentioned in the background art, such as the obstructed field of view of existing endoscopes in smoke and blood environments, and the problems of image registration difficulties, low light throughput or motion blur, and severe spectral crosstalk in existing RGB+NIR imaging schemes. It also provides two implementation architectures: optical endoscope and electronic endoscope.

[0009] To achieve the above objectives, the present invention provides the following technical solution: An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor includes a handle (1), a tube body (2), an optical lens (3), an image sensor (4), and an image processing circuit (5). The image sensor (4) is a single RGB+NIR image sensor.

[0010] This invention provides two system architectures for implementing the above functions: 1. Optical endoscope structure: The endoscope tube (2) is an optical image guiding system. The optical lens (3), image sensor (4) and image processing circuit (5) are embedded in the handle (1). At this time, the optical lens (3) in the handle (1) is used to receive the optical image transmitted from the observation site by the endoscope tube (2). II. Electronic endoscope structure: The endoscope tube (2) is an insertion tube, and an optical lens (3) and an image sensor (4) are installed at the front end of the tube. The optical lens (3) and the image sensor (4) directly perform optical imaging and photoelectric conversion at the observation site in the patient's body, and the generated image electrical signal is transmitted to the image processing circuit (5) of the handle (1).

[0011] The core of this invention lies in the fact that the image sensor (4) adopts a single-chip RGB+NIR imaging sensor, which can simultaneously acquire visible light images and near-infrared images. Specifically, the pixel array of the image sensor (4) integrates both RGB and NIR channels. After light passes through the optical lens (3), it can be directly projected onto the sensor without any beam splitting device, thus achieving synchronous imaging. This structural design fundamentally avoids the image registration error and light flux loss problems of dual-sensor schemes, and also avoids the motion artifact problem of time-division imaging schemes.

[0012] Furthermore, to achieve the aforementioned monolithic integration, the image sensor (4) adopts a Bayer-like structure, and its color filter array (CFA) is specially designed. Specifically, each 2×2 pixel unit includes one NIR pixel, one R pixel, one G pixel, and one B pixel. This four-channel mosaic arrangement is the basis for achieving simultaneous acquisition of RGB and NIR information on a single chip.

[0013] Furthermore, to achieve the blood-penetrating and fog-penetrating functions, this invention defines the spectral characteristics of the NIR channel. The NIR channel band range is 700–900 nm. This band range is selected based on the optical absorption and scattering characteristics of blood (hemoglobin) and water (the main component of smoke). Within this window, near-infrared light has strong penetrability and can penetrate thin layers of blood and smoke scattering areas to obtain structural information of the obscured tissue.

[0014] Furthermore, the present invention also includes a targeted image processing scheme, which is executed by the image processing circuit (5). The image processing circuit (5) processes the sparse mosaic data acquired by the image sensor (4), and then fuses the images of the four channels NIR, R, G, and B to finally output an RGB-encoded color image for the doctor to observe.

[0015] Furthermore, the image fusion of this invention is not a simple linear superposition, but an intelligent adaptive fusion. The image processing circuit (5) analyzes each region of the image in real time based on image features such as local contrast, brightness distribution, and scattering estimation. When it detects that a region is obscured by blood (e.g., low brightness, high red component, low contrast) or shrouded in smoke (e.g., low contrast, loss of high-frequency information), it adaptively adjusts the fusion ratio of RGB and NIR, significantly increasing the weight of the NIR component in that region. In normal regions with clear vision, the weight of the RGB image is maintained to present natural colors. This adaptive adjustment mechanism achieves real-time, local blood and fog penetration enhancement effects.

[0016] Furthermore, in order to solve the two core problems of single-chip RGB-NIR sensors mentioned in the background art (de-mosaicing and spectral crosstalk), this invention provides a specific data processing flow. The data processing flow of the image processing circuit (5) is as follows: 1. De-mosaic processing. Image interpolation methods (such as bilinear interpolation, edge-preserving interpolation, or deep learning-based interpolation methods) are used to process the sparse mosaic data obtained from the sensor (4) to reconstruct four full-resolution images: NIR channel image, R channel image, G channel image, and B channel image; II. Spectral Crosstalk Correction. Using the full-resolution NIR channel image obtained in the first step, color correction is performed on the RGB channel image. Specifically, the NIR channel data is subtracted proportionally from the RGB channel data. This is because the response of the RGB channels to the NIR band (i.e., crosstalk) can be approximated as a proportion of the NIR channel signal. By subtracting this crosstalk, the RGB channel data can more accurately reflect visible light information, restoring the true colors of the image. 3. Color space decomposition. The RGB channel data (with restored colors) after crosstalk correction is decomposed into a luminance image (Y component) and a color image (U, V components or Cb, Cr components); IV. Multi-level Image Fusion. The RGB and NIR channels are fused. The fusion process in this invention is step-by-step: First, the luminance image (Y component) of the RGB channel is fused with the NIR channel image. This step is the core of the enhancement, combining the luminance details of visible light with the penetration details of near-infrared light (e.g., blood vessels, tissue boundaries) to generate a high-contrast, structurally rich fused luminance image. This fusion employs the aforementioned adaptive weight adjustment strategy. Then, the fused luminance image is fused (i.e., merged) with the color images (U and V components) of the RGB channels separated in the third step. Finally, the fused YUV (or YCbCr) image is converted back to the RGB space to obtain the final enhanced color image output.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. No registration error and motion artifacts: Using a single RGB+NIR sensor, all channels share the same optical path and the same chip, and image at the same time, fundamentally eliminating the spatial registration error of multi-sensor systems and the motion blur problem of time-division systems; 2. High signal-to-noise ratio and system simplicity: No beam splitting device is required, resulting in high light throughput and a good imaging signal-to-noise ratio, making it particularly suitable for the light-constrained deep cavity environment of endoscopy. Simultaneously, the system has a simple structure, small size, and low cost. 3. Solved spectral crosstalk: This invention proposes a processing flow of "first removing mosaic, then using the NIR channel to correct the RGB channel", which effectively solves the inherent spectral crosstalk problem of single-chip RGB-NIR sensors and ensures the color fidelity of the unobstructed area while enhancing the perspective capability. 4. Intelligent Adaptive Enhancement: This invention employs an adaptive fusion algorithm that can locally and in real-time adjust the fusion weights of the NIR channels based on the obstruction caused by smoke and blood. This achieves clear fluoroscopic effects in obstructed areas while preserving natural color anatomical views in normal areas, providing doctors with intuitive and easy-to-interpret surgical images. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the pixel arrangement principle of the four channels (R, G, B, NIR) of the RGB+NIR single sensor of the present invention. Figure 2 This is a schematic diagram of the image processing flow of the endoscope of the present invention; Figure 3 This is a schematic diagram of the overall structure of the endoscope of the present invention; Figure 4 This is a partial cross-sectional view of the handle using the optical mirror scheme of the present invention; Figure 5This is a partial cross-sectional view of the handle and endoscope tube of the electronic mirror solution of the present invention.

[0019] In the diagram: 1. Handle; 2. Lens tube; 3. Optical lens; 4. RGB+NIR single image sensor; 5. Image processing circuit. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Please refer to Figures 1-4 The present invention provides the following technical solution: an endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor, comprising a handle (1), a scope tube (2) mounted on the left end of the handle (1), and a lens assembly embedded inside the handle (1). The lens assembly is the core of the present invention, and it adopts a single RGB+NIR image sensor scheme. The lens assembly consists of an outermost optical lens (3), an image sensor (4) disposed opposite to the optical lens (3), and an image processing circuit (5) electrically connected to the sensor (4).

[0022] In this embodiment, the image sensor (4) is a single-chip RGB+NIR imaging sensor, which integrates visible light (RGB) channels and near-infrared (NIR) channels on its pixel array. After light passes through the optical lens (3), it can be imaged on the sensor (4) simultaneously without beam splitting, thus achieving synchronous acquisition of visible light images and near-infrared images.

[0023] Specifically, the color filter array of the image sensor (4) adopts a Bayer-like structure, with its smallest repeating unit being a 2×2 pixel, which includes one NIR pixel, one R pixel, one G pixel, and one B pixel. The response band range of the NIR channel is designed to be 700–900 nm to ensure good penetration of blood and surgical smoke.

[0024] Example 2: Based on Example 1, this example further discloses the working method and data processing flow of the image processing circuit (5). The image processing circuit (5) is responsible for receiving the raw mosaic data from the image sensor (4) and processing and fusing it into the final RGB encoded color image data.

[0025] The fusion process is adaptive: the image processing circuit (5) has a built-in analysis module that calculates the local contrast, brightness distribution, and scattering estimation parameters of the image to determine the occlusion situation in the field of view. In areas determined to be blood occlusion or smoke environments, the circuit (5) automatically increases the fusion weight of the NIR channel image; in clear areas, the weight of the RGB channel is maintained. This enables the endoscope to achieve blood and fog penetration enhancement in real time and intelligently. The specific data processing flow is as follows: 1. Perform de-mosaic processing. The image processing circuit (5) uses an edge-preserving interpolation image interpolation algorithm to process the RGBN mosaic data output by the sensor (4) and reconstruct the full-resolution images of the four channels: R, G, B, and NIR. 2. Perform spectral crosstalk correction. Using the NIR channel image obtained in the previous step, correct the R, G, and B channel images. The correction formula is as follows (the G and B channels are corrected similarly, where k is the crosstalk coefficient): R_corrected = R_raw - k_r * NIR_raw This move aims to reduce the contamination of the RGB channels by the NIR band and restore color fidelity. 3. Perform color space conversion. Convert the corrected RGB image to the YUV (or YCbCr) color space, and decompose it into a luminance image Y and two color images U and V; IV. Performing Fusion and Combination. Fusion consists of two steps. First, the luminance image Y of the RGB channel is fused with the NIR channel image to obtain the fused luminance Y_fused. Second, the fused luminance Y_fused is merged with the previously separated color images U and V to reassemble them into a YUV image. Finally, the YUV image is converted back to the RGB space and output as the final video signal to the display.

[0026] Example 3: This invention also provides an implementation scheme for an electronic endoscope. (Reference) Figure 5 In this scheme, the endoscope tube (2) is an insertion tube, with an optical lens (3) and an image sensor (4) installed at its front end. The optical lens (3) and the image sensor (4) directly perform optical imaging and photoelectric conversion at the observation site inside the patient's body, and the resulting image electrical signal is transmitted to the image processing circuit (5) of the handle (1). After receiving the mosaic data transmitted from the remote end, the image processing circuit (5) executes the same image processing flow as in Embodiment 2, and finally outputs an enhanced color image with blood penetration and fog penetration functions.

[0027] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0028] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor, comprising a handle (1), wherein a scope body (2) is mounted on the left end of the handle (1), and an optical lens (3), an image sensor (4), and an image processing circuit (5) are mounted inside the handle (1) or the scope body (2), wherein the mounting method is one of the following two structures: I. Optical endoscope structure: The endoscope tube (2) is an optical image guiding system. The optical lens (3), image sensor (4) and image processing circuit (5) are embedded in the handle (1) to receive and capture the optical image transmitted by the endoscope tube (2). II. Electronic endoscope structure: The optical lens (3) and image sensor (4) are installed inside the front end of the endoscope tube (2) for direct imaging at the observation site; the image processing circuit (5) is installed inside the handle (1) for processing image signals. Its features are: The image sensor (4) adopts a single RGB+NIR image sensor, which can simultaneously obtain visible light images and near-infrared images. The pixel array of the image sensor (4) integrates RGB and NIR channels, and synchronous imaging can be achieved without light splitting.

2. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor as described in claim 1, characterized in that: The image sensor (4) adopts a Bayer-like structure, and each 2×2 pixel unit includes 1 NIR pixel, 1 R pixel, 1 G pixel and 1 B pixel.

3. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor as described in claim 1, characterized in that: The NIR channel has a wavelength range of 700–900 nm and is used to penetrate the scattering areas of blood and smoke.

4. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor as described in claim 1, characterized in that: The image processing circuit (5) fuses the images of the four channels NIR, R, G and B acquired by the image sensor (4) and outputs RGB encoded color image data.

5. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor as described in claim 4, characterized in that: The image processing circuit (5) adaptively adjusts the RGB and NIR fusion ratio based on local contrast, brightness distribution and scattering estimation, and increases the weight of NIR components in blood-covered or smoke-filled environments to achieve real-time blood and fog penetration enhancement.

6. An endoscope with blood-penetrating and fog-penetrating functions using an RGB+NIR single sensor as described in claim 4, characterized in that: The data processing flow of the image processing circuit (5) is as follows:

1. De-mosaic processing is performed on the NIR and RGB channels using image interpolation methods; 2. Use the NIR channel image to perform color correction on the RGB channel image. Specifically, subtract the NIR channel data from the RGB channel data proportionally to reduce the crosstalk effect of the NIR channel.

3. Decompose the RGB channel data into luminance and color images; Fourth, perform image fusion of RGB and NIR channels. Specifically, first fuse the luminance image of the RGB channel with the NIR image, and then fuse the fusion result with the color image of the RGB channel.

Images