Endoscopic hdr imaging method and system

Abstract

This invention discloses an endoscopic HDR imaging method and system, comprising: controlling incident light to be split into two paths by a beam splitter according to a preset beam splitting ratio K, and projecting them onto two independent physical photosensitive areas of an image sensor, simultaneously acquiring the bright channel image and dark channel image of the current frame; firstly, independently metering the bright channel image and dark channel image to obtain dual-region brightness statistics, then establishing dual constraints on exposure parameters based on the preset beam splitting ratio K, and finally adjusting the exposure parameters according to the dual-region brightness statistics and under the constraints of the dual constraints, outputting the optimal exposure parameters for the next frame acquisition; and fusing the dark channel image and bright channel image pixel by pixel to generate a high dynamic range image. The solution of this application not only adapts to various small-diameter medical endoscopes, but also deeply integrates hardware beam splitting characteristics into the image processing process, enabling the generation of high-quality enhanced images.

Description

Technical Field

[0001] This application relates to the field of endoscopic imaging, and more particularly to an endoscopic HDR imaging method and system. Background Technology

[0002] Medical endoscopes are essential equipment for minimally invasive diagnosis and treatment in clinical practice. Small-diameter endoscopes such as otolaryngoscopes, cystoscopes, hysteroscopes, and arthroscopes are widely used for the examination and surgery of human cavities. The human body cavity environment is complex, often presenting extreme scenarios where mucosal reflections, metallic instrument reflections, and dark areas due to tissue folds coexist. This places extremely high demands on the dynamic range of the imaging system.

[0003] Currently, high dynamic range (HDR) imaging in endoscopy mainly employs three technical solutions. The first is single-frame, single-image imaging, which acquires a single image through global exposure compromise. The second is multi-frame HDR synthesis, which fuses multiple images captured at different exposures. The third is beam splitting imaging, which uses optical elements to split the light path into two, acquiring images of different brightness levels separately.

[0004] Existing technologies have significant drawbacks. Single-frame, single-view solutions cannot simultaneously capture details in both bright and dark areas, easily resulting in the loss of crucial clinical information. Multi-frame synthesis solutions suffer from motion ghosting and latency issues, affecting doctors' real-time judgment. Dual-sensor beam splitting solutions are bulky, unsuitable for small-diameter endoscopes, and suffer from parallax and high power consumption. Existing single-sensor beam splitting solutions lack algorithms designed to incorporate hardware beam splitting characteristics, making them prone to noise degradation and poor fusion results. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide an endoscopic HDR imaging method and system to solve problems such as motion ghosting, excessive size that cannot be adapted to small diameter, inability to capture both bright and dark details, and noise degradation caused by the disconnect between software and hardware control in existing endoscopic high dynamic range imaging technologies.

[0006] The present invention solves the above-mentioned technical problems through the following technical means:

[0007] On the one hand, an endoscopic HDR imaging method is provided, applied to an endoscopic imaging device including a beam splitter and an image sensor, the imaging method comprising:

[0008] S100, Spectral imaging: The incident light is controlled to be split into two paths by the spectral structure according to a preset spectral ratio K, and projected onto two independent physical photosensitive areas of the image sensor respectively. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously.

[0009] S200, Linked Exposure Control: First, the bright channel image and the dark channel image are independently metered to obtain dual-region brightness statistics. Then, based on the preset splitting ratio K, dual constraints are established for the exposure parameters. Finally, based on the dual-region brightness statistics and under the constraints of the dual constraints, the exposure parameters are adjusted to output the optimal exposure parameters for the next frame acquisition.

[0010] S300, Image fusion and reconstruction: The dark channel image and the bright channel image are fused pixel by pixel to generate a high dynamic range image.

[0011] In one possible implementation, the beam-splitting structure includes an optical adapter, a beam-splitting prism, a reflecting prism, a first optical path compensation element, and a second optical path compensation element. The incident light is first focused and imaged by the optical adapter, and then transmitted to the beam-splitting prism. The beam-splitting prism splits the light beam into a first optical path and a second optical path. The first optical path is projected onto one photosensitive area of ​​the image sensor after passing through the first optical path compensation element. The second optical path is deflected by the reflecting prism, and then projected onto another photosensitive area of ​​the image sensor after passing through the second optical path compensation element.

[0012] In one possible implementation, the first optical path compensation element and the second optical path compensation element are compensation prisms or air gap structures.

[0013] In one possible implementation, the bright channel image and the dark channel image are independently metered to obtain dual-region brightness statistics, including:

[0014] S211, Bright Channel Statistics: Statistics on the proportion of low-brightness pixels in the bright channel image. and average brightness The formula is as follows:

[0015]

[0016] = in bright channel average brightness

[0017] The effective dark pixel value range in the bright channel is [0, 80]; the effective pixel value range in the bright channel is [0, 255].

[0018] S212, Dark Channel Statistics: Statistical analysis of the peak brightness of effective pixels in the dark channel image. , It equals the average brightness of the first A% of the brightest pixels in the dark channel.

[0019] In one possible implementation, the value of A is in the range of [0.001, 10].

[0020] In one possible implementation, the exposure parameters are established based on the preset spectral ratio K under dual constraints, including:

[0021] S221. Establish a dark channel saturation warning threshold, that is, set the maximum allowable brightness threshold for the dark channel. The formula is as follows:

[0022]

[0023] in, The safety factor has a value range of [0.85, K].

[0024] S222, Establish physical gating for signal-to-noise ratio (SNR) The formula is as follows:

[0025]

[0026]

[0027] in, For the scene The average brightness This is due to the inherent noise of the system. This is an image of the dark channel region.

[0028] In one possible implementation, in step S300, before fusing the dark channel image and the bright channel image pixel by pixel, the dark channel image and the bright channel image are geometrically aligned and extracted.

[0029] In one possible implementation, in step S300, before fusing the dark channel image and the bright channel image pixel by pixel, the dark channel image is brightness corrected based on the preset spectral ratio K.

[0030] In one possible implementation, brightness correction of the dark channel image based on the preset spectral ratio K includes:

[0031] S321. Calculate the brightness limit guarantee weight of the bright and dark channels based on the preset splitting ratio K. The formula is as follows:

[0032]

[0033]

[0034] S322, Weighting based on brightness limit Brightness limiting processing is performed on the bright channel image and the dark channel image. Then, based on the preset splitting ratio K, the dark channel image is... To perform K times linear optical correction, the formula is as follows:

[0035]

[0036]

[0037] in, Represents the bright channel image. Dark channel image.

[0038] In one possible implementation, in step S300, an adaptive weighted fusion algorithm is used to fuse the dark channel image and the bright channel image pixel by pixel.

[0039] On the other hand, an endoscopic HDR imaging system is provided, comprising:

[0040] The beam splitting imaging module controls the incident light to be split into two beams by the beam splitting structure according to a preset beam splitting ratio K, and projected onto two independent physical photosensitive areas of the image sensor respectively. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously.

[0041] The linked exposure control module first independently meters the bright channel image and the dark channel image to obtain dual-region brightness statistics. Then, based on the preset splitting ratio K, it establishes dual constraints on the exposure parameters. Finally, based on the dual-region brightness statistics and under the constraints of the dual constraints, it adjusts the exposure parameters and outputs the optimal exposure parameters for the next frame acquisition.

[0042] The image fusion and reconstruction module fuses the dark channel image and the bright channel image pixel by pixel to generate a high dynamic range image.

[0043] The beneficial effects of this application are:

[0044] 1. This application uses a single-channel image sensor, which makes the endoscope tip smaller and can be adapted to various small-diameter medical endoscopes. At the same time, the system has low power consumption and low manufacturing cost.

[0045] 2. This application acquires two images from the same source, one bright and one dark, simultaneously in a single frame and a single exposure, which can effectively improve the problem of motion ghosting and has low imaging delay, meeting the needs of real-time clinical diagnosis and treatment.

[0046] 3. This application deeply integrates hardware spectral characteristics into the exposure control and image fusion process, which can simultaneously preserve clinical details in bright reflective areas and dark shadow areas, thereby reducing the loss of key information. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the endoscopic HDR imaging method in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the endoscopic imaging device in the embodiments of this application. Figure 1 ;

[0050] Figure 3 This is a schematic diagram of the endoscopic imaging device in the embodiments of this application. Figure 2 ;

[0051] Figure 4 This is a schematic diagram of the endoscopic imaging device in the embodiments of this application. Figure 3 ;

[0052] Figure 5 This is a schematic diagram of the endoscopic imaging device in the embodiments of this application. Figure 4 ;

[0053] Figure 6 This is a schematic diagram of a single-frame dual-image image within the effective imaging area of ​​the image sensor in an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the linked exposure control area in the embodiments of this application;

[0055] Figure 8 This is a schematic diagram comparing input and output in an embodiment of this application;

[0056] Explanation of reference numerals: 111, Optical adapter; 112, Beam splitter prism; 113, Reflecting prism; 114, Auxiliary prism; 115, First optical path compensation component; 116, Second optical path compensation component; 120, Image sensor. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0059] Furthermore, the use of terms such as "first," "second," etc., in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0060] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0061] like Figure 1-8 As shown, this application provides an endoscopic HDR (High Dynamic Range) imaging method, applied to an endoscopic imaging device, which includes a beam splitting structure and a single-channel image sensor 120. Specifically, the imaging method includes the following steps:

[0062] S100, Spectroscopic Imaging: The incident light is split into two paths by the spectroscopic structure according to the preset splitting ratio K, and projected onto two independent physical photosensitive areas of the image sensor 120 respectively. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously.

[0063] S200, Linked Exposure Control: First, the bright channel image and the dark channel image are independently metered to obtain dual-region brightness statistics. Then, based on the preset split ratio K, dual constraints are established for the exposure parameters. Finally, based on the dual-region brightness statistics and under the constraints of the dual constraints, the exposure parameters are adjusted to output the optimal exposure parameters for the next frame acquisition.

[0064] S300, Image fusion and reconstruction: The dark channel image and the bright channel image are fused pixel by pixel to generate a high dynamic range image.

[0065] This solution employs a single-channel image sensor architecture (120), resulting in a compact endoscope tip size, compatibility with various small-diameter endoscopes, and lower manufacturing costs and overall power consumption. Simultaneously, by simultaneously acquiring both bright and dark images in a single frame, motion ghosting is effectively eliminated, ensuring good real-time imaging performance. Furthermore, this solution deeply integrates hardware beam splitting characteristics into the image processing process. On one hand, it combines beam splitting ratio design with exposure control, simultaneously protecting against overexposure in bright areas and ensuring a good signal-to-noise ratio in dark areas. On the other hand, image fusion based on beam splitting ratio correction enhances the clarity of clinical details such as mucosal texture and minute blood vessels.

[0066] The following is a detailed explanation of each of the above steps:

[0067] In step S100, in order to acquire two images with different brightness information through an image sensor 120, this embodiment sets a beam-splitting structure with a preset splitting ratio of K at the incident light acquisition end. Specifically, the beam-splitting structure includes an optical adapter 111, a beam-splitting prism 112, a reflecting prism 113, a first optical path compensation component 115, and a second optical path compensation component 116. Before entering the image sensor 120, the parallel incident light acquired by the endoscope is first focused into an image by the optical adapter 111, and then the single-path beam is transmitted to the beam-splitting prism 112. The beam-splitting prism 112 is provided with a beam-splitting mode to split the beam into a first optical path and a second optical path. The first optical path is projected onto one of the photosensitive areas of the image sensor 120 after passing through the first optical path compensation device 115; while the second optical path is projected onto the reflecting prism 113, which is provided with a reflecting mode, which can deflect the second optical path, and then projected onto another photosensitive area of ​​the image sensor 120 through the second optical path compensation device 116.

[0068] By cooperating with the beam splitter 112 and the reflecting prism 113, the incident light can be split into two parallel beams with different brightness information, so that the image sensor 120 can simultaneously acquire bright channel images and dark channel images with inconsistent brightness information and non-overlapping positions. By cooperating with the first optical path compensation device 115 and the second optical path compensation device 116, the optical path difference between the two beams can be specifically compensated, so that the optical paths of the two beams are equivalent, thereby making the two images acquired by the image sensor 120 clearer.

[0069] In one possible implementation, the beam-splitting prism 112 is a right-angled prism structure, with a beam-splitting film coated on its inclined surface. The beam-splitting ratio K of the beam-splitting film can be any ratio between (0, 10). This beam-splitting ratio K is the intensity ratio between the bright channel image and the dark channel image. If K is too small, the dynamic range is insufficient; if it is too large, the noise in the dark channel becomes uncontrollable. To obtain two images with different brightness information, a more suitable beam-splitting ratio K is [6:4, 8:2], and the refractive index N1 of the beam-splitting prism 112 can be [1.5, 1.9]. In this embodiment, the beam-splitting ratio K of the beam-splitting prism 112 is preferably 7:3, and the refractive index N1 is preferably 1.75. Furthermore, the beam-splitting film adopts a multilayer dielectric film design to ensure a stable beam-splitting ratio throughout the entire visible light band (400-780nm).

[0070] In one possible implementation, the reflecting prism 113 uses a high refractive index transmission medium, with a refractive index N2 ranging from [1.5, 1.9]. In this embodiment, the refractive index N2 of the reflecting prism 113 is preferably 1.75, and optical glass with a 45-degree reflective surface is preferably used as the reflecting prism 113. Furthermore, to ensure more reliable and stable installation of the reflecting prism 113, an auxiliary prism 114 is also provided for assembling the reflecting prism 113.

[0071] In one possible implementation, the optional design of the first optical path compensation member 115 and the second optical path compensation member 116 is not limited to an air gap structure, a compensation lens, a compensation prism or a combination thereof, as long as the optical path of the two beams is equivalent, so that the two beams are projected onto the two independent physical photosensitive areas of the image sensor 120 strictly simultaneously, from the same angle, and without parallax.

[0072] Figures 2-5 Different forms of the beam-splitting structure in this embodiment are shown. Figure 2 Both the first optical path compensation component 115 and the second optical path compensation component 116 of the mid-splitting structure are air-spaced structures. Figure 3 The first optical path compensation component 115 of the beam splitting structure is a compensation prism, and the second optical path compensation component 116 is an air gap structure. Figure 4 The first optical path compensation component 115 of the beam splitting structure is an air gap structure, and the second optical path compensation component 116 is a compensation prism. Figure 5 Both the first optical path compensation element 115 and the second optical path compensation element 116 of the mid-beam splitting structure are compensation prisms.

[0073] In one possible implementation, the image sensor 120 employs a common area array CMOS / CCD, such as the Sony IMX series, with a resolution of 1080P / 4K. The original effective imaging area of ​​the image sensor 120 is divided into two independent physical photosensitive areas: a bright imaging area and a dark imaging area. Bright channel images and dark channel images are acquired separately, maximizing the utilization of the photosensitive area of ​​the image sensor 120.

[0074] By cooperating with the beam splitter structure and the image sensor 120, two images with different brightness levels, non-overlapping, and high clarity can be acquired using a single image sensor 120. This allows for the acquisition of more optical information through a compact structure, providing a more favorable foundation for subsequent image enhancement. Furthermore, it should be noted that this embodiment uses a single-frame, single-exposure, and synchronous acquisition model to acquire the bright channel image and the dark channel image. The two images share a set of exposure parameters (exposure time T and gain G). The output is a single-frame stitched dual-image data stream with a lossless frame rate of 30-60fps and a latency of <10ms; and the dual images are read out independently, without crosstalk or aliasing.

[0075] In step S200, the core of the linked exposure control is physical perception. Traditional automatic exposure (AE) algorithms only perform "blind adjustment" of the image through grayscale statistics. However, the linked exposure control in this embodiment establishes a physical model based on brightness-signal-noise ratio through the spectral ratio K of the hardware structure. Before exposure is executed, the optimal exposure parameters (exposure time T and gain G) of the next frame are directly predicted through the dual-region brightness statistics and dual constraints of the previous frame image, thereby obtaining a high-quality dual-channel image.

[0076] In step S200, the bright channel image and dark channel image of the current frame are first independently metered to obtain dual-region brightness statistics. The dual-region brightness statistics include the proportion of low-brightness pixels in the bright channel image. and average brightness And the peak brightness of the effective pixels in the dark channel image. Specifically, dual-region brightness statistics include the following steps:

[0077] S211, Bright Channel Statistics: Statistics on the proportion of low-brightness pixels in the bright channel image. and average brightness The formula is as follows:

[0078]

[0079] = in bright channel average brightness

[0080] In this embodiment, the effective dark pixel value range in the bright channel can be [0, 80], while [10, 80] is used to reduce the impact of extremely dark noise. Similarly, the effective pixel value range in the bright channel can be [0, 255], while [10, 255] is used to reduce the impact of extremely dark noise. The value ranges of these two values ​​can also be dynamically adjusted according to the clinical scenario (e.g., vascular mode, mucosal mode) and the data bit width and depth.

[0081] The above and It can serve as a basis for identifying whether the bright channel image is normal, whether there are extreme cases of excessive brightness or darkness, and whether the bright areas need underexposure to improve, thus preventing the loss of details in the dark areas of the bright channel.

[0082] S212, Dark Channel Statistics: Statistical analysis of the peak brightness of effective pixels in the dark channel image. , This equals the average brightness of the top A% of the brightest pixels in the dark channel. It's used to determine if the brightness in the dark areas is sufficient, whether there is overexposure, and whether noise is controllable. It can serve as a basis for constraining details in dark areas, ensuring that the texture of dark tissues is visible. The value of A can range from [0.001, 10] and can be dynamically adjusted according to clinical scenarios (such as vascular mode, mucosal mode). In this embodiment, it is preferably 1.

[0083] After obtaining the brightness statistics for both regions, dual constraints on the exposure parameters are established based on the preset splitting ratio K. These dual constraints include a dark channel saturation warning threshold. Physical gating of signal-to-noise ratio (SNR) Specifically, the process of establishing dual constraints includes the following steps:

[0084] S221. Establish a dark channel saturation warning threshold, that is, set the maximum allowable brightness threshold for the dark channel. The formula is as follows:

[0085]

[0086] in, The safety factor has a value range of [0.85, K].

[0087] It should be noted that, if > This indicates that the physical light intensity of the bright area in the dark channel is close to saturation, and even if blending is used, more highlight details cannot be recovered. In this case, the AE algorithm will force the bright channel to be protected first.

[0088] S222. Establishing a physical gating system for signal-to-noise ratio (SNR) based on a photon shot noise model. The formula is as follows:

[0089]

[0090]

[0091] in, For the scene The average brightness This refers to the inherent noise of the system (including readout noise). This is an image of the dark channel region.

[0092] It should be noted that if the calculated < 25dB (clinically usable threshold, adjustable according to system and department differences), this embodiment will forcibly lock the gain G unchanged and prioritize attempting to increase the exposure time T (if T has reached its upper limit, then maintain the current parameters), thereby avoiding the full-screen noise caused by blindly brightening 'dead black' areas in traditional AE algorithms. Traditional AE algorithms cannot predict the photon loss caused by the K value ( The signal-to-noise ratio drops by a factor of two, leading to blindly brightening images and producing "dead black" images. This embodiment uses the K value to calculate the theoretical signal-to-noise ratio limit in advance, achieving foolproof control at the physical level.

[0093] After obtaining the dual-region brightness statistics and establishing dual constraints, the exposure parameters are adjusted based on the dual-region brightness statistics and under the constraints of the dual constraints. The adjustment process includes the following judgments:

[0094] (1) If ≤ -Δ1 indicates that there is no significant underexposure in the bright channel, so further judgment is needed:

[0095] A. If < -Δ2 indicates that the brightness of the bright channel is too low. In this case, first increase T, and if T has reached its upper limit, then increase G.

[0096] B. If > +Δ2 indicates that the brightness of the bright channel is too high, and in this case, only G is reduced;

[0097] C. If -Δ2≤ +Δ2, then keep T and G unchanged;

[0098] (2) If > +Δ1 indicates underexposure in the bright channel. In this case, a further joint judgment is made by combining the dark channel information.

[0099] A. If > +Δ3 indicates that the peak brightness of the dark channel is too high, and in this case, only G should be reduced;

[0100] B. If -Δ3≤ +Δ3 indicates that there is no room for improvement in the dark area of ​​the bright channel. At this time, keep T and G unchanged.

[0101] C. If < If the value is -Δ3, it indicates that the peak brightness of the dark channel can be further increased. In this case, first increase T. If T has reached its upper limit, then increase G to further reduce the dark area in the bright channel. Since increasing the brightness of the dark channel will increase noise, this step requires further judgment based on the predicted signal-to-noise ratio.

[0102] C1, if > +Δ4 indicates that there is still room to reduce the signal-to-noise ratio. In this case, increase T first, and if T has reached its upper limit, increase G.

[0103] C2, if < -Δ4 indicates that there is no room for the signal-to-noise ratio to decrease. At this point, if T can be increased, it should be increased; otherwise, it should remain unchanged, and G should remain unchanged.

[0104] C3, if -Δ4< < +Δ4, keeping T and G constant;

[0105] (3) If -Δ1≤ ≤ If +Δ1 is applied, then T and G remain unchanged.

[0106] in, Δ1 is the threshold for the proportion of dark pixels in the bright channel (e.g., 0.15), and Δ1 is the deviation between the upper and lower limits of the threshold for the proportion of dark pixels in the bright channel (e.g., 0.02). Δ1 is the average brightness threshold of effective pixels in the bright channel (e.g., 150); Δ2 is the upper and lower deviation of the average brightness threshold of effective pixels in the bright channel (e.g., 20); Δ3 is the upper and lower deviation of the maximum allowable brightness threshold of the dark channel (e.g., 20). The threshold for predicting signal-to-noise ratio (SNR) (a clinically usable threshold) is defined as Δ4, which represents the deviation above or below the predicted SNR threshold (e.g., 1 dB). The threshold and deviation are set to ensure that the exposure converges. The chosen threshold and deviation can be dynamically adjusted based on the clinical scenario (e.g., vascular mode, mucosal mode) and the data bit width.

[0107] Through the above adjustments, the exposure time T and gain G of the next frame are output as the optimal exposure parameters for the next frame. Then, based on these exposure parameters, the bright channel image and dark channel image are obtained through step S100, and finally, image fusion is performed through step S300.

[0108] It should be noted that since the first frame of the video stream does not have a previous frame, the optimal exposure parameters cannot be calculated. Therefore, the system's default initial exposure parameters are used for this frame.

[0109] The aforementioned linked exposure control strategy can effectively reduce noise in the final enhanced image, thereby improving image clarity.

[0110] The core operation of step S300 is to fuse and reconstruct the dark channel image based on optimal exposure parameters obtained in steps S100 and S200 with the bright channel image. To ensure fusion quality, maintain real-time performance, and reduce computational overhead, the two images are geometrically aligned and extracted before fusion. Then, to ensure that the dark channel remains effective after being amplified by a preset split ratio of K, brightness correction is performed on the dark channel to assimilate the brightness of the two images and ensure color consistency between the two channels. Then, an adaptive weighted fusion algorithm is used to fuse the dark channel image and the bright channel image, reconstructing a high dynamic range image. Furthermore, to further improve the quality of the output image, the high dynamic range image is finally sent to the ISP Pipeline for processing, ultimately obtaining the enhanced endoscopic image.

[0111] Specifically, the image fusion reconstruction in step S300 includes the following steps:

[0112] S310. Dual-Image Geometric Alignment and Extraction: Since the two images originate from the same incident light source and have the same viewing angle, there is no parallax. Therefore, complex feature matching is unnecessary; only hard alignment of rows or columns based on the pre-defined physical partitions of the sensor is required. After alignment, the bright channel image is directly separated from the single-frame stitched dual-image data stream. and dark channel images This step is implemented using a pipelined architecture, with an alignment latency of less than 1 millisecond, no need for additional frame buffers, maintaining system real-time performance, and extremely low computational overhead.

[0113] S320. Brightness correction of the dark channel image based on the preset splitting ratio K:

[0114] S321. Calculate the brightness limit guarantee weight of the bright and dark channels based on the preset splitting ratio K. The formula is as follows:

[0115]

[0116]

[0117] S322, Weighting based on brightness limit For bright channel images and dark channel images Brightness limiting processing is performed, and then the dark channel image is processed based on a preset splitting ratio K. To perform K times linear optical correction, the formula is as follows:

[0118]

[0119]

[0120] Brightness limit ensures weight This ensures that the dark channel data remains valid even after Kx magnification, providing sufficient detail information in the bright areas. Furthermore, if the dark channel's brightness saturation threshold is greater than 255 / K, the dark channel will be overexposed after Kx magnification. To ensure that the dark channel data before Kx magnification provides complete detail information, the brightness of both the bright and dark channels is scaled proportionally.

[0121] Furthermore, multiplying the dark channel by the preset splitting ratio K compensates for the difference in light intensity. After assimilation, the two channels are theoretically completely identical, but in practice, their respective dynamic characteristics are preserved. If K value correction is not performed, color difference will occur at the junction of the dark and bright channels (because the splitting ratios of R / G / B may be slightly different, or the sensor response may be non-linear). K value assimilation is to ensure the color consistency of the bright and dark channels.

[0122] S330, Dual-zone adaptive weight mapping:

[0123] According to the illuminated channel The brightness values ​​are used to divide the bright channel image into three layers: a highlight layer, a mid-brightness layer, and a low-brightness layer. Then, the weights of different layers in the bright channel image are calculated. The formula is as follows:

[0124]

[0125] Then, the weights of the images by channel are obtained based on the weights of the brightness channel image. The formula is as follows:

[0126]

[0127] It should be noted that the highlight layer fully utilizes the dark channel to suppress overexposure; the mid-brightness layer features a linear transition to ensure a natural look; and the low-brightness layer fully utilizes the bright channel, primarily for enhancing details.

[0128] In addition, the segmentation points of the high, medium and low layers (80 and 220 in the above formula) and the light and dark weights of each layer can be dynamically adjusted according to the clinical scenario (such as vascular mode, mucosal mode).

[0129] S340, pixel-by-pixel weighted fusion, the formula is as follows:

[0130]

[0131] The fused image outputs a high dynamic range image. .

[0132] S350 sends the high dynamic range image into the ISP Pipeline for processing and outputs the final enhanced image. The ISP Pipeline includes, but is not limited to, bad pixel correction (DPC), black pixel flattening correction (BLC), lens shading correction (LSC), automatic white balance (AWB), demosaic, and color correction (CCM). This step can further enhance image quality.

[0133] This application embodiment also provides an endoscopic HDR imaging system, the system comprising:

[0134] The beam splitting imaging module controls the incident light to be split into two beams by a beam splitting structure according to a preset beam splitting ratio K, which are then projected onto two independent physical photosensitive areas of the image sensor. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously.

[0135] The linked exposure control module first independently meters the bright and dark channels of the current frame to obtain dual-region brightness statistics. Then, based on a preset splitting ratio K, it establishes dual constraints on the exposure parameters. Finally, based on the dual-region brightness statistics and under the constraints, it adjusts the exposure parameters and outputs the optimal exposure parameters for the next frame.

[0136] The image fusion and reconstruction module fuses the dark channel image and the bright channel image pixel by pixel to generate a high dynamic range image.

[0137] This system has all the advantages of the aforementioned endoscopic HDR imaging methods, which will not be elaborated here.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention. Technical aspects, shapes, and structures not described in detail in this invention are all well-known technologies.

Claims

1. An endoscope HDR imaging method applied to an endoscope imaging device comprising a light splitting structure and an image sensor, characterized in that, The imaging method includes: S100, Spectral imaging: The incident light is controlled to be split into two paths by the spectral structure according to a preset spectral ratio K, and projected onto two independent physical photosensitive areas of the image sensor respectively. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously. S200, Linked Exposure Control: First, the bright channel image and the dark channel image are independently metered to obtain dual-region brightness statistics. Then, based on the preset splitting ratio K, dual constraints are established for the exposure parameters. Finally, based on the dual-region brightness statistics and under the constraints of the dual constraints, the exposure parameters are adjusted to output the optimal exposure parameters for the next frame acquisition. S300, Image fusion and reconstruction: The dark channel image and the bright channel image are fused pixel by pixel to generate a high dynamic range image.

2. The endoscopic HDR imaging method of claim 1, wherein, The beam splitting structure includes an optical adapter, a beam splitting prism, a reflecting prism, a first optical path compensation device, and a second optical path compensation device. The incident light is first focused and imaged by the optical adapter, and then transmitted to the beam splitting prism. The beam splitting prism splits the light beam into a first optical path and a second optical path. The first optical path is projected onto one of the photosensitive areas of the image sensor after passing through the first optical path compensation device. The second optical path is projected onto the reflecting prism and deflected, and then, after passing through the second optical path compensation device, it is projected onto another photosensitive area of ​​the image sensor.

3. The endoscopic HDR imaging method of claim 2, wherein, The first optical path compensation component and the second optical path compensation component are compensation prisms or air gap structures.

4. The endoscopic HDR imaging method of claim 1, wherein, Independent photometric measurements are performed on the bright channel image and the dark channel image to obtain dual-region brightness statistics, including: S211, Bright Channel Statistics: Statistics on the proportion of low-brightness pixels in the bright channel image. and average brightness The formula is as follows: = in bright channel average brightness The effective dark pixel value range in the bright channel is [0, 80]; the effective pixel value range in the bright channel is [0, 255]. S212, dark channel statistics: statistics of the peak luminance of the valid pixels in the dark channel image , is equal to the mean of the luminance of the A% highest luminance pixels in the dark channel.

5. The endoscopic HDR imaging method of claim 4, wherein, The value range of A is [0.001, 10].

6. The endoscopic HDR imaging method of claim 4, wherein, The exposure parameters are subject to dual constraints based on the preset spectral ratio K, including: S221, establish a dark channel saturation early warning threshold, that is, set the maximum brightness threshold allowed by the dark channel The formula is as follows: wherein, K is a safety factor, having a value in the range [0.85, K]; S222, Establish physical gating for signal-to-noise ratio (SNR) The formula is as follows: wherein, is the average luminance of the scene , is the system inherent noise, is the dark channel image.

7. The endoscopic HDR imaging method of claim 1, wherein, In step S300, before fusing the dark channel image and the bright channel image pixel by pixel, the dark channel image and the bright channel image are geometrically aligned and extracted.

8. The endoscopic HDR imaging method of claim 1, wherein, In step S300, before fusing the dark channel image and the bright channel image pixel by pixel, the brightness of the dark channel image is corrected based on the preset spectral ratio K.

9. The endoscopic HDR imaging method of claim 8, wherein, Brightness correction of the dark channel image based on the preset spectral ratio K includes: S321, based on the preset light splitting ratio K, calculate the light and dark channel brightness limit protection weight The formula is as follows: S322, Weighting based on brightness limit Brightness limiting processing is performed on the bright channel image and the dark channel image. Then, based on the preset splitting ratio K, the dark channel image is... To perform K times linear optical correction, the formula is as follows: in, Represents the bright channel image. Dark channel image.

10. The endoscopic HDR imaging method according to any one of claims 7-9, characterized in that, In step S300, an adaptive weighted fusion algorithm is used to fuse the dark channel image and the bright channel image pixel by pixel.

11. An endoscopic HDR imaging system, characterized in that, include: The beam splitting imaging module controls the incident light to be split into two beams by the beam splitting structure according to a preset beam splitting ratio K, and projected onto two independent physical photosensitive areas of the image sensor respectively. Within a single frame exposure cycle, the bright channel image and dark channel image of the current frame are acquired simultaneously. The linkage exposure control module first performs independent light metering on the bright channel image and the dark channel image to obtain dual-region brightness statistics. Then, it establishes dual constraints on the exposure parameters based on the preset splitting ratio K. Finally, it adjusts the exposure parameters according to the dual-region brightness statistics and under the constraints of the dual constraints, and outputs the optimal exposure parameters for the next frame acquisition. as well as The image fusion and reconstruction module fuses the dark channel image and the bright channel image pixel by pixel to generate a high dynamic range image.

Images