Endoscopic device and endoscopic image processing method

Abstract

The present application relates to the field of medical technologies, and in particular to an endoscopic device and an endoscopic image processing method, so as to provide a method for solving the problem of overexposure in an endoscopic image. The method specifically comprises: receiving endoscopic image data sent by a camera main unit; when an endoscopic image corresponding to the endoscopic image data includes an over-bright pixel point, determining a light-reduction exposure rate on the basis of a brightness value of the over-bright pixel point, reducing the brightness of the over-bright pixel point in the endoscopic image on the basis of the light-reduction exposure rate, and generating a light-reduced endoscopic image; and on the basis of the light-reduced endoscopic image, displaying a target endoscopic image in a display interface.

Description

An endoscopic device and an endoscopic image processing method

[0001] Cross-references

[0002] This application claims priority to Chinese Patent Application No. 202410993742.2, filed on July 23, 2024, entitled "An Endoscopic Device and an Endoscopic Image Processing Method", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of medical technology, and in particular to an endoscope device and an endoscope image processing method. Background Technology

[0004] The rapid development of modern medical technology has made endoscopes a commonly used medical device in assisting medical diagnosis.

[0005] However, due to the limitations of endoscopic lighting and the complexity of internal human tissue structures, the images presented often exhibit localized overexposure. In areas directly illuminated by a light source, excessive light intensity can create overexposed areas, causing details in the image to be obscured by highlights and become unclear. Furthermore, water droplets flushed into the endoscope's flushing port or mucus from the inner wall of the cavity can adhere to the cavity wall, creating reflections and resulting in bright white areas in the image.

[0006] Therefore, existing endoscopic images suffer from overexposure. Summary of the Invention

[0007] To address the problems in the prior art, this application provides an endoscope device and an endoscope image processing method to solve the overexposure problem in endoscope images.

[0008] In a first aspect, an endoscope device includes a camera host and a display; the display is used for:

[0009] Receive endoscopic image data sent by the camera host, the endoscopic image data being generated based on light reflected from human tissue;

[0010] When the endoscope image corresponding to the endoscope image data contains overly bright pixels, the light reduction exposure rate is determined based on the brightness value of the overly bright pixels, and the brightness of the overly bright pixels in the endoscope image is reduced based on the light reduction exposure rate to generate a light-reduced endoscope image.

[0011] The target endoscope image is displayed on the display interface based on the reduced-exposure endoscope image; the target endoscope image does not contain overexposed areas.

[0012] In one possible implementation, the display is specifically used for:

[0013] Extract overly bright pixels from the endoscope image; the overly bright pixels refer to pixels in the endoscope image whose illumination value is greater than a set overexposure threshold;

[0014] Based on the channel values ​​of multiple color channels of the overbright pixel, the brightness value of the overbright pixel is determined, and an overbrightness brightness map is generated based on the brightness value of the overbright pixel.

[0015] Based on the overbrightness map, the image information entropy corresponding to the overbrightness map is obtained, and based on the image information entropy, the light reduction exposure rate is determined.

[0016] In one possible implementation, the display is specifically used to determine the target endoscopic image in the following manner:

[0017] When the endoscope image corresponding to the endoscope image data contains excessively dark pixels, the brightness enhancement exposure rate is determined based on the brightness value of the excessively dark pixels, and the brightness value of the excessively dark pixels in the endoscope image is increased based on the brightness enhancement exposure rate to generate a brightened endoscope image.

[0018] The endoscope image after light reduction and the endoscope image after light enhancement are fused to obtain the target endoscope image; the target endoscope image does not contain underexposed areas.

[0019] In one possible implementation, the display is further used for:

[0020] The first spatial feature map is obtained by performing local feature enhancement on the light-reduced endoscope image through a spatial attention module.

[0021] The enhanced endoscope image is subjected to global feature enhancement by the channel attention module to obtain the first channel feature map.

[0022] The first spatial feature map and the enhanced endoscope image are multiplied to obtain the second spatial feature map; and the first channel feature map and the reduced endoscope image are multiplied to obtain the second channel feature map.

[0023] The second spatial feature map and the second channel feature map are added together to obtain the target endoscope image.

[0024] In one possible implementation, the display is further used for:

[0025] In the spatial attention module, average pooling is performed on each channel corresponding to the reduced-light endoscope image along the channel axis to obtain the first intermediate feature map;

[0026] The first spatial feature map is obtained by performing convolution processing on the first intermediate feature map.

[0027] In one possible implementation, the display is further used for:

[0028] In the channel attention module, average pooling is performed on each channel corresponding to the enhanced endoscope image to obtain a second intermediate feature map.

[0029] The second intermediate feature map is convolved to obtain the first channel feature map.

[0030] Secondly, embodiments of this application provide an endoscopic image processing method, applied to a display in an endoscopic device, the method comprising:

[0031] Receive endoscopic image data sent by the camera host, the endoscopic image data being generated based on light reflected from human tissue;

[0032] When the endoscope image corresponding to the endoscope image data contains overly bright pixels, the light reduction exposure rate is determined based on the brightness value of the overly bright pixels, and the brightness of the overly bright pixels in the endoscope image is reduced based on the light reduction exposure rate to generate a light-reduced endoscope image.

[0033] The target endoscope image is displayed on the display interface based on the reduced-exposure endoscope image; the target endoscope image does not contain overexposed areas.

[0034] Thirdly, embodiments of this application provide an endoscopic image processing apparatus, comprising:

[0035] A receiving module is used to receive endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected from human tissue;

[0036] The generation module is used to determine the light reduction exposure rate based on the brightness value of the excessively bright pixels when the endoscope image corresponding to the endoscope image data contains excessively bright pixels, and to reduce the brightness of the excessively bright pixels in the endoscope image based on the light reduction exposure rate to generate a light-reduced endoscope image.

[0037] The display module is used to display the target endoscope image in the display interface based on the reduced-light endoscope image; the target endoscope image does not contain overexposed areas.

[0038] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in the second aspect.

[0039] This application provides an endoscope device and an endoscope image processing method. By identifying overly bright pixels in the endoscope image, determining the light reduction exposure rate based on the brightness value of the overly bright pixels, and reducing the brightness of the overly bright pixels in the endoscope image by the light reduction exposure rate, the target endoscope image displayed on the display interface does not contain overexposed areas. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 is a schematic diagram of the structure of an endoscope device provided in an embodiment of this application;

[0042] Figure 2 is a schematic diagram of an endoscopic examination provided in an embodiment of this application;

[0043] Figure 3 is a flowchart of an endoscopic image processing method provided in an embodiment of this application;

[0044] Figure 4 is a schematic diagram of setting an image contrast enhancement process according to an embodiment of this application;

[0045] Figure 5 is a schematic diagram of a target endoscope image provided in an embodiment of this application;

[0046] Figure 6 is a flowchart of a method for determining the light reduction exposure rate according to an embodiment of this application;

[0047] Figure 7 is a flowchart of generating an enhanced endoscope image according to an embodiment of this application;

[0048] Figure 8 is a flowchart of determining the enhancement exposure rate according to an embodiment of this application;

[0049] Figure 9 is a flowchart of a fusion of a reduced-brightness endoscope image and a brightened endoscope image provided in an embodiment of this application;

[0050] Figure 10 is a schematic diagram of a fused endoscope image after light reduction and an endoscope image after light enhancement provided in an embodiment of this application;

[0051] Figure 11 is a schematic diagram of the working principle of a spatial attention module provided in an embodiment of this application;

[0052] Figure 12 is a schematic diagram of obtaining a first intermediate feature map according to an embodiment of this application;

[0053] Figure 13 is a schematic diagram of the working principle of a channel attention module provided in an embodiment of this application;

[0054] Figure 14 is a schematic diagram of obtaining a second intermediate feature map according to an embodiment of this application;

[0055] Figure 15 is an overall flowchart provided by an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0057] Furthermore, in the description of the embodiments of this application, unless otherwise stated, "and" means "or", for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0058] Specifically, in the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0059] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] To facilitate understanding of the endoscopic device and endoscopic image processing method provided in the embodiments of this application, some terms used in the embodiments of this application are explained below for the benefit of those skilled in the art:

[0061] The meaning of bins in a histogram: Calculating a color histogram requires dividing the color space into several small color intervals, i.e., bins of the histogram. The color histogram is obtained by calculating the number of pixels of color within each small interval. The more bins there are, the stronger the color resolution of the histogram, but the more burden it increases on the computer. That is (the 10 vertical regions in the above image, each vertical region is called a bin).

[0062] Image information entropy is defined as the magnitude of image information extracted based on selected features within a given spatial dimension.

[0063] To make the objectives, technical solutions, and beneficial effects of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] Figure 1 illustrates a schematic diagram of an endoscope device according to an embodiment of this application, wherein the endoscope device includes a cold light source, a camera assembly, and a display.

[0065] The camera module can be used to capture images of endoscopic examinations and surgeries to obtain visual data. Its key components may include one or more of the following: a camera, buttons, a camera unit, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a video cable, an optical adapter, an objective lens field of view, a lens rod, a communication port, and a video output interface. The camera unit can be connected to a monitor via the video output interface and to a cold light source unit via the communication port.

[0066] In this embodiment of the application, the objective lens field of view, lens rod, optical adapter, button, CMOS and camera in the key components of the camera assembly can also be collectively referred to as an endoscope camera.

[0067] The monitor can be used to display endoscopic images acquired and generated by the camera component.

[0068] Cold light sources can be used to provide illumination for endoscopes during endoscopic examinations and surgeries; their key components typically include: a cold light source main unit, a beam guide, a communication port, and a cold light source output interface.

[0069] It should be understood that the schematic diagram of the endoscopic device shown in Figure 1 is merely an example, and the endoscopic device may have more or fewer components than shown in Figure 1, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0070] In practice, endoscopic examinations or surgeries are primarily assisted by dynamic video. After the endoscope is inserted into the body, the camera on the endoscope captures and displays images of the body. As the camera moves, the area it illuminates is displayed on a monitor connected to the endoscope, providing a clear view of the environment surrounding the lesion. Surgical instruments can be moved and manipulated in a three-dimensional, intuitive manner. For example, as shown in Figure 2, medical personnel can operate the endoscope to capture endoscopic images of the body.

[0071] The foregoing provides many different implementation methods or examples for implementing different structures of this application. To simplify the content of the embodiments of this application, only the components and settings of specific examples are described above. Of course, these are merely examples and are not intended to limit this application.

[0072] Next, as shown in Figure 3, a flowchart of an endoscopic image processing method provided in an embodiment of this application is presented. This method can be applied to the processor of the display of the endoscopic device shown in Figure 1. As shown in Figure 3, the method may include the following steps:

[0073] Step S301: Receive endoscopic image data sent by the camera host. The endoscopic image data is generated based on the light reflected from human tissue.

[0074] Step S302: When the endoscope image corresponding to the endoscope image data contains overly bright pixels, the light reduction exposure rate is determined based on the brightness value of the overly bright pixels, and the brightness of the overly bright pixels in the endoscope image is reduced based on the light reduction exposure rate to generate a light-reduced endoscope image.

[0075] Step S303: Display the target endoscope image in the display interface based on the reduced-exposure endoscope image; the target endoscope image does not contain overexposed areas.

[0076] It should be noted that when the monitor is set to perform image contrast enhancement processing, the monitor executes the endoscopic image processing method of this application.

[0077] In this embodiment of the application, the display responds to the user's activation operation triggered by the user operation interface to enable image contrast enhancement processing. The display is then set to enable image contrast enhancement processing, that is, the display is set to perform image contrast enhancement processing.

[0078] Optionally, the user operation interface in this embodiment can be a control in the display interface or a physical button on the camera host.

[0079] Figure 4 illustrates a schematic diagram of setting image contrast enhancement processing according to an embodiment of this application. As shown in the left image of Figure 4, the user triggers an entry operation through a user interface to access the image contrast enhancement processing settings interface. For example, the user triggers the entry operation for "Image Contrast Enhancement" as shown in the left image of Figure 4. The display responds to the user-triggered entry operation and displays the settings interface for setting image contrast enhancement processing, as shown in the right image of Figure 4. This settings interface includes an on / off control, used to enable and disable image contrast enhancement processing, respectively.

[0080] For example, when a user triggers the enable operation for image contrast enhancement processing using the enable control in the settings interface, the monitor is set to enable image contrast enhancement processing. As shown in the right image of Figure 4, the monitor is set to enable image contrast enhancement processing.

[0081] Figure 5 shows a schematic diagram of a target endoscope image according to an embodiment of this application. The left image in Figure 5 is the original endoscope image, and the right image is the obtained target endoscope image.

[0082] The original endoscopic image contains overexposed areas, while the target endoscopic image shown does not contain overexposed areas.

[0083] Specifically, the endoscopic image processing method of this application will be described in detail below:

[0084] In step S302, based on the endoscope image corresponding to the received endoscope image data, the overly bright pixels in the endoscope image are determined by comparing the illumination value corresponding to each pixel in the endoscope image with a set overexposure threshold.

[0085] In this embodiment, an overexposed pixel refers to a pixel in an endoscopic image whose illumination value is greater than a set overexposure threshold.

[0086] It should be noted that the illumination value is the numerical value of each pixel in the scene illumination map corresponding to the endoscopic image.

[0087] Optionally, in this embodiment of the application, excessively bright pixels in the endoscopic image are determined using Formula 1: Q1={P(x)|T(x)>0.7}——Formula 1

[0088] Where Q1 represents the set of overexposed pixels, P(x) represents the endoscope image, T(x) represents the scene illumination map corresponding to the endoscope image, and 0.7 is the set overexposure threshold.

[0089] In other words, the overly bright pixels in Q1 obtained by Formula 1 are the pixels in the endoscopic image with an illumination value greater than 0.7.

[0090] It should be noted that the process of determining the scene illumination map corresponding to the endoscopic image is a prior art in this field. The scene illumination map corresponding to the endoscopic image can be obtained through illuminance map estimation technology, so this application will not elaborate on it here.

[0091] In this embodiment, after identifying the excessively bright pixels in the endoscopic image, if the endoscopic image does not contain excessively bright pixels, the endoscopic image is processed based on a preset light reduction exposure rate to generate a light-reduced endoscopic image. If the endoscopic image contains excessively bright pixels, the light reduction exposure rate is determined based on the brightness value of the excessively bright pixels, and the brightness of the excessively bright pixels in the endoscopic image is reduced based on the light reduction exposure rate to generate a light-reduced endoscopic image.

[0092] For example, if the endoscopic image does not contain overly bright pixels, the exposure reduction rate can be preset to 1.

[0093] As shown in Figure 6, if the endoscopic image contains excessively bright pixels, this embodiment of the application provides a flowchart for determining the light reduction exposure rate, and the specific steps are as follows:

[0094] Step S601: Extract overly bright pixels from the endoscopic image.

[0095] Among them, overexposed pixels refer to pixels in the endoscopic image whose illumination value is greater than the set overexposure threshold.

[0096] Step S602: Based on the channel values ​​of multiple color channels of the overbright pixel, determine the brightness value of the overbright pixel, and generate an overbrightness brightness map based on the brightness value of the overbright pixel.

[0097] It should be noted that since the brightness of images varies significantly under different exposures, while the colors remain basically the same, this application only considers the brightness component when determining the exposure reduction rate.

[0098] In this embodiment of the application, taking the multiple color channels of overly bright pixels in the endoscope image as RGB color channels as an example, the brightness value of the corresponding overly bright pixel is determined based on the channel value of the RGB color channel corresponding to each overly bright pixel in the endoscope image.

[0099] Optionally, in this embodiment of the application, the excessive brightness map is determined using Formula 2:

[0100] B1 represents the overbrightness map, which contains the brightness value of each overbright pixel; This represents the channel value corresponding to the R (red) channel. This represents the channel value corresponding to the G (green) channel. This indicates the channel value corresponding to channel B (blue).

[0101] Step S603: Based on the overbrightness map, obtain the image information entropy corresponding to the overbrightness map, and determine the light reduction exposure rate based on the image information entropy.

[0102] It should be noted that well-exposed images have higher visibility and provide richer information than overexposed images. Different exposure reduction rates affect image visibility; therefore, the optimal exposure reduction rate used in this application should enable the image to provide the maximum amount of information. Since maximizing image information entropy represents the maximum amount of information provided by the image, this application determines the optimal exposure reduction rate by calculating the maximum image information entropy.

[0103] First, this application introduces image information entropy using Formula 3:

[0104] Where H(B1) represents the image information entropy; p i It is the i-th bin of the histogram of B1, used for calculation The number of data points in the bin, where N is the number of bins (N is usually set to 256).

[0105] It should be noted that the bins in the histogram represent color ranges.

[0106] For example, when i is 3, p i Characterization The number of pixels corresponding to the interval, that is, the number of pixels corresponding to the brightness value interval [3,4).

[0107] In this application, an unknown quantity k1 is used to represent the light reduction exposure rate. Based on the light reduction exposure rate k1, the overbrightness image B1 is processed to obtain a formula for characterizing the overbrightness image after light reduction based on k1, as shown in Formula 4.

[0108] Where g(B1,k1) represents the overbrightness map after light reduction based on k1, and a and b represent fixed camera parameters, for example, a = 0.3293, b = 1.1258.

[0109] Based on the maximum image information entropy of g(B1,k1), the value of the unknown quantity k1 is obtained, which is the light reduction exposure rate corresponding to the endoscopic image in this application.

[0110] Optionally, in this embodiment of the application, the light reduction exposure rate corresponding to the endoscopic image is determined by Formula 5:

[0111] in, H(g(B1,k1)) represents the reduced exposure rate corresponding to the endoscopic image, and H(g(B1,k1)) represents the image information entropy corresponding to the overbrightness map after reduction based on k1.

[0112] The light reduction exposure rate of the endoscope image is obtained through Formula 5. Based on the light reduction exposure rate, the brightness of overly bright pixels in the endoscope image is reduced to generate the light-reduced endoscope image.

[0113] In step S303, the target endoscope image is displayed on the display interface based on the reduced-exposure endoscope image; the target endoscope image does not contain overexposed areas.

[0114] Optionally, before displaying the target endoscope image, this embodiment of the application also needs to process the endoscope image based on the brightening exposure rate to generate a brightened endoscope image, and then fuse the darkened endoscope image and the brightened endoscope image to obtain the target endoscope image.

[0115] It should be noted that in areas directly illuminated by a light source, excessive light intensity can cause overexposure, resulting in details in the image being obscured by highlights and becoming unclear. Conversely, at the edges of the image or in areas not directly illuminated by a light source, the dim light can cause the displayed image to be too dark and unclear. Therefore, this application can also detect underexposure in endoscopic images and brighten underexposure areas.

[0116] As shown in Figure 7, a flowchart of generating an enhanced endoscope image according to an embodiment of this application is presented, and the specific steps are as follows:

[0117] Step S701: Based on the endoscopic image corresponding to the received endoscopic image data, determine the overly dark pixels in the endoscopic image by comparing the illumination value corresponding to each pixel in the endoscopic image with a set overly dark threshold.

[0118] In this embodiment, an overly dark pixel refers to a pixel in an endoscopic image whose illumination value is less than a set overexposure threshold.

[0119] It should be noted that the illumination value is the numerical value of each pixel in the scene illumination map corresponding to the endoscopic image.

[0120] Optionally, in this embodiment of the application, excessively dark pixels in the endoscopic image are determined using Formula Six: Q2={P(x)|T(x)<0.5}——Formula Six

[0121] Where Q2 represents the set of overly dark pixels, P(x) represents the endoscope image, T(x) represents the scene illumination map corresponding to the endoscope image, and 0.5 is the set overly dark threshold.

[0122] In other words, the dark pixels included in Q2 obtained by Formula 6 are the pixels in the endoscopic image with an illumination value of less than 0.5.

[0123] Step S702: Determine if there are any excessively dark pixels; if yes, proceed to step S703; if no, proceed to step S704.

[0124] Step S703: When the endoscope image corresponding to the endoscope image data contains dark pixels, the brightness enhancement exposure rate is determined based on the brightness value of the dark pixels, and the brightness value of the dark pixels in the endoscope image is increased based on the brightness enhancement exposure rate to generate the enhanced endoscope image.

[0125] As shown in Figure 8, a flowchart for determining the enhancement exposure rate according to an embodiment of this application is presented below, with specific steps as follows:

[0126] Step S801: Extract overly dark pixels from the endoscopic image.

[0127] Among them, excessively dark pixels refer to pixels in the endoscopic image whose illumination value is less than the set excessively dark threshold.

[0128] Step S802: Based on the channel values ​​of multiple color channels of the dark pixel, determine the brightness value of the dark pixel, and generate a dark brightness map based on the brightness value of the dark pixel.

[0129] The principle of generating the dark brightness map can be found in Formula 2, and will not be elaborated here.

[0130] Step S803: Based on the dark brightness map, obtain the image information entropy corresponding to the dark brightness map, and determine the brightening exposure rate based on the image information entropy.

[0131] In this application, an unknown quantity k2 represents the brightening exposure rate. Based on the brightening exposure rate k2, the dark brightness map B2 is processed to obtain a formula for characterizing the dark brightness map after brightening based on k2. Based on the maximum image information entropy corresponding to the dark brightness map after brightening, the brightening exposure rate corresponding to the endoscope image in this application is obtained.

[0132] Optionally, in this embodiment of the application, the illumination enhancement rate corresponding to the endoscopic image is determined using Formula Seven:

[0133] in, represents the exposure rate of the endoscopic image; H(g(B2,k2)) represents the image information entropy of the dark brightness map after k2 enhancement. g(B2,k2) represents the darker brightness map after enhancement based on k2, and B2 represents the darker brightness map corresponding to the endoscopic image.

[0134] The brightening exposure rate of the endoscope image is obtained through Formula 7. Based on the brightening exposure rate, the brightness of the dark pixels in the endoscope image is increased to generate the brightened endoscope image.

[0135] Step S704: When the endoscope image corresponding to the endoscope image data does not contain overly dark pixels, the endoscope image is processed based on a preset brightening exposure rate to generate a brightened endoscope image.

[0136] For example, the preset brightening exposure rate can be set to 1.

[0137] It should be noted that the embodiments of this application do not limit the order in which the endoscope image after light reduction is generated and the endoscope image after light enhancement is generated; the two can be executed simultaneously.

[0138] Optionally, embodiments of this application can execute the above process through a camera response model to output the generated endoscope image after light reduction and the endoscope image after light enhancement.

[0139] In this embodiment, the camera response model can be established through the following process:

[0140] Generally, camera manufacturers employ non-linear in-camera processing techniques such as white balance and de-pixelation to improve the visual quality of captured images. These non-linear processes can be modeled as Equation 8: P = f(E) — Equation 8

[0141] Where P represents the endoscopic image, which includes the channel values ​​corresponding to the endoscopic image in multiple color channels; E represents the image irradiance (also known as scene irradiance), which is the illumination signal obtained by collecting the light reflected back from human tissue in the scene using optical principles; and f is the nonlinear camera response function (CRF).

[0142] The mapping function between two images with only different exposures is called the Brightness Transform Function (BTF). Given an exposure rate k... i And the brightness transformation function g can map the input image P to the i-th image in the exposure set as P. i =g(P,k i ).

[0143] Then, CRF can be obtained by solving the following coparametric equation to obtain Formula 9: g(f(E),ki )=f(kiE)——Formula Nine

[0144] The camera response model consists of two parts: the CRF model and the BTF model. The parameters of the CRF model are determined solely by the camera, while the parameters of the BTF model are determined by both the camera and the exposure rate.

[0145] To estimate the BTF model, two images, P1 and P2, with only different exposures are selected. The histogram of the underexposed image is mainly concentrated in the low-brightness areas. If the pixel values ​​are linearly magnified before traditional gamma correction, the resulting image is very close to the truly well-exposed image. Therefore, the BTF model can be described using a two-parameter function as shown in Equation 10: P2 = g(P1,k) = βP1 γ Formula 10

[0146] Here, β and γ are parameters related to the exposure rate k in the BTF model. For general cameras, different color channels have similar model parameters because their response curves are approximately the same.

[0147] In the BTF model of this application, β and γ are determined by the camera parameters and the exposure ratio k. To find the relationship between them, we need to obtain the corresponding CRF model. This is achieved by solving the following coparametric equation (substituting g = βf). γ From the formula g(f(E),k)=f(kE), the CRF model can be derived, as shown in Formula 11: f(kE)=βf(E) γ Formula Eleven

[0148] For the closed-form solution of f, refer to Formula Twelve:

[0149] Where a and b are the model parameters when γ ≠ 1: a = log k γ,

[0150] c is the model parameter when γ = 1: c = log k β.

[0151] Equation 12 derives two CRF models. When γ = 1, the CRF model is a power function, while the BTF model is a simple linear function. Some camera manufacturers design f as a gamma curve, which can better fit these camera characteristics. When γ ≠ 1, the CRF model is a two-parameter function, and the BTF model is a nonlinear function. Since the BTF of most cameras is nonlinear, the case of γ ≠ 1 is mainly considered.

[0152] From the above model parameter a = log k γ, It can be seen that the parameters of the BFT model are: γ=k a .

[0153] In summary, given an input image P and an exposure rate k i We can obtain only i images with different exposures from our BTF model, as shown in Formula 13:

[0154] Among them, P i Indicates based on exposure rate k i The image after processing the input image P.

[0155] Assuming no information about the camera is provided, a fixed camera parameter (a = 0.3293, b = 1.1258) is used, which is suitable for most cameras.

[0156] The camera response model is established through the above process. In use, the endoscopic image corresponding to the received endoscopic image data is input into the camera response model. The camera response model then determines the enhancement and reduction exposure rates corresponding to the endoscopic image, and processes the endoscopic image based on these rates, outputting a reduced-light endoscopic image and an enhanced-light endoscopic image.

[0157] As shown in Figure 9, this application embodiment presents a flowchart of fusing a reduced-brightness endoscope image and a brightened endoscope image. The specific steps are as follows:

[0158] Step S901: The local feature enhancement of the endoscopic image after light reduction is performed by the spatial attention module to obtain the first spatial feature map.

[0159] Optionally, in this embodiment, the endoscope image after light reduction can be first convolved, and then the convolved endoscope image after light reduction can be enhanced with local features through a spatial attention module to obtain a first spatial feature map.

[0160] It should be noted that the light-reduced endoscope image after convolution processing is equivalent to a feature map.

[0161] Step S902: Perform global feature enhancement on the enhanced endoscope image through the channel attention module to obtain the first channel feature map.

[0162] Optionally, in this embodiment, the enhanced endoscope image may first undergo convolution processing, and then the enhanced endoscope image after convolution processing may undergo global feature enhancement through a channel attention module to obtain a first channel feature map.

[0163] It should be noted that the enhanced endoscope image after convolution processing is equivalent to a feature map.

[0164] Step S903: Multiply the first spatial feature map and the enhanced endoscope image to obtain the second spatial feature map; and multiply the first channel feature map and the reduced endoscope image to obtain the second channel feature map.

[0165] Step S904: Add the second spatial feature map and the second channel feature map together to obtain the target endoscope image.

[0166] Figure 10 illustrates a schematic diagram of fusing a reduced-brightness endoscope image and a brightened endoscope image according to an embodiment of this application. The brightened endoscope image and the reduced-brightness endoscope image are each convolved with a 1×1 convolution to obtain feature map M1 corresponding to the brightened endoscope image (i.e., the brightened endoscope image after convolution processing in this application) and feature map M2 corresponding to the reduced-brightness endoscope image (i.e., the reduced-brightness endoscope image after convolution processing in this application), respectively. The 1×1 convolution is used to change the number of channels, achieving cross-channel feature integration. After passing through the channel attention module and the spatial attention module, a first channel feature map M1' and a first spatial feature map M2' are obtained, respectively. Subsequently, feature map M1 and the first spatial feature map M2' are multiplied pixel-by-pixel to reweight each pixel, resulting in a second spatial feature map R1. The first channel feature map M1' and feature map M2 are multiplied pixel-by-pixel to reweight each channel, resulting in a second channel feature map R2. Finally, the second spatial feature map R1 and the second channel feature map R2 are added together to obtain the final target endoscopic image.

[0167] It should be noted that the primary function of the channel attention module is to obtain attention in terms of channels, i.e., to emphasize channel information. Channel information typically represents the global features of an image, such as color and brightness. Compared to the original endoscopic image, the image after enhancing low-brightness areas (the endoscope image after brightening in this application) has richer color information. The primary function of the spatial attention module is to obtain attention in terms of space, i.e., to emphasize spatial information. Spatial information typically represents the local features of an image, such as texture and edges. Compared to the original endoscopic image, the image after reducing the brightness of overexposed areas (the endoscope image after reducing brightness in this application) has richer edge information.

[0168] Regarding the spatial attention module in Figure 10, this application provides a schematic diagram of the working principle of a spatial attention module. As shown in Figure 11, in the spatial attention module, average pooling is performed on each channel corresponding to the reduced-light endoscope image along the channel axis to obtain a first intermediate feature map; convolution is then performed on the first intermediate feature map to obtain a first spatial feature map.

[0169] In this embodiment, the input to the spatial attention module is the de-illuminated endoscope image after convolution processing, which is the feature map M2 corresponding to the de-illuminated endoscope image.

[0170] It should be noted that if the input feature map M2 is of size H×W×c, average pooling is performed along the channel axis to obtain a first intermediate feature map of size H×W×1. Then, a 1×1 convolution is used to increase non-linearity (the convolution process of the 1×1 convolution kernel is equivalent to the calculation process of a fully connected layer; by adding a non-linear activation function, the expressive power of the model can be increased), finally obtaining a first spatial feature map containing local spatial information of each location.

[0171] For example, as shown in Figure 12, this embodiment of the present application provides a schematic diagram of obtaining a first intermediate feature map. Taking a feature map M2 of size 3×3×3 as an example, average pooling is performed on each channel in the feature map along the channel axis to obtain a first intermediate feature map of size 3×3×1.

[0172] For example, for channel values ​​1, 2, and 3 located at the same position in each channel shown in Figure 12, the value corresponding to that position in the first intermediate feature map obtained by average pooling is the average pooling result 2.

[0173] Regarding the channel attention module in Figure 10, this application provides a schematic diagram of the working principle of a channel attention module. As shown in Figure 13, in the channel attention module, average pooling is performed on each channel corresponding to the enhanced endoscope image to obtain a second intermediate feature map; convolution is then performed on the second intermediate feature map to obtain a first channel feature map.

[0174] In this embodiment, the input to the channel attention module is the enhanced endoscope image after convolution processing, which is the feature map M1 corresponding to the enhanced endoscope image.

[0175] It should be noted that if the input feature map M1 is of size H×W×c, after performing average pooling on each channel, a second intermediate feature map of size 1×1×c is obtained. A 1×1 convolution is used to increase non-linearity (the convolution process of a 1×1 convolution kernel is equivalent to the computation process of a fully connected layer; by adding a non-linear activation function, the expressive power of the model can be increased), and finally, a first channel feature map containing global semantic information of each channel is obtained.

[0176] For example, as shown in Figure 14, this embodiment of the present application provides a schematic diagram of obtaining a second intermediate feature map. Taking a feature map M1 of size 3×3×3 as an example, each channel undergoes average pooling processing to obtain a second intermediate feature map of size 1×1×3.

[0177] In this process, average pooling is performed separately for each channel, and the resulting second intermediate feature map contains the average pooling result for each channel. For example, in the second intermediate feature map shown in Figure 14, "3", "2", and "2" represent the average pooling results for the corresponding channels.

[0178] As shown in Figure 15, an overall flowchart of an embodiment of this application is presented, and the specific steps are as follows:

[0179] Step S1501: After receiving the endoscope image data, the display establishes a camera response model.

[0180] Step S1502: Determine the optimal exposure rate by using the camera response model.

[0181] Step S1503: Process the endoscope image based on the optimal brightening exposure rate to obtain an enhanced endoscope image with improved brightness in the underexposed areas.

[0182] Step S1504: Determine the optimal exposure rate for light reduction using the camera response model.

[0183] Step S1505: Process the endoscope image based on the optimal light reduction exposure rate to obtain an endoscope image after light reduction in which the overexposed areas are reduced.

[0184] Step S1506: An image fusion algorithm based on an attention mechanism is used to generate a target endoscope image.

[0185] It should be noted that, in existing technologies, the mainstream algorithms for enhancing the contrast of endoscopic images mainly include histogram equalization (HE) and Retinex algorithms. Considering the uneven distribution of elements across different gray levels, HE is widely used to improve contrast. Many extensions to HE take into account constraints such as brightness preservation and contrast limiting. However, HE-based methods often suffer from over-enhancement, leading to color distortion when used to enhance endoscopic images. Mimicking the human visual system, Retinex theory is also widely applied to image enhancement. By separating reflectance from illuminance, Retinex-based algorithms can significantly enhance details. However, these methods exhibit halo artifacts in high-contrast regions.

[0186] This application can enhance the low contrast of low-brightness areas in the original endoscope image while maintaining the contrast of well-exposed areas in the original endoscope image, and also reduce the light in overexposed areas in the original endoscope image. This application can solve the problem of how to enhance image contrast when some areas of the endoscope image are underexposed or overexposed.

[0187] Based on the same inventive concept, the endoscopic image processing method described above in this application can also be implemented by an endoscopic image processing device. The effect of this endoscopic image processing device is similar to that of the aforementioned method, and will not be described again here.

[0188] Based on the same inventive concept as the above method embodiments, this application also provides an electronic device. The principle of the electronic device in solving the problem is similar to that of the method in the above embodiments. Therefore, the implementation of the electronic device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0189] This application also provides a computer storage medium storing computer-executable instructions for implementing the endoscopic image processing method described in any embodiment of this application.

[0190] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0191] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0194] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An endoscope apparatus characterized by comprising: The endoscope device comprises a camera host and a display; the display is used for: receiving endoscope image data sent by the camera host, the endoscope image data being generated based on light reflected by human tissue; when the endoscope image corresponding to the endoscope image data contains over-bright pixel points, determining a light reduction exposure rate based on the brightness value of the over-bright pixel points, and reducing the brightness of the over-bright pixel points in the endoscope image based on the light reduction exposure rate to generate a light-reduced endoscope image; displaying a target endoscope image in a display interface based on the light-reduced endoscope image; the target endoscope image does not contain overexposure areas.

2. The endoscope apparatus of claim 1, wherein The display is specifically used for: extracting over-bright pixel points in the endoscope image; the over-bright pixel points refer to pixel points in the endoscope image whose illumination values are greater than a set overexposure threshold; determining the brightness value of the over-bright pixel points based on the channel values of multiple color channels of the over-bright pixel points, and generating an over-bright brightness map based on the brightness value of the over-bright pixel points; based on the over-bright brightness map, obtaining image information entropy corresponding to the over-bright brightness map, and determining the light reduction exposure rate based on the image information entropy.

3. The endoscope apparatus of claim 1, wherein The display is specifically used for determining a target endoscope image by the following way: when the endoscope image corresponding to the endoscope image data contains over-dark pixel points, determining a light increase exposure rate based on the brightness value of the over-dark pixel points, and increasing the brightness value of the over-dark pixel points in the endoscope image based on the light increase exposure rate to generate a light-increased endoscope image; fusing the light-reduced endoscope image and the light-increased endoscope image to obtain the target endoscope image; the target endoscope image does not contain underexposure areas.

4. The endoscope apparatus of claim 3, wherein The display is further used for: performing local feature enhancement on the light-reduced endoscope image through a spatial attention module to obtain a first spatial feature map; performing global feature enhancement on the light-increased endoscope image through a channel attention module to obtain a first channel feature map; performing multiplication operation on the first spatial feature map and the light-increased endoscope image to obtain a second spatial feature map, and performing multiplication operation on the first channel feature map and the light-reduced endoscope image to obtain a second channel feature map; adding the second spatial feature map and the second channel feature map to obtain the target endoscope image.

5. The endoscope apparatus of claim 4, wherein The display is further used for: in the spatial attention module, performing average pooling processing on each channel corresponding to the light-reduced endoscope image along a channel axis to obtain a first intermediate feature map; performing convolution processing on the first intermediate feature map to obtain the first spatial feature map.

6. The endoscope apparatus of claim 4, wherein The display is further used for: in the channel attention module, performing average pooling processing on each channel corresponding to the light-increased endoscope image respectively to obtain a second intermediate feature map; performing convolution processing on the second intermediate feature map to obtain the first channel feature map.

7. An endoscope image processing method applied to a display in an endoscope apparatus, characterized by, The method comprises: receiving endoscope image data sent by the camera host, the endoscope image data being generated based on light reflected by human tissue; When the endoscope image data corresponds to an endoscope image containing over-bright pixel points, a light reduction exposure rate is determined based on the brightness value of the over-bright pixel points, and the brightness of the over-bright pixel points in the endoscope image is reduced based on the light reduction exposure rate to generate a light-reduced endoscope image; A target endoscope image is displayed in a display interface based on the light-reduced endoscope image; the target endoscope image does not contain an overexposure area.

8. The method of claim 7, wherein, The light reduction exposure rate is determined based on the brightness value of the over-bright pixel points, including: The over-bright pixel points in the endoscope image are extracted; the over-bright pixel points refer to pixel points in the endoscope image whose illumination values are greater than a set overexposure threshold value; The brightness value of the over-bright pixel points is determined based on the channel values of multiple color channels of the over-bright pixel points, and an over-bright brightness map is generated based on the brightness value of the over-bright pixel points; The image information entropy corresponding to the over-bright brightness map is obtained based on the over-bright brightness map, and the light reduction exposure rate is determined based on the image information entropy.

9. The method of claim 7, wherein, The target endoscope image is determined in the following manner: When the endoscope image data corresponds to an endoscope image containing over-dark pixel points, an increase light exposure rate is determined based on the brightness value of the over-dark pixel points, and the brightness value of the over-dark pixel points in the endoscope image is increased based on the increase light exposure rate to generate an increase light endoscope image; The light-reduced endoscope image and the increase light endoscope image are fused to obtain the target endoscope image; the target endoscope image does not contain an underexposure area.

10. A computer readable storage medium having stored therein a computer program, characterized in that: The computer program is executed by the processor to implement any one of the methods in claims 7-9.

Images