Endoscope equipment and endoscope image processing method
By performing light reduction and light enhancement processing on the image in the endoscopic device, the problems of overexposure and underexposure in the endoscopic image are solved, the clarity and contrast of the image are improved, and more accurate diagnosis is achieved.
Patent Information
- Application Number
- CN202410993742.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Due to the limitations of lighting conditions and the complexity of human tissue structure, endoscopic images often have overexposure areas, resulting in unclear image details.
By implementing light reduction and light-enhancing processing of the image in the endoscope device, determining the brightness value of the over-bright pixel point is used to reduce the brightness of the over-exposure area, and increasing the brightness in the light-weak area, generating an area that does not include over-exposure or under-exposure. target image. The specific method includes extracting overly bright pixel points, calculating their brightness map and information entropy, determining the light reduction exposure rate, and performing light reduction processing; similarly, extracting overly dark pixel points, calculating their brightness map and information entropy, and determining the light-enhancing exposure rate, Perform light-enhancing processing, and fuse the image after light-reducing with the image after light-enhancing.
It effectively solves the problems of overexposure and underexposure in endoscopic images, improves the clarity and contrast of the images, ensures balanced exposure of images in the display interface, and significantly improves the accuracy of diagnosis.
Smart Images

Figure CN119014783A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical technology, and in particular to an endoscope device and an endoscope image processing method. Background Art
[0002] The rapid development of modern medical technology has made endoscopes gradually become a commonly used medical device in auxiliary medical diagnosis.
[0003] However, due to the limitations of endoscope lighting conditions and the complexity of the internal tissue structure of the human body, the images presented often have local areas that are too bright. In areas where the light source is directly illuminated, the excessive light intensity will produce overexposed areas, causing the details in the image to be covered by highlights and become unclear. In addition, the water droplets flushed into the endoscope flushing hole or the mucus on the inner wall of the cavity will adhere to the inner wall of the cavity to form reflections, resulting in a very bright white area in the image.
[0004] Therefore, existing endoscopic images have the problem of image overexposure. Summary of the invention
[0005] In order to solve the above-mentioned problems in the prior art, an embodiment of the present application provides an endoscope device and an endoscope image processing method, which is used to provide a method for solving the overexposure problem in the endoscope image.
[0006] In a first aspect, an endoscope device includes a camera host and a display; the display is used for:
[0007] Receiving endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected by human tissue;
[0008] When the endoscopic image corresponding to the endoscopic image data contains overbright pixels, determining a dimming exposure rate based on the brightness values of the overbright pixels, and reducing the brightness of the overbright pixels in the endoscopic image based on the dimming exposure rate to generate a dimmed endoscopic image;
[0009] A target endoscopic image is displayed in a display interface based on the light-reduced endoscopic image; the target endoscopic image does not include an overexposed area.
[0010] In a possible implementation manner, the display is specifically used for:
[0011] Extracting overbright pixels in the endoscopic image; the overbright pixels refer to pixels in the endoscopic image whose illumination values are greater than a set overexposure threshold;
[0012] Determine the brightness value of the overbright pixel based on the channel values of multiple color channels of the overbright pixel, and generate an overbright brightness map based on the brightness value of the overbright pixel;
[0013] Based on the overbrightness map, an image information entropy corresponding to the overbrightness map is obtained, and based on the image information entropy, the dimming exposure rate is determined.
[0014] In a possible implementation, the display is specifically used to determine the target endoscopic image in the following manner:
[0015] When the endoscopic image corresponding to the endoscopic image data contains too dark pixels, determine a light-enhancing exposure rate based on the brightness values of the too dark pixels, and increase the brightness values of the too dark pixels in the endoscopic image based on the light-enhancing exposure rate to generate a light-enhanced endoscopic image;
[0016] The light-reduced endoscopic image and the light-enhanced endoscopic image are fused to obtain the target endoscopic image; the target endoscopic image does not contain an underexposed area.
[0017] In a possible implementation manner, the display is further used for:
[0018] Performing local feature enhancement on the light-reduced endoscopic image through a spatial attention module to obtain a first spatial feature map;
[0019] Performing global feature enhancement on the enhanced endoscopic image through a channel attention module to obtain a first channel feature map;
[0020] Performing a multiplication operation on the first spatial feature map and the endoscopic image after light enhancement to obtain a second spatial feature map; and performing a multiplication operation on the first channel feature map and the endoscopic image after light reduction to obtain a second channel feature map;
[0021] The second spatial feature map and the second channel feature map are added to obtain the target endoscopic image.
[0022] In a possible implementation manner, the display is further used for:
[0023] In the spatial attention module, average pooling is performed on each channel corresponding to the reduced-light endoscopic image along the channel axis to obtain a first intermediate feature map;
[0024] Convolution processing is performed on the first intermediate feature map to obtain the first spatial feature map.
[0025] In a possible implementation manner, the display is further used for:
[0026] In the channel attention module, average pooling is performed on each channel corresponding to the enhanced endoscopic image to obtain a second intermediate feature map;
[0027] The second intermediate feature map is convolved to obtain the first channel feature map.
[0028] In a second aspect, an embodiment of the present application provides an endoscopic image processing method, which is applied to a display in an endoscopic device, and the method includes:
[0029] Receiving endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected by human tissue;
[0030] When the endoscopic image corresponding to the endoscopic image data contains overbright pixels, determining a dimming exposure rate based on the brightness values of the overbright pixels, and reducing the brightness of the overbright pixels in the endoscopic image based on the dimming exposure rate to generate a dimmed endoscopic image;
[0031] A target endoscopic image is displayed in a display interface based on the light-reduced endoscopic image; the target endoscopic image does not include an overexposed area.
[0032] In a third aspect, an embodiment of the present application provides an endoscopic image processing device, comprising:
[0033] A receiving module, used for receiving endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected by human tissue;
[0034] a generating module, configured to determine a light reduction exposure rate based on the brightness value of the overbright pixel when the endoscopic image corresponding to the endoscopic image data contains an overbright pixel, and to reduce the brightness of the overbright pixel in the endoscopic image based on the light reduction exposure rate to generate a light-reduced endoscopic image;
[0035] A display module is used to display a target endoscopic image in a display interface based on the reduced-light endoscopic image; the target endoscopic image does not include an overexposed area.
[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the second aspect is implemented.
[0037] An endoscopic device and an endoscopic image processing method provided in an embodiment of the present application determine over-bright pixels in an endoscopic image, determine a dimming exposure rate based on the brightness values of the over-bright pixels, and reduce the brightness of the over-bright pixels in the endoscopic image by using the dimming exposure rate, so that the target endoscopic image displayed in the display interface does not contain an over-exposed area. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0039] Figure 1 A schematic diagram of the structure of an endoscope device provided in an embodiment of the present application;
[0040] Figure 2 A schematic diagram of an endoscopic examination provided in an embodiment of the present application;
[0041] Figure 3 A flowchart of an endoscopic image processing method provided in an embodiment of the present application;
[0042] Figure 4 A schematic diagram of setting image contrast enhancement processing provided in an embodiment of the present application;
[0043] Figure 5 A schematic diagram of a target endoscopic image provided in an embodiment of the present application;
[0044] Figure 6 A flow chart for determining a dimming exposure rate provided in an embodiment of the present application;
[0045] Figure 7 A flow chart of generating an enhanced endoscopic image provided in an embodiment of the present application;
[0046] Figure 8 A flow chart for determining a light-enhancing exposure rate provided in an embodiment of the present application;
[0047] Fig. 9 A flow chart of fusing an endoscopic image after light reduction and an endoscopic image after light enhancement provided in an embodiment of the present application;
[0048] Fig.10 A schematic diagram of a fusion of a light-reduced endoscopic image and a light-enhanced endoscopic image provided in an embodiment of the present application;
[0049] Fig.11A schematic diagram of the working principle of a spatial attention module provided in an embodiment of the present application;
[0050] Fig.12 A schematic diagram of obtaining a first intermediate feature map provided in an embodiment of the present application;
[0051] Fig.13 A schematic diagram of the working principle of a channel attention module provided in an embodiment of the present application;
[0052] Fig.14 A schematic diagram of obtaining a second intermediate feature map provided in an embodiment of the present application;
[0053] Fig.15 An overall flow chart provided for an embodiment of the present application. DETAILED DESCRIPTION
[0054] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0055] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, “and” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.
[0056] Specifically, in the description of the present application, it needs to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application, and, unless otherwise specified, the "connection" and "connection" mentioned in the present application include direct and indirect connections (connections).
[0057] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0058] In order to facilitate understanding of an endoscope device and an endoscope image processing method provided in the embodiments of the present application, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art:
[0059] The meaning of bins in a histogram: To calculate a color histogram, the color space needs to be divided into several small color intervals, namely the bins of the histogram. The color histogram is obtained by calculating the pixels in each small interval. The more bins there are, the stronger the histogram's color resolution is, but it increases the burden on the computer. That is, (the 10 vertical bar areas in the figure above are each called a bin).
[0060] The definition of image information entropy is: in a certain spatial dimension, the magnitude of image information extracted based on the features selected by the image is the image information entropy.
[0061] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] Figure 1 A structural schematic diagram of an endoscope device in an embodiment of the present application is exemplarily shown, wherein the endoscope device includes a cold light source, a camera assembly and a display.
[0063] The camera assembly can be used to take pictures of endoscopic examinations and surgeries to collect image data of the examinations and surgeries; its key components may include one or more of a camera, a button, a camera host, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, a video cable, an optical adapter, an objective lens field of view, a mirror rod, a communication port, and a video output interface. The camera host can be connected to a display through a video output interface, and connected to a cold light source host through a communication port.
[0064] In the embodiment of the present application, the objective lens field of view, mirror rod, optical adapter, button, CMOS and camera among the key components of the camera assembly can also be collectively referred to as an endoscopic camera.
[0065] The display can be used to display the endoscopic image collected and generated by the camera assembly.
[0066] Cold light sources can be used to provide illumination for endoscopes during endoscopic examinations and surgeries; their key components usually include: a cold light source host, a light guide, a communication port, and a cold light source light output interface.
[0067] It should be understood that Figure 1 The schematic diagram of the structure of the endoscope device shown is only an example, and the endoscope device may have more Figure 1 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.
[0068] In the actual operation stage of endoscopic examination or surgery, medical personnel are basically assisted by dynamic video. After the medical personnel operate the endoscope to enter the human body, the camera on the endoscope collects and displays the image of the human body. As the camera of the endoscope moves, the range illuminated by the camera is displayed on the monitor connected to the endoscope. The environment around the lesion can be clearly seen, and the surgical instruments can be moved and operated in three dimensions intuitively. For example Figure 2 As shown, medical personnel can operate the endoscopic device to collect endoscopic images of the human body.
[0069] Many different implementations or examples are provided above to implement different structures of the present application. In order to simplify the content of the embodiments of the present application, only the components and settings of specific examples are described above. Of course, they are only examples, and the purpose is not to limit the present application.
[0070] Next, Figure 3 As shown, Figure 3 The flowchart of an endoscopic image processing method provided in an embodiment of the present application is shown. The method can be applied to Figure 1 In the processor of the display of the endoscope device shown, Figure 3 As shown, the method may include the following steps:
[0071] Step S301, receiving endoscopic image data sent by a camera host, where the endoscopic image data is generated based on light reflected from human tissue.
[0072] Step S302, when the endoscopic image corresponding to the endoscopic image data contains over-bright pixels, determine the dimming exposure rate based on the brightness value of the over-bright pixels, and reduce the brightness of the over-bright pixels in the endoscopic image based on the dimming exposure rate to generate a dimmed endoscopic image.
[0073] Step S303, displaying a target endoscopic image in a display interface based on the endoscopic image after light reduction; the target endoscopic image does not include an overexposed area.
[0074] It should be noted that when the display is set to perform image contrast enhancement processing, the display executes the endoscopic image processing method of the present application.
[0075] In the embodiment of the present application, the display responds to the start operation for starting the image contrast enhancement processing triggered by the user through the user operation interface, and the display is configured to start the image contrast enhancement processing, that is, the display is configured to perform the image contrast enhancement processing.
[0076] Optionally, the user operation interface in the embodiment of the present application may be a control in a display interface or a physical button on a camera host.
[0077] like Figure 4 As shown in FIG. 1 , a schematic diagram of setting image contrast enhancement processing in an embodiment of the present application is shown. Figure 4 As shown in the middle left figure, the user triggers the entry operation for entering the image contrast enhancement processing setting interface through the user operation interface. For example, the user Figure 4 The "Image Contrast Enhancement" trigger entry operation shown in the middle left figure. The display responds to the user-triggered entry operation and displays Figure 4 The setting interface for setting the image contrast enhancement process shown in the middle right figure includes an on control and an off control, which are respectively used to set the image contrast enhancement process on and off.
[0078] Exemplarily, when the user triggers the start-up operation for starting the image contrast enhancement process on the start-up control in the setting interface, the display is set to start the image contrast enhancement process. Figure 4 As shown in the middle right image, the monitor is set to turn on image contrast enhancement processing.
[0079] like Figure 5 FIG. 1 is a schematic diagram of a target endoscope image according to an embodiment of the present application. Figure 5 The middle left image is the original endoscopic image. Figure 5 The middle right image shows the obtained target endoscopic image.
[0080] The original endoscopic image contains an overexposed area, while the displayed target endoscopic image does not contain an overexposed area.
[0081] Specifically, the endoscope image processing method of the present application is described in detail below:
[0082] In step S302, based on the endoscopic image corresponding to the received endoscopic image data, over-bright pixels in the endoscopic image are determined by comparing the illumination value corresponding to each pixel in the endoscopic image with a set overexposure threshold.
[0083] The over-bright pixel point in the embodiment of the present application refers to the pixel point in the endoscopic image whose illumination value is greater than the set over-exposure threshold value.
[0084] It should be noted that the illumination value is a value corresponding to each pixel in the scene illumination map corresponding to the endoscope image.
[0085] Optionally, the embodiment of the present application determines the over-bright pixels in the endoscopic image by using Formula 1:
[0086] Q1={P(x)|T(x)>0.7}——Formula 1
[0087] Among them, Q1 represents a set of over-bright pixels, P(x) represents an endoscopic image, T(x) represents a scene illumination map corresponding to the endoscopic image, and 0.7 is a set overexposure threshold.
[0088] That is, the over-bright pixels included in Q1 obtained by formula 1 are pixels in the endoscopic image whose illumination values are greater than 0.7.
[0089] It should be noted that the process of determining the scene illumination map corresponding to the endoscopic image is a prior art in the art, and the scene illumination map corresponding to the endoscopic image can be obtained by illumination map estimation technology, so this application will not go into details here.
[0090] After determining the overbright pixel points corresponding to the endoscopic image, the embodiment of the present application processes the endoscopic image based on a preset light reduction exposure rate to generate a light-reduced endoscopic image if the endoscopic image does not contain overbright pixels. If the endoscopic image contains overbright pixels, the light reduction exposure rate is determined based on the brightness value of the overbright pixels, and the brightness of the overbright pixels in the endoscopic image is reduced based on the light reduction exposure rate to generate a light-reduced endoscopic image.
[0091] Exemplarily, if the endoscopic image does not contain overly bright pixels, the dimming exposure rate can be preset to 1.
[0092] like Figure 6 As shown, if the endoscopic image contains over-bright pixels, the embodiment of the present application provides a flow chart for determining the light reduction exposure rate, and the specific steps are as follows:
[0093] Step S601, extracting over-bright pixels in the endoscope image.
[0094] The over-bright pixel refers to a pixel in the endoscopic image whose illumination value is greater than a set over-exposure threshold.
[0095] Step S602, determining the brightness value of the overbright pixel based on the channel values of multiple color channels of the overbright pixel, and generating an overbright brightness map based on the brightness value of the overbright pixel.
[0096] It should be noted that, since the brightness of the image under different exposures varies significantly, while the color is basically the same, the present application only considers the brightness component when determining the dimming exposure rate.
[0097] In the embodiment of the present application, taking the multiple color channels of overbright pixels in the endoscopic image as RGB color channels as an example, the brightness value of the corresponding overbright pixel is determined based on the channel value of the RGB color channel corresponding to each overbright pixel in the endoscopic image.
[0098] Optionally, the embodiment of the present application determines the overbrightness image by using Formula 2:
[0099]
[0100] Wherein, B1 represents the overbrightness brightness map, which contains the brightness value of each overbright pixel; Indicates the channel value corresponding to the R (red) channel, Indicates the channel value corresponding to the G (green) channel, Indicates the channel value corresponding to the B (blue) channel.
[0101] Step S603: obtaining image information entropy corresponding to the overbrightness map based on the overbrightness map, and determining the dimming exposure rate based on the image information entropy.
[0102] It should be noted that a well-exposed image has higher visibility than an overexposed image and can provide richer information. Different dimming exposure rates used will affect the visibility of the image. Therefore, the optimal dimming exposure rate used in this application should be able to enable the image to provide the maximum amount of information. Since maximizing the image information entropy can characterize the maximum amount of information provided by the image, the embodiment of this application determines the optimal dimming exposure rate by calculating the maximum image information entropy.
[0103] First, this application introduces the image information entropy through Formula 3:
[0104]
[0105] Among them, H(B1) represents the image information entropy; p i is the i-th bin of the histogram of B1, used to calculate The number of data in, N is the number of bins (N is usually set to 256).
[0106] It should be noted that the bin in the histogram is a color interval.
[0107] For example, when i is 3, p i Characterization The number of pixels corresponding to the interval, that is, the number of pixels corresponding to the brightness value in the interval [3,4).
[0108] In this application, an unknown quantity k1 is used to represent the reduced light exposure rate, and the overbrightness brightness map B1 is processed based on the reduced light exposure rate k1 to obtain a formula for characterizing the overbrightness brightness map after the light reduction based on k1, such as Formula 4.
[0109]
[0110] Wherein, g(B1, k1) represents the overbrightness image after light reduction based on k1, and a and b represent fixed camera parameters, for example, a=0.3293, b=1.1258.
[0111] The embodiment of the present application obtains the value of the unknown quantity k1 based on the maximum image information entropy of g(B1, k1), that is, obtains the reduced light exposure rate corresponding to the endoscopic image in the present application.
[0112] Optionally, the embodiment of the present application determines the reduced light exposure rate corresponding to the endoscopic image by formula 5:
[0113]
[0114] in, represents the reduced light exposure rate corresponding to the endoscopic image, and H(g(B1,k1)) represents the image information entropy corresponding to the overbrightness image after light reduction based on k1.
[0115] The reduced light exposure rate of the endoscopic image is obtained by formula 5, and the brightness of overly bright pixels in the endoscopic image is reduced based on the reduced light exposure rate to generate a reduced light endoscopic image.
[0116] In step S303, a target endoscopic image is displayed in a display interface based on the endoscopic image after light reduction; the target endoscopic image does not include an overexposed area.
[0117] Optionally, before displaying the target endoscopic image, the embodiment of the present application also needs to process the endoscopic image based on the light enhancement exposure ratio to generate a light-enhanced endoscopic image, and fuse the light-reduced endoscopic image with the light-enhanced endoscopic image to obtain the target endoscopic image.
[0118] It should be noted that in the area where the light source is directly illuminated, due to the excessive light intensity, an over-exposed area will be generated, causing the details in the image to be covered by highlights and not clear; in the edge area of the image or the area where the light source is not directly illuminated, that is, the edge of the light, due to the dim light, the displayed image is too dark and cannot be presented clearly. Therefore, the present application can also detect the over-darkness of the endoscopic image and increase the brightness of the over-dark area.
[0119] like Figure 7 As shown, a flowchart of generating an enhanced endoscopic image in an embodiment of the present application, the specific steps are as follows:
[0120] Step S701 , based on the endoscopic image corresponding to the received endoscopic image data, the too-dark pixel points in the endoscopic image are determined by comparing the illumination value corresponding to each pixel point in the endoscopic image with the set too-dark threshold value.
[0121] The overly dark pixel point in the embodiment of the present application refers to the pixel point in the endoscopic image whose light value is less than the set overexposure threshold.
[0122] It should be noted that the illumination value is a value corresponding to each pixel in the scene illumination map corresponding to the endoscope image.
[0123] Optionally, the embodiment of the present application determines the dark pixels in the endoscopic image by using Formula 6:
[0124] Q2={P(x)|T(x)<0.5}——Formula 6
[0125] Among them, Q2 represents a set of pixels that are too dark, P(x) represents an endoscopic image, T(x) represents a scene illumination map corresponding to the endoscopic image, and 0.5 is a set too dark threshold.
[0126] That is to say, the too dark pixels included in Q2 obtained by formula 6 are the pixels in the endoscopic image whose illumination values are less than 0.5.
[0127] Step S702, determine whether there are too dark pixels; if so, execute step S703; if not, execute step S704.
[0128] Step S703, when the endoscopic image corresponding to the endoscopic image data contains too dark pixels, determine the brightening exposure rate based on the brightness value of the too dark pixels, and increase the brightness value of the too dark pixels in the endoscopic image based on the brightening exposure rate to generate an endoscopic image after brightening.
[0129] like Figure 8 As shown, a flow chart of determining the light-enhancing exposure rate according to an embodiment of the present application is shown, and the specific steps are as follows:
[0130] Step S801, extracting dark pixels in the endoscope image.
[0131] The too dark pixel refers to a pixel in the endoscope image whose illumination value is less than a set too dark threshold.
[0132] Step S802, determining the brightness value of the too dark pixel based on the channel values of the multiple color channels of the too dark pixel, and generating an too dark brightness map based on the brightness value of the too dark pixel.
[0133] The principle of generating the over-dark brightness map can be referred to in Formula 2, and this application will not elaborate on it here.
[0134] Step S803, based on the over-dark brightness map, obtaining the image information entropy corresponding to the over-dark brightness map, and determining the brightening exposure rate based on the image information entropy.
[0135] Among them, the present application uses an unknown quantity k2 to represent the light-enhanced exposure rate, processes the over-dark brightness image B2 based on the light-enhanced exposure rate k2, obtains a formula for characterizing the over-dark brightness image after light-enhanced based on k2, and based on the maximum image information entropy corresponding to the over-dark brightness image after light-enhanced, obtains the light-enhanced exposure rate corresponding to the endoscopic image in the present application.
[0136] Optionally, the embodiment of the present application determines the light enhancement exposure rate corresponding to the endoscopic image by formula 7:
[0137]
[0138] in, represents the light-enhanced exposure ratio corresponding to the endoscope image; H(g(B2,k2)) represents the image information entropy corresponding to the over-dark brightness image after light enhancement based on k2; g(B2, k2) represents the over-dark brightness map after light enhancement based on k2, and B2 represents the over-dark brightness map corresponding to the endoscopic image.
[0139] The light-enhanced exposure rate of the endoscopic image is obtained by formula 7, and the brightness of the too-dark pixels in the endoscopic image is increased based on the light-enhanced exposure rate to generate the light-enhanced endoscopic image.
[0140] Step S704: When the endoscopic image corresponding to the endoscopic image data does not contain too dark pixels, the endoscopic image is processed based on a preset light-enhanced exposure ratio to generate a light-enhanced endoscopic image.
[0141] Exemplarily, the preset brightening exposure ratio may be set to 1.
[0142] It should be noted that the embodiment of the present application does not limit the order of generating the endoscopic image after light reduction and generating the endoscopic image after light enhancement, and the two can also be executed at the same time.
[0143] Optionally, the embodiment of the present application may execute the above process through a camera response model to output the generated endoscopic image after light reduction and the endoscopic image after light enhancement.
[0144] Among them, the embodiment of the present application can establish a camera response model through the following process:
[0145] Generally speaking, camera manufacturers use some nonlinear in-camera processing such as white balance and demosaicing to improve the visual quality of captured images. These nonlinear processes can be modeled as Formula 8:
[0146] P=f(E)——Formula 8
[0147] Among them, P represents the endoscopic image, which includes the channel values corresponding to the endoscopic image under multiple color channels; E represents the image irradiance (also called scene irradiance), which is the light signal obtained by collecting the light reflected by the human tissue in the scene using optical principles, and f is the nonlinear camera response function (CRF).
[0148] The mapping function between two images that differ only in exposure is called brightness transfer function (BTF). i And the brightness conversion function g, the input image P can be mapped to the i-th image in the exposure set as P i =g(P,k i ).
[0149] Then the CRF can be obtained by solving the following co-parameter equations to obtain Formula 9:
[0150] g(f(E),k i )=f(k i E)——Formula 9
[0151] The camera response model consists of two parts: CRF model and BTF model. The parameters of the CRF model are determined only by the camera, while the parameters of the BTF model are determined by the camera and exposure rate.
[0152] In order to estimate the BTF model, two images P1 and P2 with different exposures are selected. The histogram of the underexposed image is mainly concentrated in the low brightness area. If the pixel values are linearly amplified before the traditional gamma correction, the resulting image is very close to the real well-exposed image. Therefore, the BTF model can be described as follows using a two-parameter function, as shown in Formula 10:
[0153] P2=g(P1,k)=βP1 γ ——Formula 10
[0154] Among them, β and γ are parameters related to the exposure rate k in the BTF model. For general cameras, different color channels have similar model parameters because their response curves are approximately the same.
[0155] In the BTF model of this application, β and γ are determined by the camera parameters and the exposure ratio k. In order to find the relationship between them, we need to obtain the corresponding CRF model. By solving the following co-parameter equation (substituting g = βf γ Formula g(f(E),k)=f(kE)), the CRF model can be derived, as shown in Formula 11:
[0156] f(kE)=βf(E) γ ——Formula 11
[0157] The closed form solution of f is given by Formula 12:
[0158]
[0159] Where a and b are model parameters when γ≠1: a=log k γ,
[0160] c is the model parameter when γ=1: c=log k β.
[0161] Two CRF models can be derived from Formula 12. When γ = 1, the CRF model is a power function, while the BTF model is a simple linear function. Some camera manufacturers design f as a gamma curve, which can better fit these camera characteristics. When γ ≠ 1, the CRF model is a two-parameter function, and the BTF model is a nonlinear function. Since the BTF of most cameras is nonlinear, the case of γ ≠ 1 is mainly considered.
[0162] From the above model parameter a = log k γ, It can be seen that the parameters of the BFT model are: γ=k a .
[0163] In summary, given an input image P and exposure rate k i , we can get i images with different exposures from our BTF model, as shown in Formula 13:
[0164]
[0165] Among them, P i Indicates based on exposure rate k i The image after processing the input image P.
[0166] Assuming no information about the camera is provided, a fixed camera parameter (a=0.3293, b=1.1258) is used, which is suitable for most cameras.
[0167] Through the above process, the camera response model is established. When in use, the endoscopic image corresponding to the received endoscopic image data is input into the camera response model, and the camera response model determines the light-enhanced exposure rate and light-reduced exposure rate corresponding to the endoscopic image, and processes the endoscopic image based on the light-enhanced exposure rate and the light-reduced exposure rate, and outputs the light-reduced endoscopic image and the light-enhanced endoscopic image.
[0168] like Fig. 9 As shown, a flowchart of a fusion of an endoscopic image after light reduction and an endoscopic image after light enhancement in an embodiment of the present application is shown, and the specific steps are as follows:
[0169] Step S901, performing local feature enhancement on the light-reduced endoscopic image through a spatial attention module to obtain a first spatial feature map.
[0170] Optionally, the embodiment of the present application may first perform convolution processing on the light-reduced endoscopic image, and then perform local feature enhancement on the light-reduced endoscopic image after the convolution processing through a spatial attention module to obtain a first spatial feature map.
[0171] It should be noted that the light-reduced endoscopic image after convolution processing is equivalent to a feature map.
[0172] Step S902: Perform global feature enhancement on the enhanced endoscopic image through a channel attention module to obtain a first channel feature map.
[0173] Optionally, the embodiment of the present application may first perform convolution processing on the enhanced endoscopic image, and then perform global feature enhancement on the enhanced endoscopic image after the convolution processing through a channel attention module to obtain a first channel feature map.
[0174] It should be noted that the enhanced endoscopic image after convolution processing is equivalent to a feature map.
[0175] Step S903, multiplying the first spatial feature map and the endoscopic image after light enhancement to obtain a second spatial feature map; and multiplying the first channel feature map and the endoscopic image after light reduction to obtain a second channel feature map.
[0176] Step S904, adding the second spatial feature map and the second channel feature map to obtain a target endoscopic image.
[0177] like Fig.10As shown, a schematic diagram of a fusion of a light-reduced endoscopic image and a light-enhanced endoscopic image according to an embodiment of the present application. Among them, the light-enhanced endoscopic image and the light-reduced endoscopic image are each subjected to 1×1 convolution to obtain a feature map M1 corresponding to the light-enhanced endoscopic image (i.e., the light-enhanced endoscopic image after convolution processing in the present application) and a feature map M2 corresponding to the light-reduced endoscopic image (i.e., the light-reduced endoscopic image after convolution processing in the present application). Among them, 1×1 convolution is used to change the number of channels to achieve cross-channel feature integration. After passing through the channel attention module and the spatial attention module, the first channel feature map M1 is obtained respectively. ′ and the first spatial feature map M2 ′ Then, the feature map M1 and the first spatial feature map M2 ′ Multiply each pixel by pixel to reweight each pixel and obtain the second spatial feature map R1. The first channel feature map M1 ′ Multiply the feature map M2 pixel by pixel, so as to re-weight each channel and obtain the second channel feature map R2. Finally, add the second spatial feature map R1 and the second channel feature map R2 to obtain the final target endoscopic image.
[0178] It should be noted that the main function of the channel attention module is to obtain attention in terms of the channel, that is, to emphasize channel information. Channel information usually represents the global features of the image, such as the color, brightness, etc. of the image. Compared with the original endoscopic image, the image after enhancement of the low-brightness area (the endoscopic image after brightness enhancement in this application) has richer color information. The main function of the spatial attention module is to obtain attention in terms of space, that is, to emphasize spatial information. Spatial information usually represents the local features of the image, such as the texture and edges of the image. Compared with the original endoscopic image, the image after brightness reduction in the overexposed area (the endoscopic image after brightness reduction in this application) has richer edge information.
[0179] against Fig.10 The spatial attention module in the present application is a schematic diagram of the working principle of the spatial attention module in the present application. Fig.11 As shown, in the spatial attention module, average pooling is performed on each channel corresponding to the reduced-light endoscopic image along the channel axis to obtain a first intermediate feature map; and convolution is performed on the first intermediate feature map to obtain a first spatial feature map.
[0180] Among them, what is input into the spatial attention module of the embodiment of the present application is the light-reduced endoscopic image after convolution processing, that is, the feature map M2 corresponding to the light-reduced endoscopic image.
[0181] It should be noted that if the input feature map M2 is of size H×W×c, average pooling is performed along the channel axis to obtain a first intermediate feature map of size H×W×1. Then, a 1×1 convolution is used to increase nonlinearity (the convolution process of the 1×1 convolution kernel is equivalent to the fully connected calculation process. By adding a nonlinear activation function, the expression ability of the model can be increased), and finally the first spatial feature map containing the local spatial information of each position is obtained.
[0182] For example, Fig.12 As shown, a schematic diagram of obtaining a first intermediate feature map in an embodiment of the present application is shown. Taking a feature map M2 of size 3×3×3 as an example, average pooling is performed on each channel in the feature map along the channel axis to obtain a first intermediate feature map of size 3×3×1.
[0183] For example, for Fig.12 The channel values 1, 2, and 3 at the same position in each channel shown are processed by average pooling, and the value corresponding to the position in the obtained first intermediate feature map is the average pooling result 2.
[0184] against Fig.10 The channel attention module in the embodiment of the present application is a schematic diagram of the working principle of the channel attention module. Fig.13 As shown, in the channel attention module, average pooling is performed on each channel corresponding to the enhanced endoscopic image to obtain a second intermediate feature map; and convolution is performed on the second intermediate feature map to obtain a first channel feature map.
[0185] Among them, what is input into the channel attention module of the embodiment of the present application is the enhanced endoscopic image after convolution processing, that is, the feature map M1 corresponding to the enhanced endoscopic image.
[0186] It should be noted that if the input feature map M1 is of size H×W×c, after average pooling is performed on each channel, a second intermediate feature map of size 1×1×c is obtained. A 1×1 convolution is used to increase nonlinearity (the convolution process of the 1×1 convolution kernel is equivalent to the fully connected calculation process. By adding a nonlinear activation function, the expression ability of the model can be increased), and finally the first channel feature map containing the global semantic information of each channel is obtained.
[0187] For example, Fig.14 As shown, a schematic diagram of obtaining a second intermediate feature map in an embodiment of the present application is shown. Taking a feature map M1 of size 3×3×3 as an example, each channel is subjected to average pooling processing to obtain a second intermediate feature map of size 1×1×3.
[0188] The average pooling process is performed on each channel, and the obtained second intermediate feature map contains the average pooling result corresponding to each channel. Fig.14 "3", "2", and "2" in the second intermediate feature map shown are the average pooling results corresponding to the corresponding channels.
[0189] like Fig.15 As shown, an overall flow chart of an embodiment of the present application, the specific steps are as follows:
[0190] Step S1501: After receiving the endoscope image data, the display establishes a camera response model.
[0191] Step S1502: determining the optimal boost exposure ratio through a camera response model.
[0192] Step S1503, processing the endoscopic image based on the optimal light-enhancing exposure ratio to obtain a light-enhanced endoscopic image in which the underexposed area is enhanced.
[0193] Step S1504: determine the optimal dimming exposure rate using a camera response model.
[0194] Step S1505, processing the endoscopic image based on the optimal light reduction exposure ratio to obtain a light-reduced endoscopic image in which the overexposed area is attenuated.
[0195] Step S1506: Generate a target endoscopic image based on an image fusion algorithm based on an attention mechanism.
[0196] It should be noted that in the prior art, in order to enhance the contrast of endoscopic images, the mainstream algorithms for endoscopic image contrast enhancement in the industry mainly include: histogram equalization algorithm and Retinex algorithm. Considering the uneven distribution of elements at different gray levels, histogram equalization (HE) is widely used to improve contrast. Many extensions of HE have considered some restrictions, such as brightness preservation and contrast limitation. However, HE-based methods often have the problem of over-enhancement, and color distortion will occur when used to enhance endoscopic images. Imitating the human visual system, Retinex theory has also been widely used in image enhancement. By separating reflectivity from illumination, Retinex-based algorithms can significantly enhance details. However, these methods have halo artifacts in high-contrast areas.
[0197] Through the present application, it is possible to maintain the contrast of the well-exposed areas in the original endoscopic image while enhancing the low contrast of the low-brightness areas in the original endoscopic image and reducing the light of the overexposed areas in the original endoscopic image, so that the present application can solve the problem of how to enhance the image contrast when some areas of the endoscopic image are underexposed or overexposed.
[0198] Based on the same inventive concept, the endoscopic image processing method described above in the present application can also be implemented by an endoscopic image processing device. The effect of the endoscopic image processing device is similar to that of the above method, and will not be described in detail here.
[0199] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in the embodiment of the present application. The principle of solving the problem by the electronic device is similar to the method of the above-mentioned embodiment. Therefore, the implementation of the electronic device can refer to the implementation of the above-mentioned method, and the repeated parts will not be repeated.
[0200] An embodiment of the present application further provides a computer storage medium, in which computer executable instructions are stored. The computer executable instructions are used to implement the endoscopic image processing method described in any embodiment of the present application.
[0201] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0203] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0205] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An endoscope device, characterized in that: The endoscope device comprises a camera host and a display; the display is used for: Receiving endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected by human tissue; When the endoscopic image corresponding to the endoscopic image data contains overbright pixels, determining a dimming exposure rate based on the brightness values of the overbright pixels, and reducing the brightness of the overbright pixels in the endoscopic image based on the dimming exposure rate to generate a dimmed endoscopic image; A target endoscopic image is displayed in a display interface based on the light-reduced endoscopic image; the target endoscopic image does not include an overexposed area.
2. The endoscope device according to claim 1, characterized in that The display is specifically used for: Extracting overbright pixels in the endoscopic image; the overbright pixels refer to pixels in the endoscopic image whose illumination values are greater than a set overexposure threshold; Determine the brightness value of the overbright pixel based on the channel values of multiple color channels of the overbright pixel, and generate an overbright brightness map based on the brightness value of the overbright pixel; Based on the overbrightness map, an image information entropy corresponding to the overbrightness map is obtained, and based on the image information entropy, the dimming exposure rate is determined.
3. The endoscope device according to claim 1, characterized in that The display is specifically used to determine the target endoscopic image in the following manner: When the endoscopic image corresponding to the endoscopic image data contains too dark pixels, determine a light-enhancing exposure rate based on the brightness values of the too dark pixels, and increase the brightness values of the too dark pixels in the endoscopic image based on the light-enhancing exposure rate to generate a light-enhanced endoscopic image; The light-reduced endoscopic image and the light-enhanced endoscopic image are fused to obtain the target endoscopic image; the target endoscopic image does not contain an underexposed area.
4. The endoscope device according to claim 3, characterized in that The display is also used for: Performing local feature enhancement on the light-reduced endoscopic image through a spatial attention module to obtain a first spatial feature map; Performing global feature enhancement on the enhanced endoscopic image through a channel attention module to obtain a first channel feature map; Performing a multiplication operation on the first spatial feature map and the endoscopic image after light enhancement to obtain a second spatial feature map; and performing a multiplication operation on the first channel feature map and the endoscopic image after light reduction to obtain a second channel feature map; The second spatial feature map and the second channel feature map are added to obtain the target endoscopic image.
5. The endoscope device according to claim 4, characterized in that The display is also used for: In the spatial attention module, average pooling is performed on each channel corresponding to the reduced-light endoscopic image along the channel axis to obtain a first intermediate feature map; Convolution processing is performed on the first intermediate feature map to obtain the first spatial feature map.
6. The endoscope device according to claim 4, characterized in that The display is also used for: In the channel attention module, average pooling is performed on each channel corresponding to the enhanced endoscopic image to obtain a second intermediate feature map; The second intermediate feature map is convolved to obtain the first channel feature map.
7. An endoscopic image processing method, applied to a display in an endoscopic device, characterized in that: The method comprises: Receiving endoscopic image data sent by the camera host, wherein the endoscopic image data is generated based on light reflected by human tissue; When the endoscopic image corresponding to the endoscopic image data contains overbright pixels, determining a dimming exposure rate based on the brightness values of the overbright pixels, and reducing the brightness of the overbright pixels in the endoscopic image based on the dimming exposure rate to generate a dimmed endoscopic image; A target endoscopic image is displayed in a display interface based on the light-reduced endoscopic image; the target endoscopic image does not include an overexposed area.
8. The method according to claim 7, characterized in that The step of determining the dimming exposure rate based on the brightness value of the overbright pixel point includes: Extracting overbright pixels in the endoscopic image; the overbright pixels refer to pixels in the endoscopic image whose illumination values are greater than a set overexposure threshold; Determine the brightness value of the overbright pixel based on the channel values of multiple color channels of the overbright pixel, and generate an overbright brightness map based on the brightness value of the overbright pixel; Based on the overbrightness map, an image information entropy corresponding to the overbrightness map is obtained, and based on the image information entropy, the dimming exposure rate is determined.
9. The method according to claim 7, characterized in that The target endoscopic image is determined by: When the endoscopic image corresponding to the endoscopic image data contains too dark pixels, determine a light-enhancing exposure rate based on the brightness values of the too dark pixels, and increase the brightness values of the too dark pixels in the endoscopic image based on the light-enhancing exposure rate to generate a light-enhanced endoscopic image; The light-reduced endoscopic image and the light-enhanced endoscopic image are fused to obtain the target endoscopic image; the target endoscopic image does not contain an underexposed area.
10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, any method described in claims 7 to 9 is implemented.
Citation Information
Patent Citations
Non-uniform illumination image enhancement method based on virtual multi-exposure fusion
CN112734650A
Attention-based image recognition method and system, electronic equipment and storage medium
CN113255821A
Endoscopic image-based large intestine polyp segmentation method and system and related components
CN113838047A
Vehicle information identification method, intelligent wearable device and readable storage medium
CN115512311A
Endoscope image quality evaluation method and device based on multiple dimensions, equipment and medium
CN116051421A
Cited By
Endoscopic device and endoscopic image processing method
WO2026021425A1
