Endoscopic Image Color Correction Method, Device, Equipment and Storage Medium

Abstract

The present application discloses an endoscopic image color correction method, device, equipment and storage medium. The method includes: performing multi-target image segmentation on an input image according to a preset target category, retaining valid regions, and generating a target segmentation image; generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm; for each pixel point in the target segmentation image, calculating a first hemoglobin attribute and a second hemoglobin attribute; calculating correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; restoring the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and outputting a color-corrected image. By this method, the purpose of maintaining high color reducibility for the same type of tissue at different illumination distances can be achieved, providing more real and accurate image information for clinical diagnosis.

Description

Technical Field

[0001] The present application relates to the technical field of medical endoscope imaging, and particularly to an endoscope image color correction method, device, equipment and storage medium. Background Art

[0002] Endoscope technology is widely used in the examination and minimally invasive surgery of human body cavities such as the digestive tract and respiratory tract. Due to the particularity of the endoscope imaging scene (such as uneven illumination and large differences in tissue hemoglobin content), the following problems will occur: (1) Color distortion caused by uneven illumination: The tissue in the area where the endoscope light source irradiates closely fades due to excessive light intensity (such as the weakening of the red tone), while the color in the distant area is distorted due to insufficient illumination, and the same tissue presents inconsistent colors at different depths. (2) The diversity of tissues is not distinguished: Currently, the RGB values are enhanced based on the global hemoglobin index, without distinguishing the characteristics of different tissues such as blood, blood vessels, and mucosa, resulting in mis-enhancement in low hemoglobin areas or insufficient enhancement in high hemoglobin areas. The above defects make it difficult for doctors to accurately identify tissue characteristics (such as the morphology of early lesion blood vessels) during diagnosis, and there is an urgent need for a color correction method that can adapt to the illumination intensity and accurately distinguish tissue categories. Summary of the Invention

[0003] The purpose of the present application is to provide an endoscope image color correction method, device, equipment and storage medium for the deficiencies of the existing technology, which can be used to solve the problem of image color distortion caused by uneven illumination and tissue diversity, and improve the accuracy and efficiency of clinical diagnosis.

[0004] In the first aspect, the present application proposes an endoscope image color correction method, and the method includes:

[0005] Performing multi-target image segmentation on the input image according to a preset target category, retaining the effective area, and generating a target segmentation image;

[0006] Generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0007] For each pixel point in the target segmentation image, calculating a first hemoglobin attribute and a second hemoglobin attribute, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0008] Calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map;

[0009] Obtaining the corrected first and second hemoglobin attribute values based on the correction proportionality coefficients;

[0010] Restore the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and output the color-corrected image.

[0011] In a possible implementation, the multi-object image segmentation of the input image according to a preset target category, retaining the valid regions, and generating a target segmentation image includes:

[0012] Segment and label the input image according to categories such as blood, blood vessels, mucosal tissue, and others. The first three categories are target categories to be processed;

[0013] Obtain the global target brightness value and the brightness mean value of the sub-block regions based on the automatic exposure module, label the sub-block regions with a brightness difference from the target brightness less than a preset threshold, and calculate the R-channel reference value thereof;

[0014] Eliminate the invalid regions through threshold screening, and retain the regions that meet the conditions as the target segmentation image.

[0015] In a possible implementation, in the step of generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm, the depth map estimation method includes any one of the following:

[0016] A deep learning model based on monocular depth estimation;

[0017] Disparity calculation based on binocular stereo vision;

[0018] Depth speculation associated with the surface reflection model and brightness based on photometric method.

[0019] In a possible implementation, in the step of calculating the first hemoglobin attribute and the second hemoglobin attribute for each pixel point in the target segmentation image:

[0020] The value of the first hemoglobin attribute The calculation formula is:

[0021] ,

[0022] where represents a pixel point, and R, G, and B are the RGB channel values of the pixel point;

[0023] The value of the second hemoglobin attribute The calculation formula is:

[0024] ,

[0025] where α is a preset parameter.

[0026] In a possible implementation manner, in the step of calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map, the correction proportionality coefficients are negatively correlated with the square of the depth value and positively correlated with the local brightness value. The specific calculation formula is:

[0027] , ,

[0028] = , = ,

[0029] Wherein, 、 represent the corrected first and second hemoglobin attribute values, is the first attribute correction coefficient, is the second attribute correction coefficient, 、 are modulation parameters, is the depth value, is the local brightness value.

[0030] In a possible implementation manner, the restoring the RGB channel values of the pixel point to be processed based on the corrected first and second hemoglobin attribute values includes:

[0031] If , then

[0032] ,

[0033] ,

[0034] If , then

[0035] ,

[0036] ,

[0037] can be obtained through the following formula, ,

[0038] Wherein, 、 、 represent the RGB channel values of the pixel point to be processed before restoration, 、 、 the pixel point after restoration of the RGB channel values, is a modulation parameter, and its value range is [0, 1].

[0039] In a possible implementation, during the RGB value restoration process, the luminance channel value of the original pixel point needs to be kept unchanged. The luminance channel value is determined based on the color space of the input image, including but not limited to:

[0040] The grayscale value L in the RGB color space is L = 0.299R + 0.587G + 0.114B;

[0041] The luminance component Y in the YUV color space;

[0042] The value component V in the HSV color space;

[0043] The lightness component L in the Lab color space.

[0044] In a second aspect, the present application also provides an endoscope image color correction device, which includes:

[0045] An image segmentation module, configured to perform multi-target image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image;

[0046] A depth estimation module, configured to generate a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0047] An attribute calculation module, configured to calculate a first hemoglobin attribute and a second hemoglobin attribute for each pixel point in the target segmentation image. The first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0048] A correction coefficient module, configured to calculate a correction proportionality coefficient corresponding to the first and second hemoglobin attributes according to the depth values of each pixel point in the depth map; and obtain the corrected first and second hemoglobin attribute values based on the correction proportionality coefficient;

[0049] A color restoration module, configured to restore the RGB value of the pixel point to be processed based on the corrected first and second hemoglobin attribute values, and output the color-corrected image.

[0050] In a third aspect, the present application also provides an electronic device, including a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform the method provided by the first aspect or any possible implementation manner of the first aspect of the present application.

[0051] Fourthly, the present application also provides a computer storage medium storing a plurality of instructions adapted to be loaded and executed by a processor to perform the method provided in the first aspect of the present application or any possible implementation manner of the first aspect.

[0052] In the technical solution provided by the present application, multi-object image segmentation is performed on the input image according to a preset target category, the valid region is retained, and a target segmentation image is generated; based on the target segmentation image, a corresponding depth map is generated through a depth estimation algorithm; for each pixel point in the target segmentation image, a first hemoglobin attribute and a second hemoglobin attribute are calculated, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; according to the depth values of the pixel points in the depth map, correction proportionality coefficients corresponding to the first and second hemoglobin attributes are calculated; based on the correction proportionality coefficients, corrected first and second hemoglobin attribute values are obtained; based on the corrected first and second hemoglobin attribute values, the RGB value of the pixel point to be processed is restored, and a color-corrected image is output. In view of the problem of color deviation of different tissues caused by the illumination distance (intensity), the present application first proposes a method for accurately distinguishing different tissue regions to be processed according to the endoscopic scene characteristics, eliminates the invalid regions or the regions that do not need to be corrected, and only retains the regions that need to be corrected. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB value of the pixel point to be processed is restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissues at different illumination distances and providing more real and accurate image information for clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on the structures shown in these drawings without creative efforts.

[0054] Figure 1 is a schematic flowchart of an endoscopic image color correction method provided by an embodiment of the present application;

[0055] Figure 2 is a schematic structural diagram of an endoscopic image color correction device provided by an embodiment of the present application;

[0056] Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0058] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present application, then the directional indications are only used to explain the relative positional relationship, movement conditions, etc. between components in a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indications will also change accordingly.

[0059] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, then the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In addition, the meaning of "and / or" appearing throughout the text includes three parallel scenarios. Taking "A and / or B" as an example, it includes scenario A, scenario B, or the scenario where A and B are satisfied simultaneously. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions conflicts with each other or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0060] Figure 1 Exemplarily shown is a flowchart of an endoscopic image color correction method provided by an embodiment of the present application. As Figure 1 shown, the endoscopic image color correction method may at least include the following steps:

[0061] S101: Perform multi-target image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image.

[0062] The above-mentioned multi-target image segmentation of the input image according to the preset target category may specifically include: segmenting and labeling the image into four categories: blood, blood vessels, mucosal tissue, and others. Among them, the first three categories (blood, blood vessels, mucosal tissue) are target categories to be processed, and their common feature is that the R-channel value is the largest. The processing flow for all categories is the same, only the parameters are different. The following takes the processing flow of a certain target category as an example for illustration.

[0063] Specifically, in the analysis of medical endoscope images, multi-target image segmentation of the input image according to preset target categories (such as blood, blood vessels, mucosal tissues, and other categories) is a key pre-step for color correction. Accurate segmentation helps to perform targeted color correction on different tissue regions subsequently and improve the diagnostic value of the image.

[0064] Before performing multi-target image segmentation, it is necessary to preprocess the input endoscope image (color space conversion, filtering and noise reduction) to improve the accuracy and efficiency of segmentation.

[0065] Generally, the images collected by endoscopes are in the RGB color space. However, in some cases, converting to other color spaces (such as HSV, YCrCb) may be more beneficial for subsequent processing. For example, the HSV color space separates color information (hue H, saturation S, and value V), which is convenient for segmentation according to the color characteristics of different tissues.

[0066] Endoscope images may be disturbed by noise, such as Gaussian noise, salt-and-pepper noise, etc. Filtering methods can be used to effectively remove noise and smooth the images. Common filtering methods include Gaussian filtering, median filtering, etc.

[0067] Among them, this step adopts an algorithm for multi-target image segmentation, which can include but is not limited to: (1) threshold-based segmentation (local threshold method: different thresholds are selected in local regions to adapt to different characteristics of the targets); (2) region growing method (starting from seed points, the region is gradually expanded according to similarity (such as color, intensity, etc.). After the image is initially segmented, similar regions are merged); (3) edge detection (using edge detection operators such as canny and sobel, and using gradient information to detect edges and extract the target boundaries); (4) clustering-based method (such as Mean Shift clustering method, by clustering pixels, pixels of the same category are assigned to the same target, and multi-target segmentation is performed by finding the density peak); (5) deep learning-based method (such as network models such as FCN, U-Net, SegNet, SENet, etc.). The segmented image marked with a certain target category is denoted as .

[0068] The above-mentioned threshold-based segmentation can specifically be based on the gray value or color characteristics of the image, and the image is segmented into different regions by setting appropriate thresholds. For example, for the blood region, since its color is usually darker, thresholds can be set according to the hue and saturation values in the HSV color space for segmentation.

[0069] The above-mentioned edge detection can specifically be to segment different tissue regions by detecting the edge information in the image. Commonly used edge detection operators include Sobel operator, Canny operator, etc.

[0070] Optionally, the above deep learning-based method can specifically be segmentation by using the U-Net model. Among them, U-Net is a convolutional neural network model commonly used in medical image segmentation, with an encoder-decoder structure, which can effectively extract image features and perform pixel-level classification. Before using U-Net for segmentation, it is necessary to prepare a training dataset, including endoscopic images and corresponding annotated images (annotating blood, blood vessels, mucosal tissue, and other categories). Then use the trained model for segmentation: load the trained U-Net model and segment the input image.

[0071] As an alternative implementation, the multi-target image segmentation of the input image according to the preset target categories, retaining the valid regions, and generating the target segmentation image can specifically include: segmenting and marking the input image according to blood, blood vessels, mucosal tissue, and other categories, with the first three categories being the target categories to be processed; obtaining the global target brightness value and the brightness mean value of the sub-block regions based on the automatic exposure module, marking the sub-block regions with a brightness difference from the target less than the preset threshold, and calculating the R-channel reference value thereof; screening out the invalid regions through threshold screening, and retaining the regions that meet the conditions as the target segmentation image.

[0072] Specifically, the automatic exposure module plays an important role in modern endoscopic devices. It can automatically adjust the exposure parameters according to the overall brightness of the image to obtain a clear image with appropriate brightness. This step uses this module to obtain the global target brightness value and the brightness mean value of the sub-block regions, and performs subsequent processing.

[0073] Among them, obtaining the global target brightness value can specifically be that the automatic exposure module calculates a global target brightness value according to the overall brightness of the image, and this value reflects the ideal brightness level of the image. This value can be directly obtained through the interface or related functions of the device.

[0074] Then, calculate the brightness mean value of the sub-block regions: divide the input image into multiple sub-block regions of the same size. For example, the image can be divided into sub-blocks of 16×16 pixels. For each sub-block, calculate the brightness mean value of all its pixel points. In the RGB color space, the brightness can be calculated by the formula L = 0.299R + 0.587G + 0.114B. Mark the sub-block regions with a brightness difference from the target less than the preset threshold: set a preset threshold, such as 10. Traverse all sub-block regions, compare their brightness mean values with the global target brightness value, and if the difference is less than the threshold, mark the sub-block region as a valid region.

[0075] Finally, calculate the R-channel reference value: In the sub-blocks marked as valid regions, extract the R-channel values of all pixel points, and calculate the average of these values as the R-channel reference value.

[0076] Furthermore, after obtaining the R-channel reference value, use this value and a preset threshold to screen the segmented and marked image, eliminate the invalid regions, and retain the regions that meet the conditions as the target segmented image.

[0077] Determine the screening condition: For each pixel point in the segmented and marked image, calculate the absolute value of the difference between its R-channel value and the R-channel reference value. If this absolute value is greater than the preset threshold, for example, 5, then the region where this pixel point is located is considered an invalid region.

[0078] Eliminate the invalid regions: Traverse the segmented and marked image, and set the pixel values of the pixel points that do not meet the screening conditions to 0 (black), thereby eliminating the invalid regions.

[0079] Through the above steps, first use a suitable segmentation method to segment and mark the input image according to the preset target categories, then use the brightness information obtained by the automatic exposure module to mark the effective sub-block regions and calculate the R-channel reference value, and finally eliminate the invalid regions through threshold screening, successfully generating the target segmented image, laying a foundation for subsequent steps such as depth estimation and color correction.

[0080] S102: Based on the target segmented image, generate a corresponding depth map through a depth estimation algorithm.

[0081] Specifically, based on the endoscope scenario, the methods for estimating depth information include but are not limited to photometric method (for the target segmented image , since the material properties (blood or blood vessels or tissues) are determined, the surface reflection model corresponding to this material can be obtained through off-line optical calibration, and the depth information of the object surface can be inferred by using the change in image brightness.) (2) Stereo vision (By using dual-sensor technology, compare the disparity between two images and calculate the depth information.) (3) Monocular depth estimation (For small-sized endoscopes that can only arrange a single sensor, a single image can be used for depth estimation, usually through deep learning models (such as DepthNet, FCRN, MiDaS, etc.) to extract image features and predict the corresponding depth values.) (4) Other methods, such as structured light, TOF, etc.

[0082] S103: For each pixel point in the target segmented image, calculate the first hemoglobin property and the second hemoglobin property, where the first hemoglobin property represents the relative concentration of hemoglobin, and the second hemoglobin property represents the relative color of hemoglobin.

[0083] Specifically, the common feature of blood, blood vessels, and mucosal tissues is that they all contain hemoglobin at different concentrations and have their respective spectral absorption and reflection characteristics. Based on this, the first hemoglobin attribute corresponding to the target category can be defined. , where can represent the relative concentration of hemoglobin; define the second hemoglobin attribute , where can represent the relative color of hemoglobin; among them, can be expressed as ; can be expressed as , is a preset parameter.

[0084] Let the value of the first hemoglobin attribute of the image to be processed be , and the value of the second hemoglobin attribute be . For the pixel point (x, y) to be processed, ; . Among them, represents the pixel point, and R, G, and B are the RGB channel values of the pixel point.

[0085] S104: Calculate the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map.

[0086] Specifically, let the value of the first hemoglobin attribute to be adjusted to be , and the value of the second hemoglobin attribute be ; let the first attribute correction coefficient be , and the second attribute correction coefficient be . Among them, the first and second attribute correction coefficients , are negatively correlated with the image depth corresponding to the current pixel point, then , .

[0087] The first and second correction proportionality coefficients of the pixel point to be processed , are negatively correlated with the square of its depth value and positively correlated with its brightness value.

[0088] One of the representation formulas is: = , = , where , represent the corrected first and second hemoglobin attribute values, is the first attribute correction coefficient, is the second attribute correction coefficient, , is a modulation parameter, is a depth value, is a local brightness value.

[0089] S105: Obtain the corrected first and second hemoglobin attribute values based on the correction proportionality coefficient.

[0090] S106: Based on the corrected first and second hemoglobin attribute values, restore the RGB values of the pixel to be processed and output the color-corrected image.

[0091] Specifically, if , then

[0092] ,

[0093] ,

[0094] If , then

[0095] ,

[0096] ,

[0097] can be obtained by the following formula ,

[0098] where , , represent the RGB channel values of the pixel to be processed before restoration, , , the RGB channel values of the pixel after restoration, is a modulation parameter with a value range of [0, 1].

[0099] In the embodiments of the present application, for the implementation of multi-color spaces with a brightness invariant constraint, the RGB value restoration process needs to keep the brightness channel value of the original pixel unchanged. The specific implementation method is determined according to the color space of the input image, and may specifically include the following methods:

[0100] 1. RGB color space: The brightness value is calculated by the formula: L = 0.299R + 0.587G + 0.114B

[0101] The restored RGB values need to satisfy: L dst = L src .

[0102] where L srcRepresents the luminance value of the original pixel point (Source Luminance), that is, the luminance channel value of the input image before color correction, L dst Represents the luminance value of the pixel point after color correction (Destination Luminance), that is, the luminance channel value corrected by the hemoglobin attribute.

[0103] 2. YUV color space: If the input image is converted to the YUV space during preprocessing, the luminance component Y is directly retained unchanged: Y dst =Y src ,

[0104] The restored RGB values are achieved through the inverse transformation from YUV to RGB, ensuring that the Y component remains constant.

[0105] 3. HSV color space: Extract the value component V of the original image. The restored RGB values need to satisfy: V dst =V src ,

[0106] Color correction is achieved by adjusting the hue H and saturation S, while keeping the V channel unchanged.

[0107] 4. Lab color space: Retain the value component L unchanged: L dst =L src ,

[0108] Adjust the a (red - green axis) and b (yellow - blue axis) channels to restore the RGB values, ensuring that the L channel remains constant.

[0109] Among them, in the parameter L dst / L src / Y dst / Y src / V dst / V src The subscript src represents the luminance value of the original pixel point (Source Luminance), that is, the luminance channel value of the input image before color correction, and the subscript dst represents the luminance value of the pixel point after color correction (Destination Luminance), that is, the luminance channel value corrected by the hemoglobin attribute.

[0110] In the technical solution provided by the present application, multi-object image segmentation is performed on the input image according to a preset target category, the valid region is retained, and a target segmentation image is generated; based on the target segmentation image, a corresponding depth map is generated through a depth estimation algorithm; for each pixel point in the target segmentation image, a first hemoglobin attribute and a second hemoglobin attribute are calculated, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; according to the depth values of the pixel points in the depth map, correction ratio coefficients corresponding to the first and second hemoglobin attributes are calculated; based on the correction ratio coefficients, corrected first and second hemoglobin attribute values are obtained; based on the corrected first and second hemoglobin attribute values, the RGB value of the pixel point to be processed is restored, and a color-corrected image is output. In the present application, aiming at the problem of color deviation of different tissues caused by the illumination distance (intensity), first, according to the endoscopic scene characteristics, a method for accurately distinguishing different tissue regions to be processed is proposed, invalid regions or regions that do not need to be corrected are removed, and only the regions that need to be corrected are retained. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB value of the pixel point to be processed is restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissue under different illumination distances, and providing more real and accurate image information for clinical diagnosis.

[0111] To facilitate the understanding of the method provided by the embodiments of the present application, the embodiments of the present application also provide an endoscopic image color correction device. Figure 2 Exemplarily shown is a schematic structural diagram of an endoscopic image color correction device provided by the embodiments of the present application. As Figure 2 shown, it may at least include:

[0112] An image segmentation module 210, configured to perform multi-object image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image;

[0113] A depth estimation module 220, configured to generate a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0114] An attribute calculation module 230, configured to calculate a first hemoglobin attribute and a second hemoglobin attribute for each pixel point in the target segmentation image, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0115] A correction coefficient module 240, configured to calculate correction ratio coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; and obtain corrected first and second hemoglobin attribute values based on the correction ratio coefficients;

[0116] A color restoration module 250 is configured to restore the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and output an image with color correction.

[0117] It should be noted that when the endoscopic image color correction device provided in the above embodiments executes the method, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the endoscopic image color correction device provided in the above embodiments and the embodiments of the endoscopic image color correction method belong to the same inventive concept, and the specific implementation process can be seen in the method embodiments, which will not be elaborated here.

[0118] In the technical solution provided in this application, for the problem of color deviation of different tissues due to the illumination distance (intensity), first, according to the endoscopic scene characteristics, a method for accurately distinguishing different tissue regions to be processed is proposed to eliminate invalid regions or regions that do not need to be corrected, and only the regions that need to be corrected are retained. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB values of the pixel points to be processed are restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissues at different illumination distances, and provide more real and accurate image information for clinical diagnosis.

[0119] The embodiment of this application also provides a computer storage medium, which can store multiple program instructions. The program instructions are suitable for being loaded and executed by a processor to execute the solution described in the above method embodiments, which will not be elaborated here.

[0120] The embodiment of this application also provides a computer program product, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to execute the solution described in the above method embodiments, which will not be elaborated here.

[0121] Please refer to Figure 3 , which shows a schematic structural diagram of an electronic device provided by an exemplary embodiment of this application. The electronic device can be provided as a server. The electronic device in this specification can include one or more of the following components: a processor 1110, a memory 1120, an input device 1130, an output device 1140, and a bus 1150. The processor 1110, the memory 1120, the input device 1130, and the output device 1140 can be connected through the bus 1150.

[0122] The processor 1110 may include one or more processing cores. The processor 1110 connects various parts within the entire electronic device through various interfaces and lines, and executes various functions of the electronic device 1100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1120, and by invoking the data stored in the memory 1120. Optionally, the processor 1110 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1110 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the displayed content; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 1110 and may be implemented separately by a communication chip.

[0123] The memory 1120 may include a random access memory (RAM), and may also include a read-only memory (ROM). Optionally, the memory 1120 includes a non-transitory computer-readable storage medium. The memory 1120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1120 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be an Android system, including a system developed based on the Android system in depth, an IOS system developed by Apple Inc., including a system developed based on the IOS system in depth, or other systems.

[0124] In order for the operating system to distinguish the specific application scenarios of third-party application programs, it is necessary to establish data communication between the third-party application programs and the operating system, so that the operating system can obtain the current scenario information of the third-party application programs at any time, and then perform targeted system resource adaptation based on the current scenario.

[0125] Among them, the input device 1130 is used to receive input instructions or data. The input device 1130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 1140 is used to output instructions or data. The output device 1140 includes, but is not limited to, a display device, a speaker, etc. In one example, the input device 1130 and the output device 1140 can be combined, and the input device 1130 and the output device 1140 are a touch display screen.

[0126] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the drawings, or combine some components, or have different component arrangements. For example, the electronic device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a Wireless Fidelity (WiFi) module, a power supply, a Bluetooth module, etc., which will not be elaborated here.

[0127] In Figure 3 In the shown electronic device, the processor 1110 can be used to call the application program for endoscopic image color correction stored in the memory 1120 to execute the method described in the above method embodiments.

[0128] The above is a schematic solution of an electronic device according to an embodiment of the present application. It should be noted that the technical solution of the electronic device and the technical solution of the above endoscopic image color correction method belong to the same inventive concept. For the details not described in the technical solution of the electronic device, reference can be made to the description of the technical solution of the above endoscopic image color correction method.

[0129] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium of the computer program can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0130] The above are only optional embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included within the protection scope of this specification.

[0131] The above description has been made of specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An endoscopic image color correction method, characterized in that, The method includes: Performing multi-target image segmentation on the input image according to a preset target category, retaining the valid regions, and generating a target segmentation image; Generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm; For each pixel point in the target segmentation image, calculating a first hemoglobin attribute and a second hemoglobin attribute, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; Calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; Obtaining the corrected first and second hemoglobin attribute values based on the correction proportionality coefficients; Restoring the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and outputting a color-corrected image; Wherein, in the step of calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map, the correction proportionality coefficient is negatively correlated with the square of the depth value and positively correlated with the local brightness value, and the specific calculation formula is: ， ， = ， = ， Among them, and represent the first and second hemoglobin attribute values before correction, and represent the first and second hemoglobin attribute values after correction, is the first attribute correction coefficient, is the second attribute correction coefficient, and are modulation parameters, is the depth value, is the local brightness value.

2. The method according to claim 1, characterized in that, The performing multi-target image segmentation on the input image according to a preset target category, retaining the valid regions, and generating a target segmentation image includes: Segmenting and marking the input image according to categories such as blood, blood vessels, mucosal tissues, and others, and the first three categories are the target categories to be processed; Obtaining the global target brightness value and the brightness mean value of the sub-block regions based on the automatic exposure module, marking the sub-block regions with a target brightness difference less than a preset threshold, and calculating the R-channel reference value thereof; Eliminating the invalid regions through threshold screening, and retaining the regions that meet the conditions as the target segmentation image.

3. The method according to claim 1, characterized in that, In the step of generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm, the depth map estimation methods include any one of the following: A deep learning model based on monocular depth estimation; Disparity calculation based on binocular stereo vision; Depth speculation associated with the surface reflection model and brightness based on photometric method.

4. The method according to claim 1, wherein In the step of calculating the first hemoglobin attribute and the second hemoglobin attribute for each pixel point in the target segmentation image: The first hemoglobin attribute value is calculated by the following formula: ， Among them, represents a pixel point, and R, G, and B are the RGB channel values of the pixel point; The second hemoglobin attribute value The calculation formula is as follows: ， Where α is a preset parameter.

5. The method according to claim 4, characterized in that, The restoring the RGB channel values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values includes: If , then ， ， If , then ， ， It can be obtained by the following formula , Among them, , , represent the pixel points to be processed RGB channel values before restoration, , , pixel points RGB channel values after restoration, is the modulation parameter, and the value range is [0, 1].

6. The method according to claim 5, wherein During the RGB value restoration process, it is necessary to keep the brightness channel value of the original pixel point unchanged, and the brightness channel value is determined based on the color space of the input image, including but not limited to: The gray value L in the RGB color space = 0.299R + 0.587G + 0.114B; The luminance component Y in the YUV color space; The value component V in the HSV color space; The luminance component L in the Lab color space.

7. An endoscopic image color correction device, characterized in that, The device includes: An image segmentation module for performing multi-target image segmentation on the input image according to a preset target category, retaining the valid regions, and generating a target segmentation image; A depth estimation module for generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm; An attribute calculation module, configured to calculate a first hemoglobin attribute and a second hemoglobin attribute for each pixel point in the target segmentation image, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; A correction coefficient module, configured to calculate correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; and obtain corrected first and second hemoglobin attribute values based on the correction proportionality coefficients; A color restoration module, configured to restore the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and output a color-corrected image; Wherein, the correction proportionality coefficient is negatively correlated with the square of the depth value and positively correlated with the local brightness value, and the specific calculation formula is: ， ， = ， = ， Among them, and represent the first and second hemoglobin attribute values before correction, and represent the first and second hemoglobin attribute values after correction, is the first attribute correction coefficient, is the second attribute correction coefficient, and are modulation parameters, is the depth value, is the local brightness value.

8. An electronic device, characterized in that, Including: A processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform the method according to any one of claims 1 to 6.

9. A computer storage medium, which stores multiple instructions, and the instructions are adapted to be loaded and executed by a processor to perform the method according to any one of claims 1 to 6.

Images