Endoscope image color correction method and device, equipment and storage medium

Abstract

The invention discloses an endoscope image color correction method and device, equipment and a storage medium, and the method comprises the steps: carrying out the multi-target image segmentation of an input image according to a preset target category, reserving an effective region, and generating a target segmentation image; based on the target segmentation image, generating a corresponding depth map through a depth estimation algorithm; for each pixel point in the target segmented image, calculating a first hemoglobin attribute and a second hemoglobin attribute; calculating correction proportionality coefficients corresponding to the first hemoglobin attribute and the second hemoglobin attribute according to the depth value of each pixel point in the depth map; and based on the corrected first and second hemoglobin attribute values, recovering the RGB value of the pixel point to be processed, and outputting an image after color correction. Through the method, the purpose of keeping high color reducibility of the same type of tissues under different illumination distances can be achieved, and more real and accurate image information is provided for clinical diagnosis.

Description

Technical Field

[0001] The present application relates to the technical field of medical endoscope imaging, and particularly relates to an endoscope image color correction method, device, equipment, and storage medium. Background Art

[0002] Endoscope technology is widely used in the examination and minimally invasive surgery of human body cavities such as the digestive tract and respiratory tract. Due to the particularity of the endoscope imaging scene (such as uneven illumination and large differences in tissue hemoglobin content), the following problems will occur: (1) Color distortion caused by uneven illumination: The tissue in the area where the endoscope light source irradiates closely fades due to excessive light (such as the weakening of the red tone), while the color in the distant area is distorted due to insufficient illumination, and the same tissue presents inconsistent colors at different depths. (2) Tissue diversity is not distinguished: Currently, the RGB values are enhanced based on the global hemoglobin index without distinguishing the characteristics of different tissues such as blood, blood vessels, and mucosa, resulting in incorrect enhancement in low hemoglobin areas or insufficient enhancement in high hemoglobin areas. The above defects make it difficult for doctors to accurately identify tissue characteristics (such as the morphology of early lesion blood vessels) during diagnosis, and there is an urgent need for a color correction method that can adapt to the illumination intensity and accurately distinguish tissue categories. Summary of the Invention

[0003] The purpose of the present application is to provide an endoscope image color correction method, device, equipment, and storage medium for the deficiencies of the existing technology, which can be used to solve the problem of image color distortion caused by uneven illumination and tissue diversity, and improve the accuracy and efficiency of clinical diagnosis.

[0004] In the first aspect, the present application proposes an endoscope image color correction method, and the method includes:

[0005] Performing multi-target image segmentation on the input image according to a preset target category, retaining the valid area, and generating a target segmentation image;

[0006] Generating a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0007] For each pixel point in the target segmentation image, calculating a first hemoglobin attribute and a second hemoglobin attribute, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0008] Calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of each pixel point in the depth map;

[0009] Obtaining the corrected first and second hemoglobin attribute values based on the correction proportionality coefficients;

[0010] Based on the corrected first and second hemoglobin attribute values, restore the RGB values of the pixel points to be processed, and output the color-corrected image.

[0011] In a possible implementation, the multi-target image segmentation of the input image according to a preset target category, retaining the valid region, and generating the target segmentation image includes:

[0012] Segment and label the input image according to blood, blood vessels, mucosal tissue and other categories, and the first three categories are the target categories to be processed;

[0013] Based on the automatic exposure module, obtain the global target brightness value and the brightness mean value of the sub-block area, mark the sub-block areas with a brightness difference from the target brightness less than the preset threshold, and calculate the R-channel reference value thereof;

[0014] Eliminate the invalid regions through threshold screening, and retain the regions that meet the conditions as the target segmentation image.

[0015] In a possible implementation, in the step of generating the corresponding depth map based on the target segmentation image through the depth estimation algorithm, the depth map estimation method includes any one of the following:

[0016] Deep learning model based on monocular depth estimation;

[0017] Disparity calculation based on binocular stereo vision;

[0018] Depth speculation based on the surface reflection model of photometric method and the association with brightness.

[0019] In a possible implementation, in the step of calculating the first hemoglobin attribute and the second hemoglobin attribute for each pixel point in the target segmentation image:

[0020] The value of the first hemoglobin attribute The calculation formula is:

[0021] ,

[0022] Where represents the pixel point, and R, G, and B are the RGB channel values of the pixel point;

[0023] The value of the second hemoglobin attribute The calculation formula is:

[0024] ,

[0025] Where α is a preset parameter.

[0026] In a possible implementation, in the step of calculating the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map, the correction proportionality coefficient is negatively correlated with the square of the depth value and positively correlated with the local brightness value. The specific calculation formula is:

[0027] , ,

[0028] = , = ,

[0029] Wherein, 、 represent the corrected first and second hemoglobin attribute values, is the first attribute correction coefficient, is the second attribute correction coefficient, 、 are modulation parameters, is the depth value, is the local brightness value.

[0030] In a possible implementation, the restoring of the RGB channel values of the pixel point to be processed based on the corrected first and second hemoglobin attribute values includes:

[0031] If , then

[0032] ,

[0033] ,

[0034] If , then

[0035] ,

[0036] ,

[0037] Can be obtained by the following formula, ,

[0038] Wherein, 、 、 represent the RGB channel values of the pixel point to be processed before restoration, 、 、 The RGB channel values of the pixel point after restoration, is a modulation parameter, and its value range is [0, 1].

[0039] In a possible implementation, during the RGB value restoration process, the luminance channel value of the original pixel point needs to be kept unchanged. The luminance channel value is determined based on the color space of the input image, including but not limited to:

[0040] The grayscale value L in the RGB color space is L = 0.299R + 0.587G + 0.114B;

[0041] The luminance component Y in the YUV color space;

[0042] The value component V in the HSV color space;

[0043] The lightness component L in the Lab color space.

[0044] In a second aspect, the present application also proposes an endoscope image color correction device, and the device includes:

[0045] An image segmentation module, configured to perform multi-target image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image;

[0046] A depth estimation module, configured to generate a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0047] An attribute calculation module, configured to calculate a first hemoglobin attribute and a second hemoglobin attribute for each pixel point in the target segmentation image, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0048] A correction coefficient module, configured to calculate the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; and obtain the corrected first and second hemoglobin attribute values based on the correction proportionality coefficients;

[0049] A color restoration module, configured to restore the RGB value of the pixel point to be processed based on the corrected first and second hemoglobin attribute values, and output the color-corrected image.

[0050] In a third aspect, the present application also proposes an electronic device, including a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform the method provided in the first aspect or any possible implementation manner of the first aspect of the present application.

[0051] In a fourth aspect, the present application also provides a computer storage medium storing a plurality of instructions adapted to be loaded and executed by a processor to perform the method provided in the first aspect or any possible implementation of the first aspect of the present application.

[0052] In the technical solution provided by the present application, an input image is subjected to multi-target image segmentation according to a preset target category, the valid region is retained, and a target segmentation image is generated; based on the target segmentation image, a corresponding depth map is generated through a depth estimation algorithm; for each pixel point in the target segmentation image, a first hemoglobin attribute and a second hemoglobin attribute are calculated, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; according to the depth values of the pixel points in the depth map, correction proportionality coefficients corresponding to the first and second hemoglobin attributes are calculated; based on the correction proportionality coefficients, corrected first and second hemoglobin attribute values are obtained; based on the corrected first and second hemoglobin attribute values, the RGB values of the pixel points to be processed are restored, and a color-corrected image is output. In the present application, for the problem of color cast caused by illumination distance (intensity) in different tissues, first, according to the endoscopic scene characteristics, a method for accurately distinguishing different tissue regions to be processed is proposed, the invalid regions or regions that do not need to be corrected are removed, and only the regions that need to be corrected are retained. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB values of the pixel points to be processed are restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissue at different illumination distances, and providing more real and accurate image information for clinical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.

[0054] Figure 1 It is a flowchart showing the method for endoscopic image color correction provided by an embodiment of the present application;

[0055] Figure 2 It is a schematic structural diagram of an endoscopic image color correction device provided by an embodiment of the present application;

[0056] Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts belong to the scope of protection of the present application.

[0058] It should be noted that if there are directional indications (such as up, down, left, right, front, back...) involved in the embodiments of the present application, then the directional indications are only used to explain the relative positional relationship, movement conditions, etc. between components in a certain specific posture (as shown in the accompanying drawings). If this specific posture changes, then the directional indications will also change accordingly.

[0059] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, then the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the meaning of "and / or" appearing throughout the text includes three parallel scenarios. Taking "A and / or B" as an example, it includes scenario A, or scenario B, or the scenario where both A and B are satisfied simultaneously. In addition, the technical solutions between various embodiments can be combined with each other, but it must be based on the ability of those of ordinary skill in the art to implement. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0060] Figure 1 Exemplarily shown is a flowchart of an endoscopic image color correction method provided by an embodiment of the present application. As Figure 1 shown, the endoscopic image color correction method may at least include the following steps:

[0061] S101: Perform multi-target image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image.

[0062] The above-mentioned multi-target image segmentation of the input image according to a preset target category may specifically include: segmenting and labeling the image into four categories: blood, blood vessels, mucosal tissue, and others. Among them, the first three categories (blood, blood vessels, mucosal tissue) are target categories to be processed, and their common characteristic is that the R-channel value is the largest. The processing flow for all categories is the same, only the parameters are different. The following takes the processing flow of a certain target category as an example for illustration.

[0063] Specifically, in the analysis of medical endoscope images, multi-target image segmentation of the input image according to preset target categories (such as blood, blood vessels, mucosal tissue, and other categories) is a key pre-step for color correction. Accurate segmentation helps to perform targeted color correction on different tissue regions subsequently, improving the diagnostic value of the images.

[0064] Before performing multi-target image segmentation, it is necessary to preprocess the input endoscope image (color space conversion, filtering and denoising) to improve the accuracy and efficiency of segmentation.

[0065] Generally, the images collected by endoscopes are in the RGB color space. However, in some cases, converting to other color spaces (such as HSV, YCrCb) may be more beneficial for subsequent processing. For example, the HSV color space separates color information (hue H, saturation S, and value V), facilitating segmentation according to the color characteristics of different tissues.

[0066] Endoscope images may be disturbed by noise, such as Gaussian noise, salt-and-pepper noise, etc. Filtering methods can be used to effectively remove noise and smooth the images. Common filtering methods include Gaussian filtering, median filtering, etc.

[0067] Among them, this step adopts an algorithm for multi-target image segmentation, which can include but is not limited to: (1) Threshold-based segmentation (local threshold method: different thresholds are selected in local regions to adapt to different characteristics of the targets.); (2) Region growing method (starting from seed points, gradually expanding the region according to similarity (such as color, intensity, etc.). After the image is initially segmented, similar regions are merged.); (3) Edge detection (using edge detection operators such as canny and sobel, and using gradient information to detect edges and extract the boundaries of the targets.); (4) Clustering-based method (such as Mean Shift clustering method, by clustering pixels, making pixels of the same category be assigned to the same target, and performing multi-target segmentation by finding the density peaks.); (5) Deep learning-based method (such as network models like FCN, U-Net, SegNet, SENet, etc.). Denote the segmented image marked with a certain target category as .

[0068] The above-mentioned threshold-based segmentation can specifically be based on the gray value or color characteristics of the image, and the image is segmented into different regions by setting appropriate thresholds. For example, for the blood region, since its color is usually darker, thresholds can be set according to the hue and saturation values in the HSV color space for segmentation.

[0069] The above-mentioned edge detection can specifically be to segment different tissue regions by detecting the edge information in the image. Commonly used edge detection operators include Sobel operator, Canny operator, etc.

[0070] Optionally, the above deep learning-based method can specifically be segmentation by a U-Net model. Among them, U-Net is a convolutional neural network model commonly used in medical image segmentation, with an encoder-decoder structure, which can effectively extract image features and perform pixel-level classification. Before using U-Net for segmentation, a training dataset needs to be prepared, including endoscopic images and corresponding annotated images (annotating blood, blood vessels, mucosal tissues, and other categories). Then, use the trained model for segmentation: load the trained U-Net model and segment the input image.

[0071] As an optional implementation manner, the multi-target image segmentation of the input image according to a preset target category, retaining the effective region, and generating a target segmentation image can specifically include: segmenting and marking the input image according to categories such as blood, blood vessels, mucosal tissues, and other categories, and the first three categories are target categories to be processed; obtaining the global target brightness value and the brightness mean value of the sub-block region based on the automatic exposure module, marking the sub-block region with a brightness difference less than a preset threshold from the target, and calculating its R-channel reference value; screening out invalid regions through threshold screening, and retaining the regions that meet the conditions as the target segmentation image.

[0072] Specifically, the automatic exposure module plays an important role in modern endoscopic devices. It can automatically adjust the exposure parameters according to the overall brightness of the image to obtain a clear and appropriately bright image. This step uses this module to obtain the global target brightness value and the brightness mean value of the sub-block region and perform subsequent processing.

[0073] Among them, obtaining the global target brightness value can specifically be that the automatic exposure module calculates a global target brightness value according to the overall brightness of the image, and this value reflects the ideal brightness level of the image. This value can be directly obtained through the interface or related functions of the device.

[0074] Then, calculate the brightness mean value of the sub-block region: divide the input image into multiple sub-block regions of the same size. For example, the image can be divided into sub-blocks of 16×16 pixels. For each sub-block, calculate the brightness mean value of all its pixel points. In the RGB color space, the brightness can be calculated by the formula L = 0.299R + 0.587G + 0.114B. Mark the sub-block region with a brightness difference less than a preset threshold from the target: set a preset threshold, such as 10. Traverse all sub-block regions, compare their brightness mean values with the global target brightness value, and if the difference is less than this threshold, mark this sub-block region as an effective region.

[0075] Finally, calculate the R-channel reference value: In the sub-blocks marked as valid regions, extract the R-channel values of all pixel points, and calculate the average of these values as the R-channel reference value.

[0076] Furthermore, after obtaining the R-channel reference value, use this value and a preset threshold to screen the segmented and marked image, eliminate the invalid regions, and retain the regions that meet the conditions as the target segmented image.

[0077] Determine the screening condition: For each pixel point in the segmented and marked image, calculate the absolute value of the difference between its R-channel value and the R-channel reference value. If this absolute value is greater than the preset threshold, for example, 5, then the region where this pixel point is located is considered an invalid region.

[0078] Eliminate the invalid regions: Traverse the segmented and marked image, and set the pixel values of the pixel points that do not meet the screening conditions to 0 (black), thereby eliminating the invalid regions.

[0079] Through the above steps, first use a suitable segmentation method to segment and mark the input image according to the preset target categories, then use the brightness information obtained by the automatic exposure module to mark the effective sub-block regions and calculate the R-channel reference value, and finally eliminate the invalid regions through threshold screening, successfully generating the target segmented image, which lays the foundation for subsequent steps such as depth estimation and color correction.

[0080] S102: Based on the target segmented image, generate a corresponding depth map through a depth estimation algorithm.

[0081] Specifically, based on the endoscopic scenario, the methods for estimating depth information include but are not limited to photometric method (for the target segmented image , since the material properties (blood or blood vessels or tissues) are determined, the surface reflection model corresponding to this material can be obtained through offline optical calibration, and the depth information of the object surface can be inferred by using the change in image brightness. (2) Binocular stereo vision (by comparing the disparity between two images through dual-sensor technology to calculate the depth information). (3) Monocular depth estimation (for small-sized endoscopes that can only arrange a single sensor, single-frame images can be used for depth estimation, usually through deep learning models (such as DepthNet, FCRN, MiDaS, etc.) to extract image features and predict the corresponding depth values). (4) Other methods, such as structured light, TOF, etc.

[0082] S103: For each pixel point in the target segmented image, calculate the first hemoglobin property and the second hemoglobin property, where the first hemoglobin property represents the relative concentration of hemoglobin, and the second hemoglobin property represents the relative color of hemoglobin.

[0083] Specifically, the commonality of blood, blood vessels, and mucosal tissues is that they all contain hemoglobin at different concentrations and have their respective spectral absorption and reflection characteristics, based on which the first hemoglobin attribute corresponding to the target category can be defined. , where can characterize the relative concentration of hemoglobin; define the second hemoglobin attribute , where can characterize the relative color of hemoglobin; among them, can be expressed as ; can be expressed as , is a preset parameter.

[0084] Let the value of the first hemoglobin attribute of the image to be processed be , and the value of the second hemoglobin attribute be . For the pixel point (x, y) to be processed, ; . Among them, represents the pixel point, and R, G, and B are the RGB channel values of the pixel point.

[0085] S104: Calculate the correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map.

[0086] Specifically, let the value of the first hemoglobin attribute to be adjusted to be , and the value of the second hemoglobin attribute be ; let the first attribute correction coefficient be , and the second attribute correction coefficient be . Among them, the first and second attribute correction coefficients , are negatively correlated with the image depth corresponding to the current pixel point, then , .

[0087] The first and second correction proportionality coefficients of the pixel point to be processed , are negatively correlated with the square of its depth value and positively correlated with its brightness value.

[0088] One of the representation formulas is: = , = , where , represent the corrected first and second hemoglobin attribute values, is the first attribute correction coefficient, is the second attribute correction coefficient, , is a modulation parameter, is a depth value, is a local brightness value.

[0089] S105: Obtain the corrected first and second hemoglobin attribute values based on the correction ratio coefficient.

[0090] S106: Based on the corrected first and second hemoglobin attribute values, restore the RGB values of the pixel to be processed and output the color-corrected image.

[0091] Specifically, if , then

[0092] ,

[0093] ,

[0094] If , then

[0095] ,

[0096] ,

[0097] can be obtained by the following formula ,

[0098] where , , represent the RGB channel values of the pixel to be processed before restoration, , , the RGB channel values of the pixel after restoration, is a modulation parameter with a value range of [0,1].

[0099] In the embodiments of the present application, for the implementation of multi-color spaces with brightness invariance constraints, the RGB value restoration process needs to keep the brightness channel value of the original pixel unchanged. The specific implementation method is determined according to the color space of the input image, and may specifically include the following methods:

[0100] 1. RGB color space: The brightness value is calculated by the formula: L = 0.299R + 0.587G + 0.114B

[0101] The restored RGB values need to satisfy: L dst = L src .

[0102] where L srcRepresents the luminance value of the original pixel point (Source Luminance), that is, the luminance channel value of the input image before color correction, L dst Represents the luminance value of the pixel point after color correction (Destination Luminance), that is, the luminance channel value corrected by the hemoglobin property.

[0103] 2. YUV color space: If the input image is converted to the YUV space during preprocessing, the luminance component Y is directly retained unchanged: Y dst =Y src ,

[0104] The restored RGB values are achieved through the inverse transformation from YUV to RGB, ensuring that the Y component remains constant.

[0105] 3. HSV color space: Extract the value component V of the original image. The restored RGB values need to satisfy: V dst =V src ,

[0106] Color correction is achieved by adjusting the hue H and saturation S while keeping the V channel unchanged.

[0107] 4. Lab color space: Keep the value component L unchanged: L dst =L src ,

[0108] Adjust the a (red - green axis) and b (yellow - blue axis) channels to restore the RGB values, ensuring that the L channel remains constant.

[0109] Among them, in the parameter L dst / L src / Y dst / Y src / V dst / V src The subscript src represents the luminance value of the original pixel point (Source Luminance), that is, the luminance channel value of the input image before color correction, and the subscript dst represents the luminance value of the pixel point after color correction (Destination Luminance), that is, the luminance channel value corrected by the hemoglobin property.

[0110] In the technical solution provided by the present application, the input image is subjected to multi-target image segmentation according to a preset target category, the valid region is retained, and a target segmentation image is generated; based on the target segmentation image, a corresponding depth map is generated through a depth estimation algorithm; for each pixel point in the target segmentation image, a first hemoglobin attribute and a second hemoglobin attribute are calculated, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin; according to the depth values of the pixel points in the depth map, correction proportionality coefficients corresponding to the first and second hemoglobin attributes are calculated; based on the correction proportionality coefficients, corrected first and second hemoglobin attribute values are obtained; based on the corrected first and second hemoglobin attribute values, the RGB values of the pixel points to be processed are restored, and a color-corrected image is output. In view of the problem of color deviation caused by the illumination distance (intensity) of different tissues, the present application first proposes a method for accurately distinguishing different tissue regions to be processed according to the endoscopic scene characteristics, eliminates the invalid regions or the regions that do not need to be corrected, and only retains the regions that need to be corrected. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB values of the pixel points to be processed are restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissue under different illumination distances, and provide more real and accurate image information for clinical diagnosis.

[0111] To facilitate the understanding of the method provided by the embodiments of the present application, the embodiments of the present application also provide an endoscopic image color correction device. Figure 2 Exemplarily shows a schematic structural diagram of an endoscopic image color correction device provided by the embodiments of the present application. As Figure 2 shown, it may at least include:

[0112] An image segmentation module 210, configured to perform multi-target image segmentation on the input image according to a preset target category, retain the valid region, and generate a target segmentation image;

[0113] A depth estimation module 220, configured to generate a corresponding depth map based on the target segmentation image through a depth estimation algorithm;

[0114] An attribute calculation module 230, configured to calculate a first hemoglobin attribute and a second hemoglobin attribute for each pixel point in the target segmentation image, where the first hemoglobin attribute represents the relative concentration of hemoglobin, and the second hemoglobin attribute represents the relative color of hemoglobin;

[0115] A correction coefficient module 240, configured to calculate correction proportionality coefficients corresponding to the first and second hemoglobin attributes according to the depth values of the pixel points in the depth map; and obtain corrected first and second hemoglobin attribute values based on the correction proportionality coefficients;

[0116] The color restoration module 250 is configured to restore the RGB values of the pixel points to be processed based on the corrected first and second hemoglobin attribute values, and output the color-corrected image.

[0117] It should be noted that when the endoscopic image color correction device provided in the above embodiment executes the method, only the above division of each functional module is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the endoscopic image color correction device provided in the above embodiment and the endoscopic image color correction method embodiment belong to the same inventive concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0118] In the technical solution provided in this application, for the problem of color deviation caused by the illumination distance (intensity) of different tissues, first, according to the endoscopic scene characteristics, a method for accurately distinguishing different tissue regions to be processed is proposed, and invalid regions or regions that do not need to be corrected are eliminated, and only the regions that need to be corrected are retained. At the same time, two hemoglobin attributes related to depth information are defined, and the RGB values of the pixel points to be processed are restored through the corrected two hemoglobin attribute values, so as to achieve the purpose of maintaining high color reducibility for the same type of tissue under different illumination distances, and provide more real and accurate image information for clinical diagnosis.

[0119] The embodiment of this application also provides a computer storage medium, which can store multiple program instructions, and the program instructions are suitable for being loaded and executed by a processor to execute the solution described in the above method embodiment, which will not be repeated here.

[0120] The embodiment of this application also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to execute the solution described in the above method embodiment, which will not be repeated here.

[0121] Please refer to Figure 3 , which shows a schematic structural diagram of an electronic device provided by an exemplary embodiment of this application. The electronic device can be provided as a server. The electronic device in this specification may include one or more of the following components: a processor 1110, a memory 1120, an input device 1130, an output device 1140, and a bus 1150. The processor 1110, the memory 1120, the input device 1130, and the output device 1140 can be connected through the bus 1150.

[0122] The processor 1110 may include one or more processing cores. The processor 1110 connects various parts within the entire electronic device through various interfaces and circuits. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 1120, and by invoking data stored in the memory 1120, it performs various functions of the electronic device 1100 and processes data. Optionally, the processor 1110 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1110 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor 1110 and may be implemented separately through a communication chip.

[0123] The memory 1120 may include random access memory (RAM) and may also include read-only memory (ROM). Optionally, the memory 1120 includes a non-transitory computer-readable storage medium. The memory 1120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1120 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the following various method embodiments, etc. The operating system may be the Android system, including a system developed based on the Android system in depth, the IOS system developed by Apple Inc., including a system developed based on the IOS system in depth, or other systems.

[0124] In order for the operating system to distinguish the specific application scenarios of third-party application programs, it is necessary to establish data communication between the third-party application programs and the operating system, so that the operating system can obtain the current scenario information of the third-party application programs at any time, and then perform targeted system resource adaptation based on the current scenario.

[0125] Among them, the input device 1130 is used to receive input instructions or data. The input device 1130 includes, but is not limited to, a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 1140 is used to output instructions or data. The output device 1140 includes, but is not limited to, a display device, a speaker, etc. In one example, the input device 1130 and the output device 1140 can be combined, and the input device 1130 and the output device 1140 are a touch display screen.

[0126] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than those shown in the drawings, or combine some components, or have different component arrangements. For example, the electronic device further includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a Wireless Fidelity (WiFi) module, a power supply, a Bluetooth module, etc., which will not be elaborated here.

[0127] In Figure 3 In the electronic device shown, the processor 1110 can be used to call the application program for endoscopic image color correction stored in the memory 1120 to execute the method described in the above method embodiments.

[0128] The above is a schematic solution of an electronic device according to an embodiment of the present application. It should be noted that the technical solution of the electronic device and the technical solution of the above endoscopic image color correction method belong to the same inventive concept. For the details not described in detail in the technical solution of the electronic device, reference can be made to the description of the technical solution of the above endoscopic image color correction method.

[0129] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. Among them, the storage medium of the computer program can be a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0130] The above are only optional embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this specification shall be included in the protection scope of this specification.

[0131] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for color correction of an endoscopic image, characterized in that: The method comprises: Perform multi-target image segmentation on the input image according to the preset target categories, retain the effective area, and generate the target segmentation image; Based on the target segmentation image, a corresponding depth map is generated through a depth estimation algorithm; For each pixel point in the target segmented image, calculating a first hemoglobin attribute and a second hemoglobin attribute, wherein the first hemoglobin attribute represents a relative hemoglobin concentration, and the second hemoglobin attribute represents a relative hemoglobin color; Calculate the correction ratio coefficients corresponding to the first and second hemoglobin attributes according to the depth values ​​of each pixel in the depth map; Obtaining the corrected first and second hemoglobin attribute values ​​based on the correction proportional coefficient; Based on the corrected first and second hemoglobin attribute values, the RGB values ​​of the pixels to be processed are restored, and a color-corrected image is output.

2. The method according to claim 1, characterized in that The step of performing multi-target image segmentation on the input image according to preset target categories, retaining effective areas, and generating target segmentation images includes: The input image is segmented and labeled according to blood, blood vessels, mucosal tissue and other categories, and the first three categories are the target categories to be processed; Based on the automatic exposure module, the global target brightness value and the average brightness of the sub-block area are obtained, the sub-block area whose brightness difference with the target is less than the preset threshold is marked, and its R channel reference value is calculated; Invalid areas are eliminated through threshold screening, and areas that meet the conditions are retained as the target segmentation image.

3. The method according to claim 1, characterized in that: In the step of generating a corresponding depth map by a depth estimation algorithm based on the target segmented image, the depth map estimation method includes any one of the following: Deep learning model based on monocular depth estimation; Disparity calculation based on binocular stereo vision; Depth estimation based on photometric surface reflectance model and brightness correlation.

4. The method according to claim 1, characterized in that In the step of calculating the first hemoglobin attribute and the second hemoglobin attribute for each pixel point in the target segmented image: The first hemoglobin attribute value The calculation formula is: ， in, Represents a pixel, where R, G, and B are the RGB channel values ​​of the pixel; The second hemoglobin attribute value The calculation formula is: ， Among them, α is a preset parameter.

5. The method according to claim 4, characterized in that In the step of calculating the correction proportional coefficients corresponding to the first and second hemoglobin attributes according to the depth values ​​of each pixel in the depth map, the correction proportional coefficient is negatively correlated with the square of the depth value and positively correlated with the local brightness value. The specific calculation formula is: ， ， = ， = ， in, , represents the corrected first and second hemoglobin attribute values, is the first attribute correction coefficient, is the second attribute correction coefficient, , is the modulation parameter, is the depth value, is the local brightness value.

6. The method according to claim 5, characterized in that The method of restoring the RGB channel value of the pixel to be processed based on the corrected first and second hemoglobin attribute values ​​includes: like ,but ， ， like ,but ， ， It can be obtained by the following formula: , in, , , Indicates the pixel to be processed The RGB channel values ​​before restoration, , , Pixels The restored RGB channel values, is the modulation parameter, and its value range is [0,1].

7. The method according to claim 6, characterized in that During the RGB value restoration process, the brightness channel value of the original pixel point needs to be kept unchanged. The brightness channel value is determined based on the color space of the input image, including but not limited to: The gray value in the RGB color space is L=0.299R+0.587G+0.114B; The brightness component Y in the YUV color space; The lightness component V in the HSV color space; The lightness component L in the Lab color space.

8. An endoscope image color correction device, characterized in that: The device comprises: An image segmentation module is used to perform multi-target image segmentation on the input image according to preset target categories, retain the effective area, and generate a target segmentation image; A depth estimation module is used to generate a corresponding depth map based on the target segmentation image through a depth estimation algorithm; an attribute calculation module, for calculating, for each pixel point in the target segmented image, a first hemoglobin attribute and a second hemoglobin attribute, wherein the first hemoglobin attribute represents a relative hemoglobin concentration, and the second hemoglobin attribute represents a relative hemoglobin color; A correction coefficient module, used to calculate the correction ratio coefficients corresponding to the first and second hemoglobin attributes according to the depth values ​​of each pixel in the depth map; and to obtain the corrected first and second hemoglobin attribute values ​​based on the correction ratio coefficients; The color restoration module is used to restore the RGB value of the pixel to be processed based on the corrected first and second hemoglobin attribute values, and output a color-corrected image.

9. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 7.

10. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the method according to any one of claims 1 to 7.

Images