Image defogging method, storage medium, and system

Abstract

Embodiments of the present specification disclose an image defogging method, a storage medium and a system. The method comprises: acquiring a first histogram of a smoke image; segmenting the first histogram to obtain a plurality of first sub-histograms; respectively performing contrast expansion on each of the first sub-histograms to obtain second sub-histograms; merging the expanded second sub-histograms to obtain a second histogram; and generating a smoke-removed image according to the second histogram. The embodiments of the present specification can improve the local and global contrast of the smoke image, thereby improving the smoke-removing effect.

Description

Technical Field

[0001] This specification relates to the field of endoscopy technology, and in particular to an image defogging method, storage medium, and system. Background Technology

[0002] During endoscopic surgery, surgeons may use surgical instruments, such as electrocautery, to cauterize tissues or organs. This cauterization produces smoke, which the endoscope captures in the surgical environment as smoke-containing images. These smoke images can impair the surgeon's field of vision, increasing surgical risks. Therefore, dehazing of these smoke images is necessary. Current image dehazing methods, such as dark channel prior (DCT) algorithms, have shown poor performance in practical applications, resulting in lower contrast in the smoke-free images and negatively impacting their visual appeal. Summary of the Invention

[0003] This specification provides an image dehazing method, storage medium, and system to improve image contrast.

[0004] A first aspect of the embodiments of this specification provides an image dehazing method, comprising:

[0005] Obtain the first histogram of the smoke image;

[0006] The first histogram is segmented to obtain multiple segments of the first sub-histogram;

[0007] Contrast expansion is performed on each segment of the first sub-histogram to obtain the second sub-histogram;

[0008] The expanded second sub-histograms are merged to obtain the second histogram;

[0009] Based on the second histogram, generate an image after removing the smoke.

[0010] In some embodiments, the first histogram includes a histogram of a grayscale image of a smoke image; the segmentation of the first histogram includes: obtaining a segmentation threshold for the grayscale image using an image segmentation algorithm; and segmenting the first histogram into foreground and background according to the segmentation threshold to obtain two first sub-historical images, wherein the two first sub-historical images include a foreground sub-historical image and a background sub-historical image.

[0011] In some embodiments, the grayscale image includes a first grayscale image after grayscale processing of the smoke image; obtaining the segmentation threshold of the grayscale image using an image segmentation algorithm includes: obtaining the segmentation threshold of the first grayscale image using an image segmentation algorithm.

[0012] In some embodiments, the grayscale image includes a second grayscale image of a smoke image in multiple color channels; obtaining the segmentation threshold of the grayscale image using an image segmentation algorithm includes: obtaining the segmentation threshold of each second grayscale image using an image segmentation algorithm; segmenting the first histogram into foreground and background based on the segmentation threshold includes: segmenting the first histogram of the second grayscale image into foreground and background based on the segmentation threshold of each second grayscale image to obtain two first sub-historical segments, the two first sub-historical segments including a foreground sub-historical segment and a background sub-historical segment.

[0013] In some embodiments, the contrast expansion of each segment of the first sub-histogram includes: obtaining a reference benchmark for the first sub-histogram; dividing the first sub-histogram into a first low gray level region and a first high gray level region according to the reference benchmark; moving the gray levels in the first low gray level region away from the reference benchmark in a decreasing direction; and moving the gray levels in the first high gray level region away from the reference benchmark in an increasing direction.

[0014] In some embodiments, obtaining the reference benchmark for the first sub-histogram includes: obtaining the maximum and minimum gray levels in the first sub-histogram; and calculating the median of the maximum and minimum gray levels as the reference benchmark for the first sub-histogram.

[0015] In some embodiments, the image dehazing method further includes: determining an adjustment coefficient set based on the first sub-histogram, the adjustment coefficient set including at least one adjustment coefficient corresponding to a gray level in the first sub-histogram; the step of moving the gray level in the first low gray level region away from the reference reference in a decreasing direction includes: for the gray level in the first low gray level region, selecting a corresponding adjustment coefficient from the adjustment coefficient set, and adjusting the gray level according to the selected adjustment coefficient, the reference reference, and the range of gray levels in the first low gray level region; the step of moving the gray level in the first high gray level region away from the reference reference in an increasing direction includes: for the gray level in the first high gray level region, selecting a corresponding adjustment coefficient from the adjustment coefficient set, and adjusting the gray level according to the selected adjustment coefficient, the reference reference, and the range of gray levels in the first high gray level region.

[0016] In some embodiments, determining the adjustment coefficient set based on the first sub-histogram includes: dividing the first sub-histogram into a second low-gray-level region and a second high-gray-level region based on the first mean of the gray levels in the first sub-histogram; calculating the adjustment coefficient for each gray level in the second low-gray-level region based on the second mean of the gray levels in the second low-gray-level region and the first mean, as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set; and calculating the adjustment coefficient for each gray level in the second high-gray-level region based on the third mean of the gray levels in the second high-gray-level region and the first mean, as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set.

[0017] In some embodiments, adjusting the gray level based on the selected adjustment coefficient, reference benchmark, and the range of gray levels in the first low gray level region includes: adjusting the gray level according to the formula Adjust the grayscale level; where p mid The reference benchmark is represented by α, which represents the selected adjustment factor, and g i Let range1 represent the gray level, and range1 represent the range of gray levels in the first low gray level region; the adjustment of gray levels based on the selected adjustment coefficient, reference benchmark, and the range of gray levels in the first high gray level region includes: adjusting the gray level according to the formula Adjust the grayscale level; where p mid The reference benchmark is represented by α, which represents the selected adjustment factor, and g j represents the gray level, and range2 represents the range of gray levels in the first high gray level region.

[0018] In some embodiments, merging the expanded second sub-histograms further includes: performing equalization processing on the expanded second sub-histograms respectively to obtain a third sub-histogram; performing smoothing processing on the third sub-histograms respectively to obtain a fourth sub-histogram; and merging the fourth sub-histograms to obtain a second histogram.

[0019] In some embodiments, generating the image after smoke removal includes: determining a grayscale value set based on the second histogram, the grayscale value set including at least one grayscale value, the at least one grayscale value corresponding to a grayscale level in the second histogram; for a pixel in the smoke image, selecting a grayscale value from the grayscale value set based on the grayscale level corresponding to the grayscale value of the pixel in each color channel, as the grayscale value of the pixel in the corresponding color channel in the image after smoke removal.

[0020] In some embodiments, merging the expanded second sub-histograms includes: merging the second sub-histograms under each color channel to obtain a second histogram under that color channel; generating the smoke-free image includes: determining a grayscale value set based on the second histogram under each color channel, the grayscale value set including at least one grayscale value, the at least one grayscale value corresponding to a grayscale level in the second histogram under that color channel; for each pixel in the second grayscale image under each color channel, selecting a grayscale value from the grayscale value set of that color channel based on the grayscale level corresponding to the pixel's grayscale value, as the grayscale value of the pixel in the third grayscale image; and generating the smoke-free image based on the third grayscale images under multiple color channels.

[0021] In some embodiments, the image dehazing method further includes: performing color level adjustment processing and / or brightness enhancement processing on the image after dehazing.

[0022] In some embodiments, the brightness enhancement processing of the image after smoke removal includes: for a pixel in the image after smoke removal, selecting a brightness gain from a brightness gain set according to the grayscale value of the pixel in each color channel, and adjusting the grayscale value of the pixel in that color channel according to the brightness gain; wherein the brightness gain set includes at least one brightness gain, and the at least one brightness gain corresponds to a grayscale value.

[0023] In some embodiments, in the brightness gain set, the brightness gain tends to increase as the grayscale value increases; or, the brightness gain set includes a first sub-brightness gain set, a second sub-brightness gain set, and a third sub-brightness gain set; the grayscale value corresponding to the brightness gain in the first sub-brightness gain set is less than that in the second sub-brightness gain set, and the grayscale value corresponding to the brightness gain in the second sub-brightness gain set is less than that in the third sub-brightness gain set; in the first sub-brightness gain set, the brightness gain tends to increase first and then decrease as the grayscale value increases; in the second sub-brightness gain set, the brightness gain remains unchanged as the grayscale value increases; and in the third sub-brightness gain set, the brightness gain tends to decrease as the grayscale value increases.

[0024] A second aspect of the embodiments of this specification provides a computer storage medium storing computer program instructions that, when executed, implement the steps of the method as described in the first aspect.

[0025] A third aspect of the embodiments of this specification provides an image processing system, comprising:

[0026] Endoscopes are used to capture images of smoke in the surgical environment;

[0027] An image processing device is configured to acquire a first histogram of the smoke image; segment the first histogram to obtain multiple first sub-histographs; perform contrast expansion on each segment of the first sub-histograph to obtain a second sub-histograph; merge the expanded second sub-histographs to obtain a second histogram; and generate an image after removing the smoke based on the second histogram.

[0028] Display device for displaying images after smoke removal.

[0029] The technical solution provided in the embodiments of this specification can obtain a first histogram of a smoke image; the first histogram can be segmented to obtain multiple first sub-historical segments; each first sub-historical segment can be contrast-expanded to obtain a second sub-historical segment; the expanded second sub-historical segments can be merged to obtain a second histogram; and an image after smoke removal can be generated based on the second histogram. By segmenting the histogram for contrast expansion, the local contrast of the smoke image can be improved. By merging the expanded sub-historical segments, the global contrast of the smoke image can be improved. Since both the local and global contrast of the smoke image are improved, the image after smoke removal is clearer and more transparent, thus improving the smoke removal effect. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a functional structure diagram of the image processing system in the embodiments of this specification;

[0032] Figure 2 This is a schematic diagram of the endoscope lens end in an embodiment of this specification;

[0033] Figure 3 This is a schematic flowchart of the image dehazing method in the embodiments of this specification;

[0034] Figure 4 This is a schematic diagram of the original histogram and the merged histogram in the embodiments of this specification;

[0035] Figure 5 This is a schematic diagram of the brightness gain set in the embodiments of this specification;

[0036] Figure 6 This is a schematic diagram of the brightness gain set in the embodiments of this specification;

[0037] Figure 7 This is a schematic diagram of the smoke image and the image after smoke removal in the embodiments of this specification. Detailed Implementation

[0038] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. The specific embodiments described herein are only used to explain this disclosure, and not to limit this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure are within the scope of protection of this disclosure. In addition, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0039] Please see Figure 1 and Figure 2 This specification provides an image processing system. The image processing system may include an endoscope 11, an image processing device 12, a display device 13, and a lighting device 14.

[0040] The endoscope 11 is a slender cylindrical structure that can be inserted into the patient's organs through an incision or natural cavity on the patient's body surface during surgery. Depending on the application scenario, the endoscope 11 may include a bronchoscope, laparoscope, etc. An imaging component is provided at the tip of the endoscope 11. The imaging component may include an image sensor and a lens. The illumination device 14 provides a light source. The light source is transmitted to the tip of the endoscope 11 via optical fiber to illuminate the surgical environment. Illumination brightens the field of view of the surgical environment, allowing the image sensor to capture images from the lens. During surgery, the surgeon may use surgical instruments, such as an electrocautery knife, to cauterize tissues or organs. Cauterizing tissues or organs produces smoke. Therefore, the images captured by the endoscope 11 in the surgical environment will contain smoke.

[0041] The image processing device 12 is used to acquire smoke images collected by the endoscope, perform defogging processing on the smoke images to obtain a smoke-free image, and provide the smoke-free image to the display device 13. The display device 13 is used to display the smoke-free image. The smoke-free image has improved local and global contrast, making the image clearer and more transparent.

[0042] In practical applications, the image processing device 12 can acquire smoke images captured by an endoscope; and can provide the smoke-free image to the display device 13. Alternatively, the image processing device 12 can also acquire video captured by an endoscope; can extract video frames from the video as smoke images; can perform defogging processing on the smoke images to obtain smoke-free images; can generate smoke-free videos based on the smoke-free images; and can provide the smoke-free videos to the display device 13. The image processing device 12 and the display device 13 can be different devices, or they can be integrated into a single device.

[0043] This specification provides an image dehazing method that can be applied to the image processing device 12.

[0044] Please see Figure 3 The image dehazing method may include the following steps.

[0045] Step S21: Obtain the first histogram of the smoke image.

[0046] In some embodiments, the smoke image may include an endoscopic image. The smoke image is obtained in a smoke environment. The smoke image can be acquired by means of a video acquisition, or it may also include video frames extracted from a video. The smoke image may be a color image. The color space of the smoke image may include YUV color space, YCbCr color space, RGB color space, HSL color space, etc. The color space may be a mathematical model that uses multiple color channels to describe colors. The YUV color space may include three color channels: Y, U, and V. The YCbCr color space may include three color channels: Y, Cb, and Cr. The RGB color space may include three color channels: R, G, and B. Each pixel in the smoke image may have a pixel value. The smoke image may be a color image, such that the pixel value may include multiple grayscale values. The multiple grayscale values ​​may correspond to multiple color channels. The range of the grayscale values ​​may be 0 to 255. Of course, the grayscale values ​​may be other ranges. It is worth noting that each pixel in a grayscale image may also have a pixel value. The pixel value of each pixel in a grayscale image may include a grayscale value.

[0047] In some embodiments, the first histogram may include a histogram of the grayscale image of the smoke image. The histogram reflects the statistical characteristics of pixels in the grayscale image. Specifically, the histogram can be used to represent the number of pixels corresponding to each grayscale level in the grayscale image. Here, the grayscale level can be understood as a grayscale grade. Specifically, in the histogram, each grayscale value of the grayscale image can be considered as a grayscale level, or multiple grayscale values ​​can be combined into a single grayscale level. For example, the pixel value of each pixel in the grayscale image may include one grayscale value. The range of the grayscale value can be 0 to 255. If each grayscale value in the grayscale image is considered as a grayscale level, then the histogram includes 256 grayscale levels. Alternatively, every two grayscale values ​​in the grayscale image can be considered as a grayscale level; specifically, for example, grayscale values ​​0 and 1 can be considered as one grayscale level, and grayscale values ​​2 and 3 as another grayscale level. Then the histogram includes 128 grayscale levels. Of course, you can also use every 4 gray values ​​in a grayscale image as a gray level, then the histogram will include 64 gray levels.

[0048] In some embodiments, the smoke image can be grayscaled to obtain a first grayscale image; the first histogram can be calculated based on the first grayscale image. Grayscale processing removes color information from a color image, thereby converting it into a grayscale image. For example, the color space of the smoke image can be RGB. The smoke image can be grayscaled using the formula Gray = R × 0.3 + G × 0.59 + B × 0.11 to obtain the first grayscale image. Here, Gray represents the grayscale value of a pixel in the first grayscale image, R represents the grayscale value of that pixel in the smoke image under the R color channel, G represents the grayscale value of that pixel in the smoke image under the G color channel, and B represents the grayscale value of that pixel in the smoke image under the B color channel.

[0049] Step S23: Divide the first histogram into multiple segments of the first sub-histogram.

[0050] In some embodiments, an image segmentation algorithm can be used to obtain a segmentation threshold for the first grayscale image; the first histogram can then be segmented according to the segmentation threshold to obtain multiple first sub-historical images. The image segmentation algorithm may include Otsu's method, the mean iteration method, the maximum entropy method, etc. Different first sub-historical images reflect different content in the image. Through segmentation, it is convenient to perform appropriate contrast expansion based on the content in the image, thereby achieving local contrast expansion.

[0051] In some embodiments, the number of segmentation thresholds can be one, and the number of multiple first sub-historical segments can be two. For example, a region in the first histogram with a gray level less than or equal to the segmentation threshold can be considered as one first sub-historical segment, and a region in the first histogram with a gray level greater than the segmentation threshold can be considered as another first sub-historical segment. In some scenario examples, the first histogram can be segmented into foreground and background based on the segmentation threshold to obtain two first sub-historical segments. The two first sub-historical segments can include a foreground sub-historical segment and a background sub-historical segment. The foreground sub-historical segment includes regions in the first histogram with a gray level greater than the segmentation threshold, used to reflect the foreground content of the image. The background sub-historical segment includes regions in the first histogram with a gray level less than or equal to the segmentation threshold, used to reflect the background content of the image. Of course, the number of segmentation thresholds can also be multiple, and the number of multiple first sub-historical segments can also be other numbers.

[0052] Step S25: Perform contrast expansion on each segment of the first sub-histogram to obtain the second sub-histogram.

[0053] In some embodiments, each segment of the first sub-histogram can be contrast-expanded to obtain a second sub-histogram. By segmenting the histogram for contrast expansion, the local contrast of the smoke image can be improved. For example, the gray level range of a certain segment of the first sub-histogram can be 60–140. By contrast-expanding this first sub-histogram, a second sub-histogram can be obtained. The gray level range of the second sub-histogram can be 20–180.

[0054] In some embodiments, a reference benchmark can be obtained for each segment of the first sub-histogram; the first sub-histogram can be divided into a first low gray-level region and a first high gray-level region based on the reference benchmark; the gray levels in the first low gray-level region can be moved away from the reference benchmark in a decreasing direction; the gray levels in the first high gray-level region can be moved away from the reference benchmark in an increasing direction. By using the reference benchmark as a boundary, the gray levels in the first sub-histogram can be separated to both sides, thereby expanding the contrast of the first sub-histogram and mapping the first sub-histogram to a second sub-histogram.

[0055] In some embodiments, a gray level can be selected from the first sub-histogram as a reference benchmark. Alternatively, the maximum and minimum gray levels in the first sub-histogram can be obtained; the median of the maximum and minimum gray levels can be calculated as the reference benchmark for the first sub-histogram. For example, it can be based on the formula... Calculate the reference datum. min p represents the smallest one. max p represents the largest of the three. midThis represents the median. Specifically, for example, if the maximum and minimum gray levels in a certain sub-histogram are 100 and 30 respectively, then the reference value for the first sub-histogram can be 65.

[0056] In some embodiments, regions in the first sub-histogram with gray levels less than or equal to the reference reference can be designated as first low-gray-level regions, and regions in the first sub-histogram with gray levels greater than the reference reference can be designated as first high-gray-level regions. Gray levels in the first low-gray-level regions can be moved away from the reference reference in a decreasing direction. After moving away, the relative magnitude relationship of each gray level in the first low-gray-level region can remain unchanged. Similarly, gray levels in the first high-gray-level regions can be moved away from the reference reference in an increasing direction. After moving away, the relative magnitude relationship of each gray level in the first high-gray-level region can remain unchanged.

[0057] In some embodiments, an adjustment coefficient set can be determined for each segment of the first sub-histogram. The adjustment coefficient set is used to adjust the gray levels in the first sub-histogram. Specifically, the adjustment coefficient set may include at least one adjustment coefficient, which may correspond to a gray level in the first sub-histogram. The adjustment coefficient can be understood as a local coefficient used to adjust its corresponding gray level. In practical applications, a first mean value of the gray levels in the first sub-histogram can be obtained; based on the first mean value, the first sub-histogram can be divided into a second low gray level region and a second high gray level region. Specifically, the region in the first sub-histogram where the gray level is less than or equal to the first mean value can be considered the second low gray level region; the region in the first sub-histogram where the gray level is greater than the first mean value can be considered the second high gray level region. The mean value of the gray levels is used to represent the central tendency of the gray levels corresponding to each pixel in the histogram. You can multiply each gray level in the histogram by the number of pixels corresponding to that gray level; you can add the results of multiplying the gray levels; you can divide the sum by the number of pixels in the histogram to obtain the mean of the gray levels in the histogram.

[0058] The second mean of gray levels in the second low gray level region can be obtained. For each gray level in the second low gray level region, an adjustment coefficient can be calculated based on the second mean and the first mean, serving as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set. The third mean of gray levels in the second high gray level region can be obtained. For each gray level in the second high gray level region, an adjustment coefficient can be calculated based on the third mean and the first mean, serving as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set. Further, a contrast coefficient can be preset. The contrast coefficient can be understood as a global coefficient used to adjust each gray level in the first histogram. Specifically, for each gray level in the second low gray level region, an adjustment coefficient can be calculated based on the second mean, the first mean, and the contrast coefficient. For example, for each gray level in the second low gray level region, an adjustment coefficient can be calculated based on the formula... Calculate the adjustment factor for this gray level. Where c represents the contrast coefficient, and x... m Let x represent the first mean. mu x represents the third mean. ml This represents the second mean. For each gray level in the second high gray level region, an adjustment coefficient for that gray level can be calculated based on the third mean, the first mean, and the contrast coefficient. For example, for each gray level in the second high gray level region, an adjustment coefficient can be calculated based on the formula... Calculate the adjustment factor for this gray level. Where c represents the contrast coefficient, and x... m Let x represent the first mean. mu x represents the third mean. ml This represents the second mean.

[0059] In some embodiments, for each gray level in the first low gray level region, a corresponding adjustment coefficient can be selected from the set of adjustment coefficients; the gray level can be adjusted based on the selected adjustment coefficient, a reference standard, and the range of gray levels in the first low gray level region. The range of gray levels in the first low gray level region can be the difference between the maximum and minimum gray levels in the first low gray level region. Specifically, the gray level can be adjusted according to a first mapping rule. For example, the first mapping rule may include a formula. p mid The reference standard is represented by α, and the adjustment factor is represented by g. iThis represents the grayscale level, where `range1` represents the range of grayscale levels in the first low grayscale region. For each grayscale level in the first high grayscale region, a corresponding adjustment coefficient can be selected from the set of adjustment coefficients. The grayscale level can be adjusted based on the selected adjustment coefficient, the reference standard, and the range of grayscale levels in the first high grayscale region. The range of grayscale levels in the first high grayscale region can be the difference between the maximum and minimum grayscale levels in the first high grayscale region. Specifically, the grayscale level can be adjusted according to the second mapping rule. For example, the second mapping rule may include the formula... p mid The reference standard is represented by α, and the adjustment factor is represented by g. j represents the gray level, and range2 represents the range of gray levels in the first high gray level region.

[0060] Step S27: Merge the expanded second sub-histograms to obtain the second histogram.

[0061] In some embodiments, the expanded second sub-histograms may contain one or more overlapping gray levels. For each overlapping gray level, the number of pixels corresponding to that gray level in each second sub-histogram can be added together to obtain the number of pixels corresponding to that gray level in the second histogram. Of course, for non-overlapping gray levels, the number of pixels corresponding to that gray level in the second sub-histogram can be directly used as the number of pixels corresponding to that gray level in the second histogram.

[0062] By merging the expanded second sub-histograms, the global contrast of the smoke image can be improved. Figure 4 This is a schematic diagram of the first and second histograms of a smoke image in an embodiment of this specification. According to... Figure 4 It can be seen that the second histogram is expanded compared to the first histogram of the smoke image.

[0063] In some embodiments, the expanded second sub-histograms can be directly merged. Alternatively, the expanded second sub-histograms can be individually equalized to obtain a third sub-histogram; the equalized third sub-histograms can then be merged. Alternatively, the expanded second sub-histograms can be individually smoothed to obtain a fourth sub-histogram; the smoothed fourth sub-histograms can then be merged. Alternatively, the expanded second sub-histograms can be individually equalized to obtain a third sub-histogram; the equalized third sub-histograms can be individually smoothed to obtain a fourth sub-histogram; the smoothed fourth sub-histograms can then be merged. The equalization process transforms the histogram into a uniform distribution (equivalent distribution), ensuring pixels are evenly distributed across gray levels, thereby enhancing image contrast. The smoothing process reduces the difference in the number of pixels corresponding to adjacent gray levels in the histogram, making the image smoother. For example, the histogram can be smoothed using interpolation methods.

[0064] Step S29: Generate the image after removing smoke based on the second histogram.

[0065] In some embodiments, a grayscale value set can be determined based on the second histogram. The grayscale value set may include at least one grayscale value, which may correspond to a grayscale level in the second histogram. Thus, for each pixel in the smoke image, a corresponding grayscale value can be selected from the grayscale value set based on the grayscale level corresponding to the pixel's grayscale value in each color channel, and used as the grayscale value of that pixel in the same color channel of the image after smoke removal. In this way, the smoke image can be mapped to the smoke-free image using the grayscale value set. The grayscale value set also helps maintain consistent image color, avoiding color cast and color shift problems caused by smoke removal.

[0066] For each gray level in the second histogram, a gray level set can be obtained from the second histogram; the number of pixels corresponding to each gray level in the gray level set can be added together; the sum can be divided by the number of pixels in the second histogram; the division result can be multiplied by a set threshold to obtain the gray value corresponding to that gray level. The gray level set may include the gray level itself. And / or, the gray level set may also include other gray levels in the second histogram that are smaller than the gray level. The set threshold may include the maximum gray value of the image, for example, 255.

[0067] A grayscale value set can be directly generated from the second histogram. Alternatively, the smoke image may include video frames extracted from a video. The video frame corresponding to the smoke image can be used as the current video frame. The second histogram obtained from the current video frame can be fused with the second histogram obtained from the previous video frame; a grayscale value set can be generated from the fused second histogram. For example, the two second histograms can be weighted and fused. Specifically, for example, the weight of the second histogram obtained from the current video frame can be e, and the weight of the second histogram obtained from the previous video frame can be f. The number of pixels corresponding to each gray level in the second histogram obtained from the current video frame can be multiplied by e; the number of pixels corresponding to the corresponding gray level in the second histogram obtained from the previous video frame can be multiplied by f; the two multiplication results can be added together to obtain the number of pixels corresponding to the corresponding gray level in the fused second histogram. Where f = 1 - e. For example, e can be 0.9, and f can be 0.1. This can reduce the differences between adjacent video frames in the video.

[0068] For example, the color space of the smoke image may include the RGB color space, which may include three color channels: R, G, and B. For each pixel in the smoke image, a corresponding gray value can be selected from the set of gray values ​​based on the gray level corresponding to the pixel's gray value in the R color channel, and this gray value can be used as the gray value of the pixel in the R color channel of the image after smoke removal; similarly, a corresponding gray value can be selected from the set of gray values ​​based on the gray level corresponding to the pixel's gray value in the G color channel, and similarly, a corresponding gray value can be selected from the set of gray values ​​based on the gray level corresponding to the pixel's gray value in the B color channel, and this gray value can be used as the gray value of the pixel in the B color channel of the image after smoke removal.

[0069] In some embodiments, the image after smoke removal may also be subjected to color level adjustment and / or brightness enhancement processing.

[0070] Automatic level adjustment (Auto Level) can be performed on images after smoke removal. Brightness enhancement can be applied to images after smoke removal based on a brightness gain set. Brightness enhancement improves the brightness of the image after smoke removal, avoiding issues such as underexposure caused by smoke removal. The brightness gain set can include at least one brightness gain, which corresponds to a grayscale value. The brightness gain set can be preset. See also... Figure 5 In the brightness gain set, the brightness gain can tend to increase as the grayscale value increases. Alternatively, please refer to... Figure 6The brightness gain set may include a first sub-brightness gain set, a second sub-brightness gain set, and a third sub-brightness gain set. The grayscale value corresponding to the brightness gain in the first sub-brightness gain set may be smaller than that in the second sub-brightness gain set, and the grayscale value corresponding to the brightness gain in the second sub-brightness gain set may be smaller than that in the third sub-brightness gain set. In the first sub-brightness gain set, the brightness gain initially increases and then decreases as the grayscale value increases. In the second sub-brightness gain set, the brightness gain remains constant as the grayscale value increases. In the third sub-brightness gain set, the brightness gain decreases as the grayscale value increases.

[0071] For each pixel in the smoke-free image, a corresponding luminance gain can be selected from the luminance gain set based on the pixel's grayscale value in each color channel. The selected luminance gain can then be used to adjust the pixel's grayscale value in that color channel. For example, the selected luminance gain can be multiplied by the pixel's grayscale value in that color channel to obtain the adjusted grayscale value. For instance, the color space of the smoke-free image may include the RGB color space. Similarly, for each pixel in the smoke-free image, a corresponding luminance gain can be selected from the luminance gain set based on the pixel's grayscale value in the R color channel, and multiplied by the selected luminance gain to obtain the adjusted grayscale value in the R color channel; the same applies to the pixel's grayscale value in the G color channel; and the same applies to the pixel's grayscale value in the B color channel.

[0072] In some embodiments, a second grayscale image of the smoke image across multiple color channels can be obtained. The second grayscale image can be understood as a single-channel image. A first histogram can be calculated for each second grayscale image. Each first histogram can be segmented to obtain multiple first sub-histographs. Contrast expansion can be performed on each first sub-histograph to obtain a second sub-histograph. Thus, each color channel in the multiple color channels can have multiple second sub-histographs. The second sub-histographs under each color channel can be merged to obtain the second histogram for that color channel. An image excluding smoke can be generated based on the second histograms under the multiple color channels.

[0073] Specifically, an image segmentation algorithm can be used to obtain the segmentation threshold for each second grayscale image; based on the segmentation threshold, the foreground and background of the first histogram of the second grayscale image can be segmented to obtain multiple first sub-histograms.

[0074] Specifically, the grayscale value set can be determined separately for each second histogram. For each pixel in the second grayscale image under each color channel, a grayscale value can be selected from the grayscale value set of that color channel based on the grayscale level corresponding to the pixel's grayscale value, and used as the pixel's grayscale value in the third grayscale image. The third grayscale image can be understood as a single-channel image. An image after removing smoke can be generated based on the third grayscale images under the multiple color channels.

[0075] Figure 7 This is a schematic diagram showing the smoke image and the image after smoke removal. According to... Figure 7 It can be seen that, compared to the smoke image, the histogram of the image after removing the smoke is expanded, the contrast is improved, and the image becomes clearer.

[0076] The image dehazing method of this specification embodiment can obtain a first histogram of a smoke image; segment the first histogram to obtain multiple first sub-historical segments; perform contrast expansion on each first sub-historical segment to obtain a second sub-historical segment; merge the expanded second sub-historical segments to obtain a second histogram; and generate a smoke-free image based on the second histogram. By performing segmented contrast expansion on the histogram, the local contrast of the smoke image can be improved. By merging the expanded sub-historical segments, the global contrast of the smoke image can be improved. Since both the local and global contrast of the smoke image are improved, the smoke-free image is clearer and more transparent, thus improving the smoke removal effect.

[0077] This specification also provides an image defogging device, which includes the following units.

[0078] The acquisition unit is used to acquire the first histogram of the smoke image;

[0079] A segmentation unit is used to segment the first histogram to obtain multiple segments of the first sub-historical map;

[0080] An extension unit is used to perform contrast expansion on each segment of the first sub-histogram to obtain a second sub-histogram;

[0081] The merging unit is used to merge the expanded second sub-histograms to obtain the second histogram;

[0082] The generation unit is used to generate an image after removing smoke based on the second histogram.

[0083] This specification also provides a computing device through its embodiments.

[0084] The computing device may include a memory and a processor.

[0085] In this embodiment, the memory includes, but is not limited to, Dynamic Random Access Memory (DRAM) and Static Random Access Memory (SRAM). The memory can be used to store computer instructions.

[0086] In this embodiment, the processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The processor can be used to execute the computer instructions to implement... Figure 3 The corresponding implementation examples.

[0087] This specification provides a computer storage medium storing computer program instructions, which, when executed, implement... Figure 3 The steps of the method are described.

[0088] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0089] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. A computer can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0090] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0091] Those skilled in the art will understand that the descriptions of the various embodiments have different focuses, and parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Furthermore, it is understood that those skilled in the art, after reading this specification, can conceive of any combination of some or all of the embodiments listed in this specification without creative effort, and such combinations are also within the scope of disclosure and protection of this specification.

Claims

1. A method for dehazing endoscopic images, characterized in that, include: Acquire a first histogram of a smoke image, wherein the smoke image is an endoscopic image in a surgical environment; The first histogram is segmented into foreground and background to obtain multiple first sub-historical images; Each segment of the first sub-histogram is contrast-expanded to obtain a second sub-histogram; the multiple segments of the first sub-histogram include a foreground sub-histogram and a background sub-histogram, the foreground sub-histogram is used to reflect the foreground content of the smoke image, and the background sub-histogram is used to reflect the background content of the smoke image; The step of performing contrast expansion on each segment of the first sub-histogram includes: obtaining a reference benchmark for the first sub-histogram; dividing the first sub-histogram into a first low gray level region and a first high gray level region according to the reference benchmark; moving the gray level in the first low gray level region away from the reference benchmark in a decreasing direction; and moving the gray level in the first high gray level region away from the reference benchmark in an increasing direction. The expanded second sub-histograms are merged to obtain the second histogram; Based on the second histogram, generate the image after removing the smoke; The image after smoke removal is subjected to brightness enhancement processing based on a brightness gain set; the brightness gain set includes a first sub-brightness gain set, a second sub-brightness gain set, and a third sub-brightness gain set; the gray value corresponding to the brightness gain in the first sub-brightness gain set is less than that in the second sub-brightness gain set, and the gray value corresponding to the brightness gain in the second sub-brightness gain set is less than that in the third sub-brightness gain set; in the first sub-brightness gain set, the brightness gain first increases and then decreases as the gray value increases; in the second sub-brightness gain set, the brightness gain remains unchanged as the gray value increases; in the third sub-brightness gain set, the brightness gain decreases as the gray value increases.

2. The endoscopic image defogging method according to claim 1, characterized in that, The first histogram includes a histogram of the grayscale image of the smoke image; the segmentation of the first histogram into foreground and background includes: The segmentation threshold of the grayscale image is obtained using an image segmentation algorithm; The first histogram is segmented into foreground and background based on the segmentation threshold to obtain two first sub-historical segments, each including a foreground sub-historical segment and a background sub-historical segment.

3. The endoscopic image defogging method according to claim 2, characterized in that, The grayscale image includes a first grayscale image obtained by grayscale conversion of the smoke image; the step of obtaining the segmentation threshold of the grayscale image using an image segmentation algorithm includes: An image segmentation algorithm is used to obtain the segmentation threshold of the first grayscale image.

4. The endoscopic image defogging method according to claim 2, characterized in that, The grayscale image includes a second grayscale image of the smoke image across multiple color channels; the step of obtaining the segmentation threshold of the grayscale image using an image segmentation algorithm includes: The segmentation threshold for each of the second grayscale images is obtained using an image segmentation algorithm. The step of segmenting the first histogram into foreground and background based on the segmentation threshold includes: Based on the segmentation threshold of each second grayscale image, the first histogram of the second grayscale image is segmented into foreground and background to obtain two first sub-historical segments, which include a foreground sub-historical segment and a background sub-historical segment.

5. The endoscopic image defogging method according to claim 1, characterized in that, The reference benchmark for obtaining the first sub-histogram includes: Obtain the maximum and minimum gray levels in the first sub-histogram; The median of the maximum and the minimum is calculated and used as a reference benchmark for the first sub-histogram.

6. The endoscopic image defogging method according to claim 1, characterized in that, The endoscopic image defogging method also includes: An adjustment coefficient set is determined based on the first sub-histogram, the adjustment coefficient set including at least one adjustment coefficient, the at least one adjustment coefficient corresponding to a gray level in the first sub-histogram; The step of moving the gray levels in the first low gray level region away from the reference reference in a decreasing direction includes: For the gray levels in the first low gray level region, select the corresponding adjustment coefficient from the set of adjustment coefficients, and adjust the gray levels according to the selected adjustment coefficient, the reference benchmark, and the range of gray levels in the first low gray level region. The step of moving the gray levels in the first high gray level region away from the reference reference in an increasing direction includes: For the gray levels in the first high gray level region, select the appropriate adjustment coefficient from the set of adjustment coefficients, and adjust the gray levels according to the selected adjustment coefficient, the reference benchmark, and the range of gray levels in the first high gray level region.

7. The endoscopic image defogging method according to claim 6, characterized in that, The step of determining the set of adjustment coefficients based on the first sub-histogram includes: Based on the first mean of the gray levels in the first sub-histogram, the first sub-histogram is divided into a second low gray level region and a second high gray level region. Based on the second mean value of the gray levels in the second low gray level region and the first mean value, calculate the adjustment coefficient for each gray level in the second low gray level region, and use it as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set; Based on the third mean and the first mean of the gray levels in the second high gray level region, the adjustment coefficient for each gray level in the second high gray level region is calculated, and used as the adjustment coefficient corresponding to that gray level in the adjustment coefficient set.

8. The endoscopic image defogging method according to claim 6, characterized in that, The adjustment of gray levels based on the selected adjustment coefficient, reference benchmark, and the range of gray levels in the first low gray level region includes: According to the formula Adjust the grayscale levels; among them, Indicates the reference datum, This indicates the selected adjustment factor. Indicates grayscale level. This represents the range of gray levels in the first low gray level region; The adjustment of gray levels based on the selected adjustment coefficient, reference benchmark, and the range of gray levels in the first high gray level region includes: According to the formula Adjust the grayscale levels; among them, Indicates the reference datum, This indicates the selected adjustment factor. Indicates grayscale level. This represents the range of gray levels in the first high gray level region.

9. The endoscopic image defogging method according to claim 1 or 3, characterized in that, The merging of the expanded second sub-histogram also includes: The expanded second sub-histogram is then subjected to equalization processing to obtain the third sub-histogram; The third sub-histogram is smoothed to obtain the fourth sub-histogram; The fourth sub-histogram is merged to obtain the second histogram.

10. The method for dehazing endoscopic images according to claim 1 or 3, characterized in that, The generation of the smoke-free image includes: A set of gray values ​​is determined based on the second histogram, the set of gray values ​​including at least one gray value, the at least one gray value corresponding to a gray level in the second histogram; For a pixel in a smoke image, a gray value is selected from the set of gray values ​​based on the gray level corresponding to the pixel's gray value in each color channel, and used as the gray value of the pixel in the corresponding color channel in the image after removing the smoke.

11. The endoscopic image defogging method according to claim 4, characterized in that, The merging of the expanded second sub-histogram includes: The second sub-histograms under each color channel are merged to obtain the second histogram under that color channel; The generation of the smoke-free image includes: A grayscale value set is determined based on the second histogram of each color channel. The grayscale value set includes at least one grayscale value, and the at least one grayscale value corresponds to the grayscale level in the second histogram of that color channel. For each pixel in the second grayscale image under each color channel, a grayscale value is selected from the grayscale value set of that color channel according to the grayscale level corresponding to the pixel's grayscale value, and used as the grayscale value of the pixel in the third grayscale image. Based on the third grayscale image with multiple color channels, an image after removing smoke is generated.

12. The endoscopic image defogging method according to claim 1, characterized in that, The endoscopic image defogging method also includes: Perform color level adjustment on the image after removing the smoke.

13. The endoscopic image defogging method according to claim 1, characterized in that, The step of performing brightness enhancement processing on the smoke-free image based on the brightness gain set includes: For each pixel in the image after smoke removal, a brightness gain is selected from the brightness gain set based on the grayscale value of the pixel in each color channel, and the grayscale value of the pixel in that color channel is adjusted according to the brightness gain.

14. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed, implement the steps of the method according to any one of claims 1 to 13.

15. An image processing system, characterized in that, include: An endoscope is used to acquire smoke images in the surgical environment, wherein the smoke images are endoscopic images; An image processing device is configured to acquire a first histogram of the smoke image; segment the first histogram into foreground and background to obtain multiple first sub-historical images; perform contrast expansion on each segment of the first sub-historical image to obtain a second sub-historical image; and merge the expanded second sub-historical images to obtain a second histogram. Based on the second histogram, generate the image after removing the smoke; The image after smoke removal is enhanced according to the brightness gain set; the multi-segment first sub-histogram includes a foreground sub-histogram and a background sub-histogram, the foreground sub-histogram is used to reflect the foreground content of the smoke image, and the background sub-histogram is used to reflect the background content of the smoke image; The step of contrast expansion for each segment of the first sub-histogram includes: obtaining a reference benchmark for the first sub-histogram; dividing the first sub-histogram into a first low gray-level region and a first high gray-level region based on the reference benchmark; moving the gray levels in the first low gray-level region away from the reference benchmark in a decreasing direction; and moving the gray levels in the first high gray-level region away from the reference benchmark in an increasing direction. The brightness gain set includes a first sub-brightness gain set, a second sub-brightness gain set, and a third sub-brightness gain set. The gray values ​​corresponding to the brightness gains in the first sub-brightness gain set are less than those in the second sub-brightness gain set, and the gray values ​​corresponding to the brightness gains in the second sub-brightness gain set are less than those in the third sub-brightness gain set. In the first sub-brightness gain set, the brightness gain initially increases and then decreases as the gray value increases. In the second sub-brightness gain set, the brightness gain remains constant as the gray value increases. In the third sub-brightness gain set, the brightness gain decreases as the gray value increases. Display devices used to display images that have undergone brightness enhancement processing.

Images