Endoscope image dehazing method and apparatus

Abstract

The application relates to an endoscope image smoke removing method and device, relates to the field of image processing technology, and is characterized in that: firstly, the depth information of an endoscope smoke image is modeled, model parameters are obtained to restore the depth information of the image, then the transmittance of an endoscope operation scene and the atmospheric light value are calculated according to the depth information of the endoscope smoke image, and finally, the smoke-free image to be restored is obtained based on an atmospheric scattering model, so that a clear smoke-free image is effectively restored, the smoke removing effect is good, the efficiency is high, the time consumption is short, and the problem of image color distortion easily occurring during smoke removing is solved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for removing smoke from endoscopic images. Background Technology

[0002] With the increasing prevalence of minimally invasive surgery, endoscopes can be inserted into the human body through natural orifices or small incisions, allowing doctors to visualize the patient's internal condition. Therefore, endoscopes can acquire surgical images or video data, aiding doctors in diagnosing the patient's condition and performing surgery. During surgery, lasers or ablation of human tissue can easily generate smoke, which reduces image quality. Doctors need to clean the endoscope aperture before continuing the procedure, making the operation difficult and affecting its progress. Therefore, it is essential to use physical methods to remove smoke and image processing algorithms to eliminate it.

[0003] Most existing dehazing methods establish a physical imaging model based on the causes of image degradation under fog conditions, and restore a clear image from the captured image. However, such methods usually result in color distortion after image processing. Summary of the Invention

[0004] To address the color distortion problem in endoscopic smoke images after processing using existing image processing techniques, this invention provides a method for removing smoke from endoscopic images. The technical solution is as follows:

[0005] A method for removing smoke from endoscopic images involves first modeling the depth information of the endoscopic smoke image, obtaining model parameters to recover the depth information of the image, then calculating the transmittance and atmospheric light value of the endoscopic surgical scene based on the depth information of the endoscopic smoke image, and finally obtaining the smoke-free image to be recovered based on an atmospheric scattering model.

[0006] The above technical solution, based on an atmospheric scattering model, models the depth information of smoke images and obtains model parameters to recover the image's depth information. Using the depth information of the smoke image, transmittance and atmospheric light values ​​can be calculated through the atmospheric scattering model, improving the estimation accuracy of transmittance and atmospheric light values, reducing color distortion after smoke removal in surgical scenes, and enabling more realistic image recovery, allowing more details to be seen. The final smoke-free image shows good smoke removal performance, high efficiency, and short processing time, and solves the problem of color distortion that easily occurs when removing smoke.

[0007] A method for removing smoke from endoscopic images includes the following steps:

[0008] Step 1: Obtain the initial endoscopic image I(x);

[0009] Step 2: Convert the acquired initial image from RGB color space to HSV color space to obtain an HSV color space image, and calculate the depth information D;

[0010] Step 3: Perform joint bilateral filtering on the depth information D to obtain Dfilter;

[0011] Step 4: Calculate the transmittance t(x) based on the filtered depth information Dfilter;

[0012] Step 5: Calculate the atmospheric light value A based on the depth information D;

[0013] Step 6: Based on the transmittance t(x) and the atmospheric light value A, obtain the smoke-free image to be recovered using the atmospheric scattering model.

[0014] Optionally, in step 2, the formula for calculating the depth information D is:

[0015] D(x)=θ0+θ1V(x)+θ2S(x)+ε(x)

[0016] Where D is depth information, V is brightness, S is saturation; θ0, θ1, θ2 are linear coefficients, and ε is a random variable representing the random graph of the model.

[0017] Using the above technical solution, after acquiring the initial image from the endoscope, the image is first converted from the RGB color space to the HSV color space. Furthermore, in some implementation processes, the depth information of the smoke image is calculated based on the image brightness and saturation in the HSV color space. Specifically: in a smoke scene, there is a linear relationship between the image depth, brightness, and saturation: D(x)=θ0+θ1V(x)+θ2S(x)+ε(x), where D is the depth information, V is the brightness, S is the saturation; θ0, θ1, and θ2 are linear coefficients, and ε is a random variable representing a random graph of the model, resulting in fast calculation speed.

[0018] Optionally, in step 3, the depth information D is subjected to joint bilateral filtering, and the specific formula is as follows:

[0019]

[0020] Where x is the coordinate of the current pixel, k is the normalization coefficient, Ω(x) is the window centered at x, q is the coordinate of the pixel within the window, f is the spatial domain filtering function, and g is the range filtering function, the calculation formula of which is as follows:

[0021]

[0022]

[0023] Where, σd These are the spatial filtering coefficients, σ r These are the range filtering coefficients.

[0024] By using the above technical solution to perform joint bilateral filtering on the depth information D, the white block effect can be avoided.

[0025] Optionally, in step 4, the formula for calculating the transmittance t(x) based on the filtered depth information Dfilter is:

[0026] t(x)=exp(-beta*((Dfilter(x)+0.5) 2 -1))

[0027]

[0028] Where beta is the atmospheric scattering coefficient, ranging from 0 to 1, a is the minimum threshold value, and b is the maximum threshold value.

[0029] Using the above technical solution, the transmittance t(x) is calculated to provide parameters for subsequent calculations to recover smoke-free images.

[0030] Optionally, the specific steps of step 5 are as follows: perform minimum value filtering on the depth information D, sort the processed image data according to the principle of descending, select the first 0.1% of pixels in the processed image data, and take the average value of the brightness of these pixels in the original image as the atmospheric light value A.

[0031] Wherein, atmospheric light value A = MAX(A, Amax), Amax is the maximum threshold value, and MAX is the maximum value function.

[0032] The above technical solution solves the problem of local block blurring in the recovered smoke-free image, ensuring the clarity of the recovered smoke-free image.

[0033] Optionally, the formula for obtaining the smoke-free image to be recovered from the atmospheric scattering model is as follows:

[0034]

[0035] J(x) is the smoke-free image.

[0036] The above technical solution, based on the atmospheric scattering model, models the depth information of the smoke image, obtains the model parameters to recover the depth information of the image, and achieves good smoke removal effect. It is efficient and time-saving, and solves the problem of image color distortion that easily occurs when removing smoke.

[0037] An endoscopic image smoke removal device includes a memory and a processor. The memory is electrically connected to the processor. The memory is pre-installed with an endoscopic image smoke removal program designed for an endoscopic image smoke removal method. An initial endoscopic image is stored in the memory. The processor runs the endoscopic image smoke removal program to process the endoscopic image and generate a smoke-free image.

[0038] Optionally, a display device may also be provided, which is electrically connected to the processor. The processor displays the smoke-free image generated after running the endoscopic image desmearing program through the display device.

[0039] Through the above technical solution, the images captured by the endoscope are stored in the memory, the processor runs the endoscope image desmoke removal program to process the endoscope image to generate a smoke-free image, and displays the generated smoke-free image through the display device. The original endoscope image can also be displayed simultaneously for comparison.

[0040] In summary, the present invention has at least one of the following beneficial technical effects:

[0041] This invention models the depth information of smoke images based on an atmospheric scattering model, obtains model parameters to recover the image's depth information, and uses the depth information of the smoke image to calculate transmittance and atmospheric light value through the atmospheric scattering model, thereby effectively recovering a clear image without smoke. It has good effect, high efficiency, and short time consumption, and solves the problem of image color distortion that easily occurs when removing smoke. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the endoscopic image smoke removal method of the present invention;

[0043] Figure 2 This is the initial endoscopic image of Embodiment 1 of the present invention;

[0044] Figure 3 This is an image depth information effect diagram of an embodiment of the present invention;

[0045] Figure 4 This is an image depth information effect diagram of an embodiment of the present invention;

[0046] Figure 5 This is a depth information rendering of an image from an embodiment of the present invention;

[0047] Figure 6 This is a smoke-free image rendering of Embodiment 1 of the present invention;

[0048] Figure 7 This is a smoke-free image rendering of Embodiment 2 of the present invention. Detailed Implementation

[0049] The following is in conjunction with the appendix Figure 1 - Appendix Figure 7 The present invention will be described in further detail below.

[0050] Reference Figure 1 A method for removing smoke from endoscopic images involves first modeling the depth information of the endoscopic smoke image, obtaining model parameters to recover the depth information of the image, then calculating the transmittance and atmospheric light value of the endoscopic surgical scene based on the depth information of the endoscopic smoke image, and finally obtaining the smoke-free image to be recovered based on the atmospheric scattering model.

[0051] Based on an atmospheric scattering model, the depth information of smoke images is modeled, and model parameters are obtained to recover the image's depth information. Using the depth information of the smoke image, transmittance and atmospheric light values ​​can be calculated through the atmospheric scattering model, improving the estimation accuracy of transmittance and atmospheric light values, reducing color distortion after smoke removal in surgical scenes, and enabling more realistic image recovery, allowing more details to be seen. The final smoke-free image shows good smoke removal results, high efficiency, and short processing time, and solves the problem of color distortion that easily occurs when removing smoke.

[0052] A method for removing smoke from endoscopic images includes the following steps:

[0053] Step 1: Obtain the initial endoscopic image I(x);

[0054] Step 2: Convert the acquired initial image from RGB color space to HSV color space to obtain an HSV color space image, and calculate the depth information D;

[0055] Step 3: Perform joint bilateral filtering on the depth information D to obtain Dfilter;

[0056] Step 4: Calculate the transmittance t(x) based on the filtered depth information Dfilter;

[0057] Step 5: Calculate the atmospheric light value A based on the depth information D;

[0058] Step 6: Based on the transmittance t(x) and atmospheric light value A, obtain the smoke-free image to be recovered using the atmospheric scattering model.

[0059] In step 2, the formula for calculating the depth information D is:

[0060] D(x)=θ0+θ1V(x)+θ2S(x)+ε(x)

[0061] Where D is depth information, V is brightness, S is saturation; θ0, θ1, θ2 are linear coefficients, and ε is a random variable representing the random graph of the model.

[0062] After acquiring the initial image from the endoscope, the image is first converted from the RGB color space to the HSV color space. Further, in some implementations, the depth information of the smoke image is calculated based on the image brightness and saturation in the HSV color space. Specifically: in a smoke scene, there is a linear relationship between image depth, brightness, and saturation: D(x) = θ0 + θ1V(x) + θ2S(x) + ε(x), where D is the depth information, V is the brightness, and S is the saturation. θ0, θ1, and θ2 are linear coefficients, and ε is a random variable representing a random graph of the model, which allows for fast computation.

[0063] In step 3, the depth information D is subjected to joint bilateral filtering, and the specific formula is as follows:

[0064]

[0065] Where x is the coordinate of the current pixel, k is the normalization coefficient, Ω(x) is the window centered at x, q is the coordinate of the pixel within the window, f is the spatial domain filtering function, and g is the range filtering function, the calculation formula of which is as follows:

[0066]

[0067]

[0068] Where, σ d These are the spatial filtering coefficients, σ r These are the range filtering coefficients.

[0069] By performing joint bilateral filtering on the depth information D, the white block effect can be avoided.

[0070] In step 4, the formula for calculating the transmittance t(x) based on the filtered depth information Dfilter is as follows:

[0071] t(x)=exp(-beta*((Dfilter(x)+0.5) 2 -1))

[0072]

[0073] Where beta is the atmospheric scattering coefficient, ranging from 0 to 1, a is the minimum threshold value, and b is the maximum threshold value.

[0074] The transmittance t(x) is calculated to provide parameters for subsequent calculations to reconstruct smoke-free images.

[0075] The specific steps of step 5 are as follows: perform minimum value filtering on the depth information D, sort the processed image data according to the principle of descending, select the first 0.1% of pixels in the processed image data, and take the average value of the brightness of these pixels in the original image as the atmospheric light value A.

[0076] Wherein, atmospheric light value A = MAX(A, Amax), Amax is the maximum threshold value, and MAX is the maximum value function.

[0077] The problem of local block blurring in the restored smoke-free image was solved, ensuring the clarity of the restored smoke-free image.

[0078] The formula for obtaining the smoke-free image to be recovered from the atmospheric scattering model is as follows:

[0079]

[0080] J(x) is the smoke-free image.

[0081] Based on the atmospheric scattering model, the depth information of the smoke image is modeled, and the model parameters are obtained to recover the depth information of the image. The smoke removal effect is good, efficient and time-saving, and it solves the problem of image color distortion that is easy to occur when removing smoke.

[0082] An endoscopic image smoke removal device includes a memory and a processor. The memory and the processor are electrically connected in communication. The memory is pre-loaded with an endoscopic image smoke removal program designed for the endoscopic image smoke removal method. An initial endoscopic image is stored in the memory. The processor runs the endoscopic image smoke removal program to process the endoscopic image and generate a smoke-free image.

[0083] A display device can also be set up, which is electrically connected to the processor. The processor displays the smoke-free image generated after running the endoscope image desmoke removal program through the display device.

[0084] The images captured by the endoscope are stored in the memory. The processor runs an endoscope image desmearing program to process the endoscope images and generate smoke-free images. The generated smoke-free images are then displayed on a display device. The original endoscope images can also be displayed simultaneously for comparison.

[0085] Specific embodiment one of the present invention:

[0086] Step 1: Obtain the initial endoscopic image I(x), such as Figure 2 As shown.

[0087] Step 2: After acquiring the initial image I(x) from the endoscope, the image is first converted from the RGB color space to the HSV color space. Further, the depth information of the smoke image is calculated based on the image brightness and saturation in the HSV color space. Specifically: In a smoke scene, there is a linear relationship between image depth, brightness, and saturation: D(x) = θ0 + θ1V(x) + θ2S(x) + ε(x), where D is the depth information, V is the brightness, and S is the saturation. Figure 3 , Figure 4 and Figure 5 As shown.

[0088] θ0, θ1, and θ2 are linear coefficients, and ε is a random variable representing the random graph of the model. Here, θ0 = 0.4199, θ1 = 0.3869, θ2 = 0.5389, and ε = 0.

[0089] Step 3: Perform joint bilateral filtering on the depth information D obtained above to avoid white block effect.

[0090]

[0091] Where x is the coordinate of the current pixel, k is the normalization coefficient, k = 25, Ω(x) is the window centered at x, q is the coordinate of the pixel within the window, f is the spatial filtering function, and g is the range filtering function.

[0092]

[0093]

[0094] Where, σ d These are the spatial filtering coefficients, σ r These are the range filtering coefficients, σ d =5, σ r =10.

[0095] Step 4: Calculate the transmittance t(x) based on the filtered result from Step 3.

[0096] t(x)=exp(-beta*((Dfilter(x)+0.5) 2 -1))

[0097]

[0098] Where Dfilter(x) represents the filtered depth information D, beta is the atmospheric scattering coefficient, where beta = 0.5, a represents the minimum threshold value, and b represents the maximum threshold value, where a = 0.05 and b = 0.8.

[0099] Step 5: Calculate the atmospheric light value A based on the depth information D obtained in Step 2.

[0100] Minimum filtering is applied to the depth information D. The processed image data is sorted from largest to smallest, and the pixel with the maximum value is selected. The average brightness of the corresponding pixels in the original image is then used as the estimate of the global atmospheric light. This method can estimate the corresponding atmospheric light value A for endoscopic smoke images. To avoid large-area overexposure after image processing, a maximum threshold value Amax = 230 can be configured. The final atmospheric light value is selected as MAX(A, Amax), where MAX indicates the maximum of the two values. This method solves the problem of local block blurring in the recovered smoke-free image and ensures the clarity of the recovered smoke-free image.

[0101] Step 6: Restore a clear image without smoke.

[0102] Based on the transmittance t(x) and atmospheric light value A obtained from the above steps, the following can be derived from the atmospheric scattering model:

[0103]

[0104] J(x) is the smoke-free image to be recovered. The result is displayed on the display device as follows: Figure 6 As shown.

[0105] Specific embodiment two of the present invention:

[0106] Step 1: Obtain the initial endoscopic image I(x), such as Figure 2 As shown.

[0107] Step 2: After acquiring the initial image I(x) from the endoscope, the image is first converted from the RGB color space to the HSV color space. Further, the depth information of the smoke image is calculated based on the image brightness and saturation in the HSV color space. Specifically: In a smoke scene, there is a linear relationship between image depth, brightness, and saturation: D(x) = θ0 + θ1V(x) + θ2S(x) + ε(x), where D is the depth information, V is the brightness, and S is the saturation. Figure 3 , 4 As shown in Figure 5.

[0108] θ0, θ1, and θ2 are linear coefficients, and ε is a random variable representing the random graph of the model. Here, θ0 = 0.4199, θ1 = 0.3869, θ2 = 0.5389, and ε = 0.

[0109] Step 3: Perform joint bilateral filtering on the depth information D obtained above to avoid white block effect.

[0110]

[0111] Where x is the coordinate of the current pixel, k is the normalization coefficient, k = 25, Ω(x) is the window centered at x, q is the coordinate of the pixel within the window, f is the spatial filtering function, and g is the range filtering function.

[0112]

[0113]

[0114] Where, σ d These are the spatial filtering coefficients, σ r These are the range filtering coefficients, σ d =15, σ r =20.

[0115] Step 4: Calculate the transmittance t(x) based on the filtered result from Step 3.

[0116] t(x)=exp(-beta*((Dfilter(x)+0.5) 2 -1))

[0117]

[0118] Where Dfilter(x) represents the filtered depth information D, beta is the atmospheric scattering coefficient, where beta = 0.7, a represents the minimum threshold value, and b represents the maximum threshold value, where a = 0.05 and b = 0.8.

[0119] Step 5: Calculate the atmospheric light value A based on the depth information D obtained in Step 2.

[0120] Minimum filtering is applied to the depth information D. The processed image data is sorted from largest to smallest, and the pixel with the maximum value is selected. The average brightness of the corresponding pixels in the original image is then used as the estimate of the global atmospheric light. This method can estimate the corresponding atmospheric light value A for endoscopic smoke images. To avoid large-area overexposure after image processing, a maximum threshold value Amax = 230 can be configured. The final atmospheric light value is selected as MAX(A, Amax), where MAX indicates the maximum of the two values. This method solves the problem of local block blurring in the recovered smoke-free image and ensures the clarity of the recovered smoke-free image.

[0121] Step 6: Restore a clear image without smoke.

[0122] Based on the transmittance t(x) and atmospheric light value A obtained from the above steps, the following can be derived from the atmospheric scattering model:

[0123]

[0124] J(x) is the smoke-free image to be recovered. The result is displayed on a display device as follows: Figure 7 As shown.

[0125] A careful comparison of Example 1 and Example 2 shows that Example 2 has a better defogging effect than Example 1. See details below. Figure 6 and Figure 7 The color retention effect of regions A and B in this method is good in both Embodiment 1 and Embodiment 2. See details below. Figure 6 and Figure 7 Regions A and B.

[0126] It should be noted that, due to the regulation that color photographs cannot be used in patent drawings, this solution... Figure 6 and Figure 7 All images have been processed to grayscale, therefore the color retention effect cannot be visually demonstrated. Please take into account any resulting color effect discrepancies.

[0127] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for removing smoke from endoscopic images, characterized in that: First, the depth information of the endoscopic smoke image is modeled, and the model parameters are obtained to restore the depth information of the image. Then, the transmittance and atmospheric light value of the endoscopic surgery scene are calculated based on the depth information of the endoscopic smoke image. Finally, the smoke-free image to be restored is obtained based on the atmospheric scattering model. Step 1: Obtain the initial endoscopic image I(x); Step 2: Convert the acquired initial image from RGB color space to HSV color space to obtain an HSV color space image, and calculate the depth information D; Step 3: Perform joint bilateral filtering on the depth information D to obtain Dfilter; Step 4: Calculate the transmittance t(x) based on the filtered depth information Dfilter; Step 5: Calculate the atmospheric light value A based on the depth information D; Step 6: Based on the transmittance t(x) and the atmospheric light value A, obtain the smoke-free image to be recovered using the atmospheric scattering model; In step 4, the formula for calculating the transmittance t(x) based on the filtered depth information Dfilter is as follows: t(x)=exp(-beta*((Dfilter(x)+0.5) 2 -1)) Where beta is the atmospheric scattering coefficient, ranging from 0 to 1, a is the minimum threshold value, and b is the maximum threshold value; The specific steps of step 5 are as follows: perform minimum value filtering on the depth information D, sort the processed image data according to the principle of descending, select the first 0.1% of pixels in the processed image data, and take the average value of the brightness of the corresponding pixels in the original image as the atmospheric light value A. Where, atmospheric light value A = MAX(A, Amax), Amax is the maximum threshold value, and MAX is the maximum value function; In step 2, the formula for calculating the depth information D is: D(x)=θ0+θ1V(x)+θ2S(x)+ε(x) Where D is depth information, V is brightness, S is saturation; θ0, θ1, θ2 are linear coefficients, and ε is a random variable representing the random graph of the model; The formula for obtaining the smoke-free image to be recovered from the atmospheric scattering model is as follows: J(x) is the smoke-free image.

2. The method for removing smoke from endoscopic images according to claim 1, characterized in that: In step 3, the depth information D is subjected to joint bilateral filtering, and the specific formula is as follows: Where x is the coordinate of the current pixel, k is the normalization coefficient, Ω(x) is the window centered at x, q is the coordinate of the pixel within the window, f is the spatial domain filtering function, and g is the range filtering function, the calculation formula of which is as follows: Where, σ d These are the spatial filtering coefficients, σ r These are the range filtering coefficients.

3. An endoscopic image smoke removal device, comprising a memory and a processor, wherein the memory and the processor are communicatively electrically connected, characterized in that: The memory is pre-loaded with an endoscope image desmoke removal program designed according to any one of claims 1-2. The initial endoscope image is stored in the memory, and the processor runs the endoscope image desmoke removal program to process the endoscope image and generate a smoke-free image.

4. The endoscopic image smoke removal device according to claim 3, characterized in that: It also includes a display device, which is electrically connected to the processor, and the processor displays a smoke-free image generated after running the endoscopic image desmoke removal program through the display device.

Images