A highlight detection and inpainting method for endoscopic images

Abstract

The application belongs to the technical field of image processing, and specifically discloses a highlight detection and repair method for endoscope images, which comprises the following steps: S1, obtaining a mirror reflection image MSF according to an original image P; S2, performing pixel classification on the mirror reflection image MSF to obtain a highlight area Omega; S3, calculating a filling color C of the highlight area and repairing the highlight area Omega by using the filling color C; wherein, the step S3 further comprises calculating a weight value w of the filling color and generating a weight map. The application can more accurately capture the highlight area by calculating the hue threshold value and the saturation threshold value, can quickly process the highlight removal operation of the endoscope image, and meets the requirement of real-time processing. The method has high processing speed and good image quality, and provides an efficient image processing scheme for endoscope examination.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method for highlight detection and repair of endoscopic images. Background Technology

[0002] Endoscopic images and videos often exhibit highlighting issues when displaying the surfaces of human organs (such as the gastrointestinal mucosa), which are bright areas created by reflections from a light source. These highlight features not only negatively impact image quality but can also lead to errors in many visual analysis algorithms, such as some gradient-based computer vision and image analysis algorithms, and image texture-based methods.

[0003] Traditional methods for de-highlighting endoscopic images mainly include thresholding, edge detection, and image enhancement. However, these methods have some limitations. For example, simple thresholding and edge detection methods may not accurately identify highlight areas, while image enhancement methods may lead to loss of detail and image artifacts.

[0004] In summary, in order to improve image quality and provide more reliable input for subsequent image analysis and processing algorithms, it is necessary to design a new method for endoscopic detection and highlight removal. Summary of the Invention

[0005] To address the problems in the prior art, this invention provides a method for detecting and removing highlights from endoscopic images, achieving rapid endoscopic highlight removal processing, meeting real-time requirements, and maintaining good image quality.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for detecting and repairing highlights in endoscopic images, comprising the following steps:

[0008] S1. Obtain the specular reflection image MSF based on the original image P;

[0009] S2. Perform pixel classification on the specular reflection image MSF to obtain the highlight region Ω;

[0010] S3. Calculate the fill color C of the highlight area, and use the fill color C to repair the highlight area Ω;

[0011] Step S3 further includes calculating the weight value w of the fill color and generating a weight map. The weight value w is calculated using the following formula:

[0012] w=(1-s)×e -αdi ,

[0013] s is sparsity, d i$d_i$ is the minimum distance from the $i$-th pixel in the highlight area to the highlight contour, and $\alpha$ is a parameter for controlling the attenuation of the distance weight.

[0014] In some embodiments of the present invention, step S1 includes the following steps:

[0015] S11. Obtain the specular reflection-free image SF using the following formula:

[0016] I SF (p) = I r，g，b (p) - min(I r (p), I g (p), I b (p)),

[0017] where, I SF (p) is the color of the specular reflection-free image SF at the current pixel, I r，g，b (p) is the color of the current pixel in the original image, I r (p), I g (p) and I b (p) are the component colors of the R channel, G channel, and B channel in the original image respectively;

[0018] S12. Calculate the specular reflection image MSF according to the specular reflection-free image SF.

[0019] In some embodiments of the present invention, the calculation formula of the specular reflection image MSF is:

[0020] I MSF (p) = I SF (p) + avg,

[0021] where, I MSF (p) is the color of the specular reflection image MSF at the current pixel, and avg is the color scalar value of each pixel in the original image.

[0022] In some embodiments of the present invention, the process of pixel classification in step S2 includes:

[0023] S21. Preprocess the specular reflection image MSF to obtain the true specular reflection image r-MSF;

[0024] S22. Judge the relationship between the color I r-MSF (p) of each pixel in the true specular reflection image and the adaptive threshold T;

[0025] S23. When I r-MSF (p) < T, the pixel belongs to the pixel set that only contains diffuse reflection; otherwise, the pixel belongs to the pixel set that contains both diffuse reflection and specular reflection components.

[0026] In some embodiments of the present invention, the formula for calculating the adaptive threshold T in step S22 is: T = a × Tv + (1-a) × Ts.

[0027] Where Tv is the hue threshold of the real specular reflection image r-MSF in the HSV model, Ts is the saturation threshold of the real specular reflection image r-MSF in the HSV model, and a is the weighting coefficient.

[0028] In some embodiments of the present invention, the hue threshold Tv is calculated using the formula Tv = μ v +kv×ε v , where μ v ε is the mean hue value. v For μ v The standard deviation of kv is the parameter for adjusting the hue threshold sensitivity.

[0029] In some embodiments of the present invention, the saturation threshold Ts is calculated using the formula Ts = μ s +ks×ε s , where μ s ε is the mean hue value. s For μ s The standard deviation of , ks is a parameter for adjusting the sensitivity of the saturation threshold.

[0030] In some embodiments of the present invention, the preprocessing in step S21 uses the following formula:

[0031] I r-MSF =Ip-I MSF ,

[0032] Where Ip represents the color of each pixel in the original image, I MSF The color of each pixel in the specular reflection image MSF is denoted by .

[0033] In some embodiments of the present invention, the formula for calculating the sparsity s in step S3 is as follows:

[0034]

[0035] Where n is the total number of specular reflection pixels in the highlight region Ω, and x i This represents the color vector of a single pixel within the highlight region Ω.

[0036] In some embodiments of the present invention, the minimum distance d i The calculation formula is:

[0037]

[0038] Where, x i and yi Let represent the coordinates of the i-th pixel in the highlight region Ω. and This represents the coordinates of a point on the outline of the highlight area.

[0039] The technical solution of the present invention has the following technical effects compared with the prior art:

[0040] This invention obtains a threshold T by weighting hue and saturation thresholds, which can more accurately capture highlight areas and quickly process highlight removal in endoscopic images, meeting real-time processing requirements. This method offers high processing speed and good image quality, providing an efficient image processing solution for endoscopic examinations. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the detection and repair method.

[0043] Figure 2 This is a flowchart illustrating the highlight region detection method. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0046] Reference Figure 1 As shown, a method for highlight detection and repair in endoscopic images includes the following steps:

[0047] S1. Obtain the specular reflection image MSF based on the original image P;

[0048] Specifically, video frames are captured via video stream data from an endoscope. The raw images are then preprocessed, and specular reflection is characterized using a dichroic reflectance model. This method assumes that the color information in the image consists of a linear combination of diffuse and specular components. The diffuse component carries the color of the object itself, while the specular component shows the reflection of the object's surface. Based on this model, the information measured by the endoscope can be expressed as:

[0049] L(λ,θ i θ r g) = L d (λ)·ρ d (λ)·f d (θ i θ r ,g)+L s (λ)·ρ s (λ)·f s (θ i θ r g)

[0050] Where, L(λ, θ) i θ r , g) represents the color information measured by the endoscope, λ represents the wavelength, θi represents the incident angle, θr represents the reflection angle, and g represents the light source position parameter. L d ρ(λ) and Ls(λ) represent the spectral distributions of the diffuse and specular reflection components, respectively. d (λ) and ρ s (λ) represents the reflectance of the diffuse and specular components, f d (θ i θ r ,g) and f s (θ i θ r , g) represents the distribution function of diffuse and specular reflection components. This formula can be further reduced, meaning the color of pixel p can be seen as a linear combination of specular reflection Ls and diffuse reflection Ld. The color of specular reflection is consistent with the color of the light source, while the color of diffuse reflection represents the inherent color of the object, i.e., its body color.

[0051] I=αL d +βL s

[0052] In this embodiment, during the specular detection step, a non-specular image (SF) and a specular image (MSF) are obtained from the original image:

[0053] S11. Obtain the non-reflective image SF using the following formula:

[0054] I SF (p)=I r，g，b (p)-min(I r (p), I g (p), I b (p)),

[0055] Among them, I SF (p) represents the color of the non-mirror image SF at the current pixel, I r，g，b (p) represents the color of the current pixel in the original image, I r (p), I g (p) and I b (p) represent the component colors of the R, G, and B channels in the original image, respectively.

[0056] In this calculation, a minimum channel value is subtracted to eliminate the effect of diffuse reflection.

[0057] S12. Calculate the specular reflection image MSF based on the non-specular reflection image SF.

[0058] In some embodiments of the present invention, the formula for calculating the specular reflection image MSF is as follows:

[0059] I MSF (p)=I SF (p)+avg,

[0060] Among them, I MSF (p) represents the color of the specular reflection image MSF at the current pixel, and avg represents the color scalar value of each pixel in the original image.

[0061] In this step, a color scalar mean is added to further emphasize the specular component.

[0062] The color scalar value avg is calculated using the following formula:

[0063]

[0064] Where pixels is the number of pixels.

[0065] The above step S1 can effectively separate the pixel set that combines diffuse reflection and specular reflection, reduce the processing range of subsequent specular reflection separation, and greatly reduce the processing time.

[0066] S2. Perform pixel classification on the specular reflection image MSF to obtain the highlight region Ω;

[0067] Pixel-level classification based on the difference between the specular reflection image (MSF) and the original image allows for the determination of the body color of each pixel. All pixels are divided into two categories: the first category is a set of pixels containing only diffuse reflection, and the second category is a set of pixels containing both diffuse and specular reflection components.

[0068] The pixel classification process in step S2 includes:

[0069] S21. Preprocess the specular reflection image MSF to obtain the real specular reflection image r-MSF;

[0070] The formula used for preprocessing is as follows:

[0071] I r-MSF =Ip-I MSF ,

[0072] Where Ip represents the color of each pixel in the original image, I MSF The color of each pixel in the specular reflection image MSF is denoted by .

[0073] S22. Determine the color I of each pixel in the real specular reflection image. r-MSF (p) and its relationship with the adaptive threshold T; refer to Figure 2 As shown,

[0074] For the adaptive threshold T, the image is first converted to the HSV color model, and the hue and saturation of each pixel are calculated.

[0075] The formula for calculating the adaptive threshold T in step S22 is: T = a × Tv + (1-a) × Ts.

[0076] Where Tv is the hue threshold of the real specular reflection image r-MSF in the HSV model, Ts is the saturation threshold of the real specular reflection image r-MSF in the HSV model, and a is the weighting coefficient.

[0077] The formula for calculating the hue threshold Tv is Tv = μ v +kv×ε v , where μ v ε is the mean hue value. v For μ v The standard deviation of kv is the parameter for adjusting the hue threshold sensitivity.

[0078] The saturation threshold Ts is calculated using the formula Ts = μ s +ks×ε s , where μ s ε is the mean hue value. s For μ s The standard deviation of , ks is a parameter for adjusting the sensitivity of the saturation threshold.

[0079] S23. When I r-MSF (p) < T, the pixel belongs to the pixel set containing only diffuse reflection; otherwise, the pixel belongs to the pixel set containing both diffuse reflection and specular reflection components.

[0080] As above, specular detection and classification can be achieved according to the adaptive threshold T. Among them, the calculation process of the adaptive threshold T involves two parameters, hue and saturation. By using the weighted definition of these two parameters for the adaptive threshold T, the characteristics of the highlight area can be captured more precisely.

[0081] Specifically, hue and saturation reflect the characteristics of colors and are not affected by changes in illumination. Compared with using the average brightness as the threshold in the prior art, the adaptive threshold T calculated in this embodiment can better adapt to images under different illumination conditions.

[0082] In addition, since the highlight area usually has a high brightness value, but brightness cannot well distinguish highlights from other bright areas. On the contrary, hue and saturation provide a more detailed description of color attributes, so highlights can be better distinguished.

[0083] S3. Calculate the filling color C of the highlight area and use the filling color C to repair the highlight area Ω;

[0084] By filling the detected highlight area with the pixel color after the contour, then adjusting the contribution degree of the filling color through weights, and performing median filtering on this basis, the brightness transition caused by the highlight area can be effectively reduced, and a more natural repair effect can be provided.

[0085] The filling color C of the highlight area Ω is determined by the following formula:

[0086]

[0087] where p represents the pixel on the highlight contour, n is the total number of pixels on the highlight contour, represents the highlight contour.

[0088] In order to retain the repair details and layering, a weight map is applied to the filled image. The color weight takes into account the sparsity of the highlight area and the distance from the filling position to the highlight contour. The generation process includes the following steps:

[0089] 1. Sparsity s calculation: Calculate the sparsity of the filled highlight area. This calculation includes representing the highlight area as a sparse coefficient matrix and calculating the sparsity index based on the sparse representation theory to evaluate the sparsity of the highlight area. Sparsity is used to measure the uniformity of the vector. It is calculated through the following formula:

[0090]

[0091] Where n is the total number of specular reflection pixels in the highlight region Ω, and x i This represents the color vector of a single pixel within the highlight region Ω. Higher sparsity indicates a more scattered vector distribution, while lower sparsity indicates a more uniform pixel distribution and a smoother corresponding object surface.

[0092] 2. Distance Calculation: For each filled pixel, calculate its minimum distance d to the highlight contour. i :

[0093]

[0094] Where, x i and y i Let represent the coordinates of the i-th pixel in the highlight region Ω. and This represents the coordinates of a point on the outline of the highlight area.

[0095] 3. Weight Generation: Based on sparsity and distance, a weighted graph is generated. This process includes mapping sparsity and distance to a range of weight values ​​and performing normalization. Specifically, the weight values ​​can be calculated using the following formula:

[0096] w=(1-s)×e -αdi ,

[0097] s is sparsity, d i denoted as s, it represents the minimum distance from the i-th pixel within the highlight region to the highlight outline, and α is a parameter controlling the decay of distance weights. A larger sparsity s indicates a sparser highlight region, corresponding to a smaller color weight. Similarly, a larger distance d indicates that the fill position is farther from the highlight outline, corresponding to a smaller weight.

[0098] To avoid artifacts and to make the filled area smoother, a Gaussian kernel (σ=8) is used to filter the filled image, resulting in a smooth image with highlights removed, similar to median filtering in segmentation algorithms.

[0099] By adding non-linear attenuation to the repair area, the quality of highlight repair can be effectively improved, and the obvious transition between the repair area and the surrounding area can be reduced, resulting in a more natural repair result.

[0100] The technical solution of the present invention has the following technical effects compared with the prior art:

[0101] 1. Improved Image Quality: By accurately removing highlight interference in endoscopic images, this method can significantly improve image visibility and quality. Eliminating interference from highlight areas allows medical professionals to more clearly observe and diagnose subtle changes on the surface of human organs.

[0102] 2. Improve the accuracy of subsequent image analysis and processing algorithms: Highlight features in endoscopic images interfere with gradient-based computer vision and image analysis algorithms, as well as image texture-based methods. The highlight removal method of this invention can accurately detect and process highlight regions, thereby reducing the error impact of highlights on subsequent image analysis and processing algorithms and improving the accuracy and reliability of the algorithms.

[0103] 3. Real-time processing and high efficiency: The method of this invention, through algorithm optimization and efficient implementation, can quickly process the highlight removal operation of endoscopic images, meeting the requirements of real-time processing. This method has high processing speed and good image quality, providing an efficient image processing solution for endoscopic examinations.

[0104] 4. Wide Applicability: This method is not only applicable to highlight removal in endoscopic images in the medical field, but can also be extended to other fields, such as industrial inspection and computational photography, to solve similar highlight problems. Therefore, the beneficial effects of this invention have wide applicability and potential in multiple application areas.

[0105] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0106] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting and repairing highlights in endoscopic images, characterized in that, Includes the following steps: S1. Obtain the specular reflection image MSF based on the original image P; S2. Perform pixel classification on the specular reflection image MSF to obtain the highlight region Ω; S3. Calculate the fill color C of the highlight area, and use the fill color C to repair the highlight area Ω; Step S3 further includes calculating the weight value w of the fill color and generating a weight map. The weight value w is calculated using the following formula: w=(1-s) ×e -αdi , s is sparsity, d i α is the minimum distance from the i-th pixel within the highlight region to the highlight contour, and α is a parameter that controls the distance weight decay. The formula for calculating the sparsity s is: ； Where n is the total number of specular reflection pixels in the highlight region Ω, and x i This represents the color vector of a pixel in the highlight region Ω; The minimum distance d i The calculation formula is: ， Where, x i and y i Let represent the coordinates of the i-th pixel in the highlight region Ω, and let represent the coordinates of a point on the outline of the highlight region.

2. The method for highlight detection and repair of endoscopic images according to claim 1, characterized in that, Step S1 includes the following steps: S11. Obtain the non-reflective image SF using the following formula: I SF (p)= I r，g，b (p)-min(I r (p),I g (p),I b (p)), Among them, I SF (p) represents the color of the non-mirror image SF at the current pixel, I r，g，b (p) represents the color of the current pixel in the original image, I r (p), I g (p) and I b (p) represent the component colors of the R, G, and B channels in the original image, respectively; S12. Calculate the specular reflection image MSF based on the non-specular reflection image SF.

3. The method for highlight detection and repair of endoscopic images according to claim 1, characterized in that, The formula for calculating the specular reflection image MSF is as follows: I MSF (p) =I SF (p)+avg, Among them, I MSF (p) represents the color of the specular reflection image MSF at the current pixel, and avg represents the color scalar value of each pixel in the original image.

4. The method for detecting and repairing highlights in endoscopic images according to claim 1, characterized in that, The pixel classification process in step S2 includes: S21. Preprocess the specular reflection image MSF to obtain the real specular reflection image r-MSF; S22. Determine the color I of each pixel in the real specular reflection image. r-MSF The relationship between (p) and the adaptive threshold T; S23. When I r-MSF (p) < T, the pixel belongs to the pixel set that only contains diffuse reflection; otherwise, the pixel belongs to the pixel set that contains both diffuse reflection and specular reflection components.

5. The method for highlight detection and repair of endoscopic images according to claim 4, characterized in that, The formula for calculating the adaptive threshold T in step S22 is: T = a × Tv + (1 - a) × Ts. Where Tv is the hue threshold of the real specular reflection image r-MSF in the HSV model, Ts is the saturation threshold of the real specular reflection image r-MSF in the HSV model, and a is the weighting coefficient.

6. The method for highlight detection and repair of endoscopic images according to claim 5, characterized in that, The formula for calculating the hue threshold Tv is Tv = μ v +kv×ε v , where μ v ε is the mean hue value. v For μ v The standard deviation of kv is the parameter for adjusting the hue threshold sensitivity.

7. The method for detecting and repairing highlights in endoscopic images according to claim 5, characterized in that, The saturation threshold Ts is calculated using the formula Ts = μ s +ks×ε s , where μ s ε is the mean hue value. s For μ s The standard deviation of , ks is a parameter for adjusting the sensitivity of the saturation threshold.

8. The method for highlight detection and repair of endoscopic images according to claim 4, characterized in that, The formula used for preprocessing in step S21 is as follows: I r-MSF =Ip- I MSF , Where Ip represents the color of each pixel in the original image, I MSF The color of each pixel in the specular reflection image MSF is denoted by .

Images