Endoscopic image highlight removal method and device

Abstract

The present invention discloses a method and device for removing highlights from an endoscopic image, and belongs to the field of image processing technology. The method for removing highlights from an endoscopic image of the present invention comprises the following steps: S1, obtaining an original endoscopic image; S2, identifying the highlight area of ​​the original endoscopic image by adaptive threshold segmentation to obtain a highlight mask image; S3, obtaining the contour length and area of ​​each highlight point in the highlight mask image; S4, repairing each corresponding highlight point in the highlight mask image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image, to obtain an endoscopic image after removing the highlights. The present invention can conveniently and efficiently remove the highlight area of ​​an endoscopic image, and can ensure the quality of the endoscopic image after removing the highlights.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically, to a method and device for removing highlights from an endoscope image. Background Art

[0002] In minimally invasive surgery, highlight phenomenon is mainly caused by the smooth characteristics of mucosa, liquid, organ surface and light source at a specific angle. This phenomenon not only brings challenges to lesion identification in the algorithm, but also easily causes visual fatigue of doctors, which in turn has an adverse effect on surgical efficiency. Therefore, it is necessary to remove highlight from endoscopic images.

[0003] Common methods for highlight removal include filtering-based methods, machine learning-based methods, two-color reflectance model-based methods, RPCA (Robust Principal Component Analysis)-based methods, and texture-based methods.

[0004] Based on the filtering method, the image is filtered to obtain information around the center point of the image, which can effectively remove highlights. This method is simple and direct, but it may blur the details of the image, causing image distortion or quality degradation.

[0005] The neural network-based approach can restore the details and content of the highlight area more accurately by learning the features and structure of the image. However, this approach requires a large amount of labeled data for training, and the training and parameter adjustment of the network model are relatively complex, which can easily lead to unstable training or model crash.

[0006] The two-color reflection model-based approach can estimate and remove highlights more accurately by simulating the reflection and refraction process of light on the tissue surface. However, this approach is prone to unstable highlight removal results due to changes in lighting conditions.

[0007] Based on the RPCA method, the image is decomposed into a low-rank matrix and a sparse matrix. The low-rank matrix represents the main information and structure of the image, and the sparse matrix represents the highlight area of ​​the image. This method extracts the highlight part of the image by minimizing the sparse matrix, which can effectively remove the highlight. However, when there is a large area of ​​highlight in the image, the iteration time of this method is long and the effect is limited.

[0008] Texture-based methods usually use local texture features of images for matching and filling, thereby repairing damaged or missing areas. The Criminisi algorithm is one of the texture-based methods. It estimates and fills missing pixels by analyzing the known pixel information around it, thereby repairing and reconstructing the image. Because this method can effectively preserve the texture and structural information of the image, the repaired image looks more natural and coherent. However, it also has problems such as a large global search range, low algorithm efficiency, and mismatching. Summary of the invention

[0009] 1. Technical problem to be solved by the invention

[0010] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and device for removing highlights from endoscopic images, which can conveniently and efficiently remove the highlight area of ​​endoscopic images and effectively maintain the integrity of image texture and structural information, so that the processed image has a better global visual effect and ensures the quality of the endoscopic image after highlight removal.

[0011] 2. Technical solution

[0012] In order to achieve the above object, the technical solution provided by the present invention is:

[0013] The method for removing highlights from endoscopic images of the present invention comprises the following steps: S1, obtaining an original endoscopic image; S2, identifying highlight areas of the original endoscopic image by adaptive threshold segmentation to obtain a highlight mask image; S3, obtaining the contour length and area of ​​each highlight point in the highlight mask image; S4, repairing each corresponding highlight point in the original endoscopic image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image to obtain an endoscopic image after removing highlights.

[0014] As a further improvement of the present invention, step S2 specifically includes: S21, converting the original endoscopic image from the RGB color space to the HSV color space; S22, in the HSV color space, dynamically screening highlight pixels through an adaptive threshold function, and generating an initial highlight mask using the screened highlight pixels; S23, optimizing the initial highlight mask using a morphological dilation operation to obtain the highlight mask image.

[0015] Step S4 specifically includes: S41, reading each highlight point in order from small to large area; S42, after reading a highlight point, determining the adaptive search range according to the contour length of the highlight point; S43, calculating the repair priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority; S44, using the improved matching criterion to find the best matching block within the adaptive search range, and copying the pixels of the best matching block to fill the target block with the highest priority; S45, updating the confidence and the edge of the highlight point, and judging whether there are still unrepaired pixels on the edge of the highlight point, if so, returning to step S43, if not, executing step S46; S46, judging whether all the highlight points are repaired, if not, returning to step S41 to read the next highlight point, if so, outputting the endoscopic image after removing the highlights.

[0016] The adaptive search range is determined according to the contour length of the highlight point, specifically including: obtaining the coordinates of four adjacent pixel points around the highlight point; calculating the width and height of the highlight point, and determining the expansion coefficient according to the contour length of the highlight point; obtaining the upper boundary, lower boundary, left boundary and right boundary of the adaptive search range according to the coordinates of the four adjacent pixel points around the highlight point, the width and height of the highlight point and the expansion coefficient.

[0017] The improved priority function is:

[0018]

[0019] Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, represents the edge of the target block, C(q) and D(q) represent the confidence and data items respectively.

[0020] The improved matching criterion is expressed as follows:

[0021]

[0022] in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

[0023] The endoscopic image highlight removal device of the present invention comprises: a first acquisition module, which is used to acquire the original endoscopic image; an identification module, which is used to identify the highlight area of ​​the original endoscopic image through adaptive threshold segmentation to obtain a highlight mask image; a second acquisition module, which is used to acquire the contour length and area of ​​each highlight point in the highlight mask image; and a repair module, which is used to repair each corresponding highlight point in the original endoscopic image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image to obtain the endoscopic image after the highlight is removed.

[0024] As a further improvement of the present invention, the repair module specifically performs the following steps: S41, reading each highlight point in order from small to large area; S42, after reading a highlight point, determining the adaptive search range according to the contour length of the highlight point; S43, calculating the repair priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority; S44, using the improved matching criterion to find the best matching block within the adaptive search range, and copying the pixels of the best matching block to the target block with the highest priority for filling; S45, updating the confidence and the edge of the highlight point, and determining whether there are any unrepaired pixels on the edge of the highlight point, if so, returning to step S43, if not, executing step S46; S46, determining whether all the highlight points are repaired, if not, returning to step S41 to read the next highlight point, if so, outputting the endoscopic image after removing the highlights.

[0025] The improved priority function is:

[0026]

[0027] Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, represents the edge of the target block, C(q) and D(q) represent the confidence and data items respectively.

[0028] The improved matching criterion is expressed as follows:

[0029]

[0030] in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

[0031] 3. Beneficial effects

[0032] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0033] The present invention identifies the highlight area of ​​the original endoscopic image through adaptive threshold segmentation to obtain a highlight mask image, and repairs each highlight point in the original endoscopic image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image. The highlight area of ​​the endoscopic image can be removed conveniently and efficiently, and the integrity of the image texture and structural information can be effectively maintained, so that the processed image has a more outstanding global visual effect, and the quality of the endoscopic image after highlight removal is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a method for removing highlights from an endoscope image according to an embodiment of the present invention;

[0035] FIG2( a ) is an original endoscopic image according to a specific embodiment of the present invention;

[0036] FIG2( b ) is a highlight mask obtained according to FIG2( a );

[0037] Figure 2(c) is the highlight mask after morphological dilation of Figure 2(b);

[0038] Figure 3 It is a flowchart of an implementation of an improved Criminisi algorithm according to a specific embodiment of the present invention;

[0039] Figure 4 It is a block diagram of a device for removing highlight from an endoscope image according to an embodiment of the present invention;

[0040] FIG5( a ) is a set of original endoscopic images used in an experiment of a specific embodiment of the present invention;

[0041] FIG5( b ) is a highlight mask after morphological expansion obtained according to FIG5( a );

[0042] FIG6( a ) is a result image after highlight removal of FIG5( a ) based on filtering;

[0043] Figure 6(b) is the result image after highlight removal of Figure 5(a) based on the RPCA method;

[0044] FIG6(c) is the result image after highlight removal of FIG5(a) based on thermal diffusion;

[0045] Figure 6(d) is the result image after highlight removal of Figure 5(a) based on the traditional Criminisi algorithm;

[0046] Figure 6(e) is the result image after highlight removal of Figure 5(a) using the Criminisi algorithm based on texture synthesis;

[0047] FIG. 6( f ) is a result image after highlight removal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.

[0049] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical content disclosed by the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and so on quoted in this specification are only for the convenience of description, and are not used to limit the scope of implementation. The change or adjustment of their relative relationship should also be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0050] The present invention will be further described below in conjunction with the embodiments.

[0051] like Figure 1 As shown, the method for removing highlight from an endoscope image according to an embodiment of the present invention comprises the following steps S1 to S4:

[0052] S1, obtain the original endoscopic image.

[0053] In one embodiment of the present invention, the original endoscopic image may be acquired by the endoscope in real time during the operation.

[0054] S2, identifying the highlight area of ​​the original endoscopic image through adaptive threshold segmentation to obtain a highlight mask image.

[0055] In one embodiment of the present invention, step S2 specifically includes steps S21 to S23:

[0056] S21, converting the original endoscope image from the RGB color space to the HSV color space so as to more accurately identify the highlight area.

[0057] Taking an 8-bit color image as an example, for any coordinate point in the image, the value range of each channel pixel in the RGB and HSV color spaces is as follows:

[0058]

[0059] To convert an image from RGB color space to HSV color space, first, the R, G, and B values ​​of each channel need to be normalized. Then, the values ​​of H, S, and V are calculated using the following formula, and the values ​​of each component are ensured to be within their range:

[0060] V = max(R, G, B)

[0061]

[0062] S22, in the HSV color space, dynamically filter highlight pixels through an adaptive threshold function, and generate an initial highlight mask using the filtered highlight pixels.

[0063] The embodiment of the present invention combines the advantages of highlight detection based on color distribution and highlight detection based on adaptive threshold, uses adaptive threshold segmentation to perform highlight detection, and sets the judgment criteria of adaptive threshold segmentation according to the distribution characteristics of highlight pixels in the HSV color space.

[0064] The embodiment of the present invention sets an adaptive threshold function T according to the mean and standard deviation. s and T v , and use them to filter highlight pixels, thereby dividing the image pixels into highlight and non-highlight categories. s and T v is defined as follows:

[0065] T s =max{T sl ,Mean(I s )+Std(I s ) * α}

[0066] T v =min{T vl ,Mean(I v )+Std(U v ) * β}

[0067] Among them, I s and I v Represents the saturation channel value and the brightness channel value respectively, T sl and T vl Represents a predefined loose empirical threshold. α and β are two weight parameters, Mean and Std represent the mean and variance respectively, and are defined as follows:

[0068]

[0069] Among them, M and N are the upper limits of the number of horizontal and vertical pixels respectively.

[0070] The adaptive threshold function of the embodiment of the present invention can dynamically adjust the threshold according to the local pixel features of the image, and select a suitable threshold between the adaptive threshold and the loose threshold to screen highlight pixels. This can avoid using an inappropriate adaptive threshold in the highlight detection process, and also help improve the efficiency and accuracy of the algorithm.

[0071] In the HSV color space, highlight pixels are mainly distributed on the leftmost side of the saturation channel and the rightmost side of the brightness channel. Therefore, the embodiment of the present invention determines pixels whose saturation channel values ​​are less than or equal to and whose brightness channel values ​​are greater than or equal to as highlight pixels. By using this adaptive threshold segmentation method for highlight detection, the present invention successfully extracts highlight pixels from the original endoscopic image shown in FIG2(a), and obtains a highlight mask as shown in FIG2(b).

[0072] S23, optimizing the initial highlight mask using a morphological dilation operation to obtain a highlight mask image.

[0073] The morphological dilation operation includes filling holes and smoothing edges to ensure the accuracy and completeness of highlight detection. Lens movement and organ peristalsis may cause a halo effect, resulting in incoherent changes in color and brightness between the repaired highlight area and the surrounding area, and an obvious non-smooth transition. Therefore, an embodiment of the present invention uses a disk structure element with a radius of 3 to perform a morphological dilation operation on the mask to expand the highlight coverage of the mask. The highlight mask after morphological dilation is shown in Figure 2(c). The morphological dilation operation better covers the edges of the highlight area, improves the accuracy and stability of the highlight detection algorithm, and provides a more reliable basis for subsequent highlight removal.

[0074] S3, obtaining the contour length and area of ​​each highlight point in the highlight mask image.

[0075] The area of ​​the highlight point is the area of ​​the region within the outline of the highlight point. In a specific embodiment of the present invention, the units of length and area can both be expressed by the number of pixels.

[0076] S4, according to the contour length and area of ​​each highlight point in the highlight mask image, each corresponding highlight point in the original endoscopic image is repaired one by one based on the improved Criminisi algorithm to obtain an endoscopic image after removing the highlights.

[0077] In one embodiment of the present invention, referring to Figure 3 , step S4 specifically includes steps S41 to S46:

[0078] S41, reading each highlight point in order from small to large area.

[0079] By improving the highlight removal order, small highlights are repaired first, and then large highlights are repaired, ensuring that the repair order is reasonable.

[0080] The strategy adopted by the embodiment of the present invention is to repair the highlight points one by one in the order of highlight area from small to large, first repair the highlight point with the smallest highlight area, and then repair other highlight points in sequence until all highlight points are repaired. The strategy of repairing in sequence based on highlight area proposed in the embodiment of the present invention only needs to calculate the priority value of the edge pixel points of the current highlight point. This improvement can not only maintain the repair effect of the algorithm, but also greatly reduce the number of calculations and computational overhead of the algorithm. In addition, by performing highlight repair in order from small to large, it is possible to more reasonably find the area closest to the target block, which helps to improve the rationality of the algorithm repair sequence.

[0081] S42, after reading a highlight point, determining an adaptive search range according to the contour length of the highlight point.

[0082] The traditional Criminisi algorithm usually searches the entire image to find the best matching block globally. This method can ensure good global visual effects in natural image restoration. Because natural images are highly random, the image content and structure distribution are relatively random. However, endoscopic images are mainly composed of human tissues with high local characteristics. When using the traditional Criminisi algorithm to repair the highlights on a certain tissue in an endoscopic image, the global search method is not only time-consuming, but also may mistakenly match the texture of other tissues to the target tissue, resulting in unreasonable matching results.

[0083] In order to solve the problems of high computational cost, low efficiency and mismatch caused by the global search of the traditional Criminisi algorithm, the embodiment of the present invention proposes an adaptive search strategy based on the contour of the highlight point. First, the contour set of the highlight point is obtained according to the mask image obtained by highlight detection, and then the range of the adaptive search is determined according to the contour length of the highlight point. This strategy optimizes the search range of the optimal matching block according to the contour length of the highlight point, which can not only reduce the computational cost of the algorithm and the probability of mismatch, but also improve the repair efficiency and repair quality of the algorithm.

[0084] Specifically, the adaptive search strategy based on the highlight point outline is as follows: First, the coordinates of the four adjacent pixels around the highlight point can be obtained. The coordinates of the four pixels above, below, left and right are respectively expressed as u(x up ,y up )、d(x down ,y down )、l(x left ,y left )、r(x right ,yright )express.

[0085] Then, the width w and height h of the highlight point are calculated, and the contour length l of the highlight point is used to calculate the width w and height h of the highlight point. c Determine the expansion factor n:

[0086]

[0087] Wherein, a is a set value, which is determined by the resolution of the original endoscopic image, and the values ​​of n1 and n2 depend on the complexity of the texture structure of the original endoscopic image.

[0088] Finally, the upper, lower, left, and right boundaries of the adaptive search range are obtained based on the coordinates of the four adjacent pixels around the highlight point, the width and height of the highlight point, and the expansion coefficient:

[0089] u s =y up -nh

[0090] d s =y down +nh

[0091] l s =x left -nw

[0092] r s =x right +nw

[0093] Among them, u s d s , l s 、r s Represent the upper, lower, left and right boundaries of the adaptive search range respectively.

[0094] Adaptive search based on highlight point contours can dynamically adjust the search range according to the image content, so that the search process is limited to the local area around the damaged area. When the search range exceeds the image boundary, the algorithm will use the image boundary as the boundary of the search range to ensure that the search range is always inside the image. In addition, if the search range also contains other highlight point areas, the algorithm will skip these areas to avoid interfering with the repair process. This method enables the algorithm to search for the best matching block near the highlight point more accurately, which helps to improve the efficiency and accuracy of the algorithm. In other words, by performing adaptive search in a small range near the highlight point, the time overhead of the global search is effectively saved, and the repair efficiency of the algorithm is improved.

[0095] S43, calculating the restoration priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority.

[0096] The calculation of the priority function of the traditional Criminisi algorithm is based on the product of confidence and data items. During the algorithm repair process, the confidence will gradually decrease, which is easy to cause a "drip effect", which may cause some areas in the repair process to be ignored or delayed, affecting the final repair result.

[0097] In order to more accurately consider the role of data items and confidence and ensure that the texture and structural information of the image can be better propagated, the priority function can be changed from a product form to an additive form, and weights are assigned to the confidence and data items. The formula is as follows:

[0098] P (p) =0.3C (p) +0.7D (p)

[0099] Among them, C (p) , D (p) and P (p) Represent confidence, data item and priority value respectively.

[0100] The addition calculation comprehensively considers the overall layout of the image and subtle texture changes, and can more balancedly consider the impact of data items and confidence, which can not only avoid the dripping effect, but also improve the restoration effect of the algorithm. In the improved priority function, the data item is given a higher weight, which enables it to play a more significant role in the priority calculation and better propagate the structural information of the image.

[0101] However, the above method may cause discontinuity in the repair result due to the presence of multiple edge pixels in the area to be repaired. Therefore, the embodiment of the present invention averages the priority values ​​of all edge pixels in the area to be repaired as a new priority function. The calculation formula of the average priority, that is, the improved priority function of the embodiment of the present invention, is as follows:

[0102]

[0103] Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, represents the edge of the target block, C(q) and D(q) represent the confidence and data items respectively.

[0104] The improved priority function comprehensively considers the information of confidence and data items. It can not only reasonably fill the texture and structure information into the unknown area, but also effectively slow down the decline of confidence, thereby obtaining a more reliable priority value.

[0105] The combination of the two improved strategies, the improved highlight removal order and the improved priority function, can not only enable the algorithm to more effectively determine the repair order of highlight points, but also make the algorithm repair process orderly.

[0106] S44, using the improved matching criterion to find the best matching block within the adaptive search range, and copying the pixels of the best matching block to the target block with the highest priority for filling.

[0107] The traditional Criminisi algorithm uses SSD as the matching criterion and selects the matching block with the smallest SSD as the optimal matching block. However, this matching criterion only considers the color information of the pixel, which may work well when repairing relatively simple images. However, when processing images with complex textures or structures, a single SSD calculation cannot fully reflect the similarity between the matching block and the target block, which can easily lead to the search for multiple optimal matching blocks. If an inappropriate optimal matching block is selected, this error may gradually accumulate in subsequent steps, reducing the quality of image restoration.

[0108] The best matching block of the target block is usually located in its adjacent space. In addition to color features, images also contain other information, such as the distance between two points. In order to improve the accuracy of searching for the best matching block, the embodiment of the present invention introduces the Euclidean distance feature, combines SSD with physical distance, and forms a new matching criterion. This comprehensively considers multiple aspects of information such as color, distance, texture, and structure, making the search for the best matching block more comprehensive and accurate. The improved matching criterion is expressed by the following formula:

[0109]

[0110] in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

[0111] The improved SSD matching criterion can not only make the process of searching for the best matching block more comprehensive, but also help reduce the probability of algorithm mismatch, thereby improving the accuracy of searching for matching blocks and making the image have better local consistency.

[0112] S45, update the confidence and the edge of the highlight point, and determine whether there are any unrepaired pixels on the edge of the highlight point. If so, return to step S43; if not, execute step S46.

[0113] S46, determining whether all highlight points are repaired, if not, returning to step S41 to read the next highlight point, if yes, outputting the endoscope image after removing the highlight.

[0114] In summary, according to the method for removing highlight from an endoscopic image in an embodiment of the present invention, the highlight area of ​​the original endoscopic image is identified through adaptive threshold segmentation to obtain a highlight mask image, and each highlight point in the original endoscopic image is repaired one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image. The highlight area of ​​the endoscopic image can be removed conveniently and efficiently, and the integrity of the image texture and structural information can be effectively maintained, so that the processed image has a better global visual effect, and the quality of the endoscopic image after highlight removal is guaranteed.

[0115] Corresponding to the method for removing highlight from an endoscopic image in the above-mentioned embodiment, the present invention further provides a device for removing highlight from an endoscopic image.

[0116] like Figure 4 As shown, the endoscope image highlight removal device of the embodiment of the present invention includes a first acquisition module 10, a recognition module 20, a second acquisition module 30 and a repair module 40. The first acquisition module 10 is used to acquire the original endoscope image; the recognition module 20 is used to identify the highlight area of ​​the original endoscope image by adaptive threshold segmentation to obtain a highlight mask image; the second acquisition module 30 is used to acquire the contour length and area of ​​each highlight point in the highlight mask image; the repair module 40 is used to repair each corresponding highlight point in the original endoscope image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image to obtain the endoscope image after the highlight is removed.

[0117] In one embodiment of the present invention, the original endoscopic image may be acquired by the endoscope in real time during the operation.

[0118] In one embodiment of the present invention, the recognition module 20 can specifically convert the original endoscopic image from the RGB color space to the HSV color space, and then dynamically filter the highlight pixels in the HSV color space through an adaptive threshold function, and use the filtered highlight pixels to generate an initial highlight mask, and finally use a morphological dilation operation to optimize the initial highlight mask to obtain a highlight mask image.

[0119] Taking an 8-bit color image as an example, for any coordinate point in the image, the value range of each channel pixel in the RGB and HSV color spaces is as follows:

[0120]

[0121] To convert an image from RGB color space to HSV color space, first, the R, G, and B values ​​of each channel need to be normalized. Then, the values ​​of H, S, and V are calculated using the following formula, and the values ​​of each component are ensured to be within their range:

[0122] V=IllaX(R,G,B)

[0123]

[0124] The embodiment of the present invention combines the advantages of highlight detection based on color distribution and highlight detection based on adaptive threshold, uses adaptive threshold segmentation to perform highlight detection, and sets the judgment criteria of adaptive threshold segmentation according to the distribution characteristics of highlight pixels in the HSV color space.

[0125] The embodiment of the present invention sets an adaptive threshold function T according to the mean and standard deviation. s and T v , and use them to filter highlight pixels, thereby dividing the image pixels into highlight and non-highlight categories. s and T v is defined as follows:

[0126] T s =max{T sl ,Mean(I s )+Std(I s ) * α}

[0127] T v =min{T vl ,Mean(I v )+Std(I v ) * β}

[0128] Among them, I s and I v Represents the saturation channel value and the brightness channel value respectively, T sl and T vl Represents a predefined loose empirical threshold. α and β are two weight parameters, Mean and Std represent the mean and variance respectively, and are defined as follows:

[0129]

[0130] Among them, M and N are the upper limits of the number of horizontal and vertical pixels respectively.

[0131] The adaptive threshold function of the embodiment of the present invention can dynamically adjust the threshold according to the local pixel features of the image, and select a suitable threshold between the adaptive threshold and the loose threshold to screen highlight pixels. This can avoid using an inappropriate adaptive threshold in the highlight detection process, and also help improve the efficiency and accuracy of the algorithm.

[0132] In the HSV color space, highlight pixels are mainly distributed on the leftmost side of the saturation channel and the rightmost side of the brightness channel. Therefore, the embodiment of the present invention determines pixels whose saturation channel values ​​are less than or equal to and whose brightness channel values ​​are greater than or equal to as highlight pixels. By using this adaptive threshold segmentation method for highlight detection, the present invention successfully extracts highlight pixels from the original endoscopic image shown in FIG2(a), and obtains a highlight mask as shown in FIG2(b).

[0133] The morphological dilation operation includes filling holes and smoothing edges to ensure the accuracy and completeness of highlight detection. Lens movement and organ peristalsis may cause a halo effect, resulting in incoherent changes in color and brightness between the repaired highlight area and the surrounding area, and an obvious non-smooth transition. Therefore, an embodiment of the present invention uses a disk structure element with a radius of 3 to perform a morphological dilation operation on the mask to expand the highlight coverage of the mask. The highlight mask after morphological dilation is shown in Figure 2(c). The morphological dilation operation better covers the edges of the highlight area, improves the accuracy and stability of the highlight detection algorithm, and provides a more reliable basis for subsequent highlight removal.

[0134] The area of ​​the highlight point is the area of ​​the region within the outline of the highlight point. In a specific embodiment of the present invention, the units of length and area can both be expressed by the number of pixels.

[0135] In one embodiment of the present invention, referring to Figure 3 The repair module 40 specifically performs the following steps S41 to S46:

[0136] S41, reading each highlight point in order from small to large area.

[0137] By improving the highlight removal order, small highlights are repaired first, and then large highlights are repaired, ensuring that the repair order is reasonable.

[0138] The strategy adopted by the embodiment of the present invention is to repair the highlight points one by one in the order of highlight area from small to large, first repair the highlight point with the smallest highlight area, and then repair other highlight points in sequence until all highlight points are repaired. The strategy of repairing in sequence based on highlight area proposed in the embodiment of the present invention only needs to calculate the priority value of the edge pixel points of the current highlight point. This improvement can not only maintain the repair effect of the algorithm, but also greatly reduce the number of calculations and computational overhead of the algorithm. In addition, by performing highlight repair in order from small to large, it is possible to more reasonably find the area closest to the target block, which helps to improve the rationality of the algorithm repair sequence.

[0139] S42, after reading a highlight point, determining an adaptive search range according to the contour length of the highlight point.

[0140] The traditional Criminisi algorithm usually searches the entire image to find the best matching block globally. This method can ensure good global visual effects in natural image restoration. Because natural images are highly random, the image content and structure distribution are relatively random. However, endoscopic images are mainly composed of human tissues with high local characteristics. When using the traditional Criminisi algorithm to repair the highlights on a certain tissue in an endoscopic image, the global search method is not only time-consuming, but also may mistakenly match the texture of other tissues to the target tissue, resulting in unreasonable matching results.

[0141] In order to solve the problems of high computational cost, low efficiency and mismatch caused by the global search of the traditional Criminisi algorithm, the embodiment of the present invention proposes an adaptive search strategy based on the contour of the highlight point. First, the contour set of the highlight point is obtained according to the mask image obtained by highlight detection, and then the range of the adaptive search is determined according to the contour length of the highlight point. This strategy optimizes the search range of the optimal matching block according to the contour length of the highlight point, which can not only reduce the computational cost of the algorithm and the probability of mismatch, but also improve the repair efficiency and repair quality of the algorithm.

[0142] Specifically, the adaptive search strategy based on the highlight point outline is as follows: First, the coordinates of the four adjacent pixels around the highlight point can be obtained. The coordinates of the four pixels above, below, left and right are respectively expressed as u(x up ,y up )、d(x down ,y down )、l(x left ,y left )、r(x right ,y right )express.

[0143] Then, the width w and height h of the highlight point are calculated, and the contour length l of the highlight point is used to calculate the width w and height h of the highlight point. c Determine the expansion factor n:

[0144]

[0145] Wherein, a is a set value, which is determined by the resolution of the original endoscopic image, and the values ​​of n1 and n2 depend on the complexity of the texture structure of the original endoscopic image.

[0146] Finally, the upper, lower, left, and right boundaries of the adaptive search range are obtained based on the coordinates of the four adjacent pixels around the highlight point, the width and height of the highlight point, and the expansion coefficient:

[0147] μ s =y up -nh

[0148] d s =y down +nh

[0149] l s =x left -nw

[0150] r s =x right +nw

[0151] Among them, u s d s , l s 、r s Represent the upper, lower, left and right boundaries of the adaptive search range respectively.

[0152] Adaptive search based on highlight point contours can dynamically adjust the search range according to the image content, so that the search process is limited to the local area around the damaged area. When the search range exceeds the image boundary, the algorithm will use the image boundary as the boundary of the search range to ensure that the search range is always inside the image. In addition, if the search range also contains other highlight point areas, the algorithm will skip these areas to avoid interfering with the repair process. This method enables the algorithm to search for the best matching block near the highlight point more accurately, which helps to improve the efficiency and accuracy of the algorithm. In other words, by performing adaptive search in a small range near the highlight point, the time overhead of the global search is effectively saved, and the repair efficiency of the algorithm is improved.

[0153] S43, calculating the restoration priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority.

[0154] The calculation of the priority function of the traditional Criminisi algorithm is based on the product of confidence and data items. During the algorithm repair process, the confidence will gradually decrease, which is easy to cause a "drip effect", which may cause some areas in the repair process to be ignored or delayed, affecting the final repair result.

[0155] In order to more accurately consider the role of data items and confidence and ensure that the texture and structural information of the image can be better propagated, the priority function can be changed from a product form to an additive form, and weights are assigned to the confidence and data items. The formula is as follows:

[0156] P (p) =0.3C (p) +0.7D (p)

[0157] Among them, C (p) , D (p) and P (p)Represent confidence, data item and priority value respectively.

[0158] The addition calculation comprehensively considers the overall layout of the image and subtle texture changes, and can more balancedly consider the impact of data items and confidence, which can not only avoid the dripping effect, but also improve the restoration effect of the algorithm. In the improved priority function, the data item is given a higher weight, which enables it to play a more significant role in the priority calculation and better propagate the structural information of the image.

[0159] However, the above method may cause discontinuity in the repair result due to the presence of multiple edge pixels in the area to be repaired. Therefore, the embodiment of the present invention averages the priority values ​​of all edge pixels in the area to be repaired as a new priority function. The calculation formula of the average priority, that is, the improved priority function of the embodiment of the present invention, is as follows:

[0160]

[0161] Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, represents the edge of the target block, C(q) and D(q) represent the confidence and data items respectively.

[0162] The improved priority function comprehensively considers the information of confidence and data items. It can not only reasonably fill the texture and structure information into the unknown area, but also effectively slow down the decline of confidence, thereby obtaining a more reliable priority value.

[0163] The combination of the two improved strategies, the improved highlight removal order and the improved priority function, can not only enable the algorithm to more effectively determine the repair order of highlight points, but also make the algorithm repair process orderly.

[0164] S44, using the improved matching criterion to find the best matching block within the adaptive search range, and copying the pixels of the best matching block to the target block with the highest priority for filling.

[0165] The traditional Criminisi algorithm uses SSD as the matching criterion and selects the matching block with the smallest SSD as the optimal matching block. However, this matching criterion only considers the color information of the pixel, which may work well when repairing relatively simple images. However, when processing images with complex textures or structures, a single SSD calculation cannot fully reflect the similarity between the matching block and the target block, which can easily lead to the search for multiple optimal matching blocks. If an inappropriate optimal matching block is selected, this error may gradually accumulate in subsequent steps, reducing the quality of image restoration.

[0166] The best matching block of the target block is usually located in its adjacent space. In addition to color features, images also contain other information, such as the distance between two points. In order to improve the accuracy of searching for the best matching block, the embodiment of the present invention introduces the Euclidean distance feature, combines SSD with physical distance, and forms a new matching criterion. This comprehensively considers multiple aspects of information such as color, distance, texture, and structure, making the search for the best matching block more comprehensive and accurate. The improved matching criterion is expressed by the following formula:

[0167]

[0168] in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

[0169] The improved SSD matching criterion can not only make the process of searching for the best matching block more comprehensive, but also help reduce the probability of algorithm mismatch, thereby improving the accuracy of searching for matching blocks and making the image have better local consistency.

[0170] S45, update the confidence and the edge of the highlight point, and determine whether there are any unrepaired pixels on the edge of the highlight point. If so, return to step S43; if not, execute step S46.

[0171] S46, determining whether all highlight points are repaired, if not, returning to step S41 to read the next highlight point, if yes, outputting the endoscope image after removing the highlight.

[0172] In summary, according to the endoscopic image highlight removal device of the embodiment of the present invention, the highlight area of ​​the original endoscopic image is identified through adaptive threshold segmentation to obtain a highlight mask image, and each highlight point in the original endoscopic image is repaired one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image. The highlight area of ​​the endoscopic image can be removed conveniently and efficiently, and the integrity of the image texture and structural information can be effectively maintained, so that the processed image has a better global visual effect, and the quality of the endoscopic image after highlight removal is guaranteed.

[0173] In order to verify the beneficial effects of the embodiments of the present invention, in a subjective evaluation, highlight removal experiments were performed on endoscopic images using the prior art and the solutions of the embodiments of the present invention, and the results were compared.

[0174] In the experiment, multiple original endoscopic images shown in Figure 5(a) are selected, and the highlight masks shown in Figure 5(b) are obtained through preliminary processing.

[0175] The endoscopic images after removing highlights obtained by the filtering-based method, the RPCA-based method, the heat diffusion-based method, the traditional Criminisi algorithm-based method, the texture synthesis-based Criminisi algorithm-based method and the method according to the embodiment of the present invention are respectively as follows: Figure 6(a) , 6(b) , 6(c), 6(d), and 6(f).

[0176] pass Figure 6(a) to Figure 6(f) From the experimental results, we can see that although the filtering-based method has achieved certain results in removing highlights, the transition between the repaired highlight area and the surrounding area is not natural enough. For example, in the marked area in Figure 6(a), the repaired highlight area appears as white dots or white patches, which form a clear contrast with the surrounding environment in color, brightness and texture, destroying the continuity between the repaired area and the original image.

[0177] Although the RPCA-based method can remove highlights to a certain extent, its restoration effect is limited. It can only completely remove some small highlights, and for some large highlights, it can only perform a small degree of restoration. For example, in the area marked in Figure 6(b), the restored image only darkens the brightness of the highlight area, but does not completely remove it.

[0178] Although the heat diffusion-based method can effectively remove highlights, the transition between the repaired highlight area and the surrounding area is not smooth enough, as shown in the marked area in Figure 6(c).

[0179] Although the traditional Criminisi algorithm can effectively remove highlights, the restored image has problems such as unnatural edge transition and mismatch, which affects the image restoration effect, as shown in the marked area in Figure 6(d). Among them, the mismatch phenomenon in the red marked area of ​​the fourth test image is the most obvious.

[0180] Although the texture synthesis Criminisi algorithm can remove highlights relatively well, the restored image also has the problem of unnatural edge transition, as shown in the marked area of ​​Figure 6(e). Among them, the marked area of ​​the fourth test image has obvious granular spots, which appear as blurred patches and the restoration boundary is obvious.

[0181] As shown in FIG6(f), the highlight area of ​​the image restored by the solution of the embodiment of the present invention blends naturally with the surrounding area, and the non-highlight part is consistent with the original image. The entire image has no obvious restoration boundary or abrupt transition, and has an excellent global visual effect. Compared with other methods, the embodiment of the present invention has significant advantages in removing highlights. The embodiment of the present invention not only maintains the integrity of the texture, color and details of the original image, but also makes the restored image present a more unified and balanced appearance. This is due to the fact that the embodiment of the present invention has made many improvements to the Criminisi algorithm. By searching for the optimal matching block in the local area near the highlight point, the image has better local consistency and the transition of structural texture is more natural; by adjusting the order of highlight point removal, optimizing the priority function and combining the Euclidean distance feature to improve the SSD matching criterion, the algorithm restoration process is more reasonable and efficient. These improvements improve the visual quality of the restored image and make it more superior in the subjective perception of the human eye. The outstanding performance of the embodiment of the present invention in subjective evaluation proves that the improved highlight removal scheme of this paper is feasible and effective.

[0182] In order to further verify the beneficial effects of the embodiments of the present invention, in an objective evaluation, two indicators, PSNR and NIQE, are used to evaluate the highlight removal results of the solutions of the prior art and the embodiments of the present invention.

[0183] Table 1 PSNR experimental results

[0184]

[0185] As can be seen from Table 1, the PSNR value of the method based on the two-color reflection model is the lowest, which is only 16.65, which indicates that the image repaired by this method has large noise and the image restoration quality is poor. The PSNR values ​​of several other highlight removal algorithms are all higher than 30dB, which indicates that the difference between the images repaired by them and the original images is small, and the image restoration quality is good. The PSNR value of the embodiment of the present invention is not only higher than 30dB, but also higher than the PSNR values ​​of other algorithms, which shows that the embodiment of the present invention performs more prominently in noise suppression and has better image restoration quality. This is mainly because the embodiment of the present invention improves the SSD matching criterion, so that the algorithm can select the matching block with the closest distance and the highest similarity for matching when processing images containing multiple information features. Through this improvement, the algorithm can more accurately match and repair the highlight area in the image, thereby reducing the distortion and defects introduced during the restoration process.

[0186] Table 2 NIQE experimental results

[0187]

[0188]

[0189] It can be seen from Table 2 that the NIQE index of the improved Criminisi algorithm of the embodiment of the present invention is lower than that of several other algorithms, which means that the embodiment of the present invention has a better image restoration effect. This is because the embodiment of the present invention only searches in a small area near the highlight point, so that the image has better local consistency, and the transition of structural texture and the like is more natural. In addition, the embodiment of the present invention improves the priority function and repairs the image in the order of highlight area from small to large, which not only provides reliable texture and edge information for the removal of large-area highlights, but also reduces image distortion and defects, so that the restored image is clearer, natural and realistic. Therefore, the NIQE value of the image restored by the embodiment of the present invention is low.

[0190] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A method for removing highlights from an endoscope image, characterized in that: The following steps are involved: S1, obtain the original endoscopic image; S2, identifying the highlight area of ​​the original endoscopic image by adaptive threshold segmentation to obtain a highlight mask image; S3, obtaining the contour length and area of ​​each highlight point in the highlight mask image; S4, repairing each corresponding highlight point in the original endoscopic image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image, to obtain an endoscopic image after removing the highlights.

2. The method for removing highlight from an endoscope image according to claim 1, characterized in that: Step S2 specifically includes: S21, converting the original endoscopic image from the RGB color space to the HSV color space; S22, in the HSV color space, dynamically screening highlight pixels through an adaptive threshold function, and generating an initial highlight mask using the screened highlight pixels; S23, optimizing the initial highlight mask using a morphological dilation operation to obtain the highlight mask image.

3. The method for removing highlight from an endoscope image according to claim 1, characterized in that: Step S4 specifically includes: S41, reading each highlight point in order from small to large area; S42, after reading a highlight point, determining an adaptive search range according to the contour length of the highlight point; S43, calculating the restoration priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority; S44, searching for an optimal matching block within the adaptive search range using an improved matching criterion, and copying pixels of the optimal matching block to the target block with the highest priority for filling; S45, updating the confidence and the edge of the highlight point, and determining whether there are any unrepaired pixels on the edge of the highlight point, if yes, returning to step S43, if no, executing step S46; S46, determining whether all highlight points are repaired, if not, returning to step S41 to read the next highlight point, if yes, outputting the endoscope image after removing the highlight.

4. The method for removing highlight from an endoscope image according to claim 3, characterized in that: The adaptive search range is determined according to the contour length of the highlight point, including: Get the coordinates of four adjacent pixels around the highlight point; Calculate the width and height of the highlight point, and determine the expansion coefficient according to the contour length of the highlight point; The upper boundary, the lower boundary, the left boundary and the right boundary of the adaptive search range are obtained according to the coordinates of four adjacent pixel points around the highlight point, the width and height of the highlight point and the expansion coefficient.

5. The method for removing highlight from an endoscope image according to claim 3, characterized in that: The improved priority function is: Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, Ω represents the edge of the target block, C(q) and D(q) represent the confidence and data terms respectively.

6. The method for removing highlight from an endoscope image according to claim 3, characterized in that: The improved matching criterion is expressed as follows: in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

7. An endoscope image highlight removal device, characterized in that: include: A first acquisition module, wherein the first acquisition module is used to acquire an original endoscopic image; A recognition module, the recognition module is used to identify the highlight area of ​​the original endoscopic image through adaptive threshold segmentation to obtain a highlight mask image; A second acquisition module, the second acquisition module is used to obtain the contour length and area of ​​each highlight point in the highlight mask image; A repair module is used to repair each corresponding highlight point in the original endoscopic image one by one based on the improved Criminisi algorithm according to the contour length and area of ​​each highlight point in the highlight mask image, so as to obtain an endoscopic image after removing the highlights.

8. The device for removing highlight from endoscope images according to claim 7, characterized in that: The repair module specifically performs the following steps: S41, reading each highlight point in order from small to large area; S42, after reading a highlight point, determining an adaptive search range according to the contour length of the highlight point; S43, calculating the restoration priority of the edge pixel points of the highlight point according to the improved priority function, and determining the target block with the highest priority; S44, searching for an optimal matching block within the adaptive search range using an improved matching criterion, and copying pixels of the optimal matching block to the target block with the highest priority for filling; S45, updating the confidence and the edge of the highlight point, and determining whether there are any unrepaired pixels on the edge of the highlight point, if yes, returning to step S43, if no, executing step S46; S46, determining whether all highlight points are repaired, if not, returning to step S41 to read the next highlight point, if yes, outputting the endoscope image after removing the highlight.

9. The device for removing highlight from endoscope images according to claim 8, characterized in that: The improved priority function is: Among them, MP (p) represents the average priority value, represents the known area centered on pixel p, Ω represents the target block to be repaired, Ω represents the edge of the target block, C(q) and D(q) represent the confidence and data terms respectively.

10. The device for removing highlight from endoscope images according to claim 9, characterized in that: The improved matching criterion is expressed as follows: in, represents the improved SSD distance from pixel p to pixel q, represents the original SSD distance from pixel p to pixel q, and d(p,q) represents the Euclidean distance from pixel p to pixel q.

Images