Endoscope weak texture image enhancement method and device

Abstract

The endoscope weak texture image enhancement method and device can realize the removal of visual error perception, improve the number of visual feature point extraction, and improve the texture gradient information of the image. The method comprises the following steps: (I) extracting endoscope image data in clinical operation; (II) obtaining a highlight area in a single endoscope image based on color channel intensity offset; (III) completing the endoscope image area according to the endoscope highlight area; (IV) describing the spatial symmetry of the weak texture area of the endoscope image according to the spatial symmetry of the image; describing the color gradient of the weak texture area of the endoscope image according to the color gradient of the image; (V) obtaining a weak texture index according to the weighted sum of the image pixel spatial symmetry and the color gradient, and quantifying the endoscope weak texture image data; (VI) performing adaptive gradient reservation on the weak texture image based on local linear relationship.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an endoscope weak texture image enhancement method and an endoscope weak texture image enhancement device. BACKGROUND

[0002] In recent years, with the extension of the research direction of computer vision, endoscope image features are widely applied in detection, identification, positioning and reconstruction, and have become an effective way to realize value improvement in the field of computer vision. Therefore, the feature matching technology of the endoscope image becomes a key technology for obtaining the surface morphology of an object. However, due to the particularity of the operating environment in the clinical operation, the time limit of the operation and the difference of the endoscope equipment, the quality of the obtained endoscope image is quite different. Some images have weak texture.

[0003] The weak texture endoscope image has a single overall color, a high structural similarity and a few visual response points, and it is difficult to extract visual features. For the subsequent feature matching, lesion identification, surgical instrument positioning and internal environment three-dimensional reconstruction, it will cause serious influence. At present, the widely used image feature point acquisition method has a small number of visual feature points detected on the weak texture endoscope image. In addition, the image enhancement method mainly depends on two points: (1) the dynamic range of changing the image lighting condition is used to enhance the image texture; (2) the object geometric boundary descriptor is constructed to enhance the description of the weak texture object. Such enhancement method can improve the number of extracted image feature points, but will introduce more random noise and cause mismatching. Moreover, the diameter of the clinical medical endoscope lens is small, and the endoscope lens is close to the surface during the operation, which causes a narrow field of view and cannot display the single object surface contour. These limiting factors will seriously affect the subsequent visual application. SUMMARY

[0004] In order to overcome the defects of the prior art, the technical problem to be solved by the present application is to provide an endoscope weak texture image enhancement method, which can remove visual error perception, improve the number of extracted visual feature points and improve the texture gradient information of the image.

[0005] The technical scheme of the present application is that the endoscope weak texture image enhancement method comprises the following steps:

[0006] (I) extracting endoscope image data in a clinical operation;

[0007] (II) obtaining a highlight area in a single frame of endoscope image based on color channel intensity offset;

[0008] (III) completing the endoscope image area according to the endoscope highlight area;

[0009] (IV) based on the spatial symmetry of the image, the weak texture region of the endoscope image is described in terms of spatial symmetry; based on the color gradient of the image, the weak texture region of the endoscope image is described in terms of color gradient;

[0010] (V) according to the weighted sum of the spatial symmetry and the color gradient of the image pixels, a weak texture index is obtained to quantify the weak texture image data of the endoscope;

[0011] (VI) based on the local linear relationship, the weak texture image is adaptively gradient preserved.

[0012] The present application is based on the endoscope image obtained in the clinical operation, and is used to enhance the weak texture image caused by the filling of the solution in the operation, the motion blur and the loss of focal length. An image RGB three-channel intensity offset optimization model is constructed, the mirror reflection region in the endoscope image is obtained, and a fast matching model is used to complete the missing information of the mirror reflection region of the image. By using the texture information of the image, the weak texture endoscope image is enhanced by an adaptive gradient preservation method. Finally, an endoscope weak texture image enhancement scheme is generated. Therefore, the removal of visual error perception can be realized, the number of visual feature point extraction is improved, and the texture gradient information of the image is improved.

[0013] Another endoscope weak texture image enhancement method is also provided, which comprises the following steps:

[0014] (1) detecting the mirror reflection region in the endoscope image;

[0015] (2) by calculating the overall spatial symmetry and the color gradient of the image, an evaluation index of the weak texture image is obtained;

[0016] (3) according to the local linear relationship of the image pixels and the general linear model q=aI+b of the filter, the pixels of the output image are represented as a weighted average of the adjacent pixels; in the area where the edge information changes, the linear coefficient changes with the image content, and the output should be consistent with the gradient information of the original image as a constraint. In order to solve a and b, a cost function is minimized:

[0017]

[0018] wherein ε is a regularization parameter, the smaller the ε, the lower the image smoothing degree, the more the gradient preservation, but noise is introduced; on the contrary, the higher the image smoothing degree, the less the gradient preservation.

[0019] An endoscope weak texture image enhancement device is also provided, which comprises:

[0020] a detection module configured to detect the mirror reflection region in the endoscope image;

[0021] an evaluation index acquisition module configured to obtain an evaluation index of the weak texture image by calculating image overall spatial symmetry and color gradient;

[0022] a minimization cost function acquisition module configured to express a pixel of an output image as a weighted average of neighboring pixels according to a local linear relationship of the image pixel and a general linear model of a filter q=aI+b, and to change a linear coefficient with image content in an area where edge information changes, and to output an image gradient information consistent with an original image as a constraint for solving

[0023] a and b, using the minimization cost function:

[0024]

[0025] wherein ε is a regularization parameter, the smaller the ε, the lower the image smoothing degree and the more the gradient preservation,

[0026] but the noise is introduced; otherwise, the higher the image smoothing degree, the less the gradient preservation. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a flowchart of an endoscopic weak texture image enhancement method according to the present application.

[0028] Figure 2 is a flowchart of step (1) of another endoscopic weak texture image enhancement method according to the present application.

[0029] Figure 3 is a flowchart of step (2) of another endoscopic weak texture image enhancement method according to the present application.

[0030] Figure 4 is a flowchart of step (3) of another endoscopic weak texture image enhancement method according to the present application.

[0031] Figure 5 is a flowchart of a medical endoscope continuous frame image feature point matching method. DETAILED DESCRIPTION

[0032] As shown in Figure 1 , the endoscopic weak texture image enhancement method comprises the following steps:

[0033] (I) extracting endoscopic image data in a clinical operation;

[0034] (II) obtaining a highlight area in a single-frame endoscopic image based on color channel intensity offset;

[0035] (III) completing the endoscopic image area according to the endoscopic highlight area;

[0036] (IV) based on the spatial symmetry of the image, the weak texture area of the endoscope image is described in terms of spatial symmetry; based on the color gradient of the image, the weak texture area of the endoscope image is described in terms of color gradient;

[0037] (V) according to the weighted sum of the spatial symmetry and the color gradient of the image pixels, a weak texture index is obtained to quantify the weak texture image data of the endoscope;

[0038] (VI) based on the local linear relationship, the weak texture image is adaptively gradient preserved.

[0039] The present application is based on the endoscope image obtained in the clinical operation, and is used to enhance the weak texture image caused by the intraoperative solution filling, motion blur and focal length loss. An image RGB three-channel intensity offset optimization model is constructed, the mirror reflection area in the endoscope image is obtained, and a fast matching model is used to complete the missing information of the mirror reflection area of the image. By using the texture information of the image, the weak texture endoscope image is enhanced by an adaptive gradient preservation method. Finally, an endoscope weak texture image enhancement scheme is generated. Therefore, the removal of visual error perception can be realized, the number of visual feature point extraction is improved, and the texture gradient information of the image is improved.

[0040] Another endoscope weak texture image enhancement method is also provided, which comprises the following steps:

[0041] (1) detecting the mirror reflection area in the endoscope image;

[0042] (2) by calculating the overall spatial symmetry and color gradient of the image, an evaluation index of the weak texture image is obtained;

[0043] (3) according to the local linear relationship of the image pixels and the general linear model q=aI+b of the filter, the pixels of the output image are represented as a weighted average of the adjacent pixels; in the area where the edge information changes, the linear coefficient changes with the image content, and the output should be consistent with the gradient information of the original image as a constraint. In order to solve a and b, a cost function is used:

[0044]

[0045] Wherein, ε is a regularization parameter, the smaller the ε is, the lower the image smoothing degree is, the more the gradient preservation is, but noise is introduced; on the contrary, the higher the image smoothing degree is, the less the gradient preservation is.

[0046] Preferably, as shown in Figure 2 In step (1), the single-channel saturation of the endoscope image is regarded as the color channel intensity offset, and the color balance ratio is calculated as:

[0047]

[0048] Among them, P 95 (·) is 95%, c G and c B is the blue and green channel component of the pixel point, and the grayscale intensity c E Set to c E =0.3·c R +0.6·c G +0.1·c B , c R is the red channel component; by analyzing the specular reflection area of ​​the entire image, the specular reflection area is determined and saved in a binary mask; based on the fast matching model, the missing information of the specular reflection area of ​​the image is supplemented.

[0049] Preferably, if Figure 3 As shown, the step (2) includes the following sub-steps:

[0050] (2.1) The spatial symmetry of the weak texture region is described as:

[0051]

[0052] Among them, in the region R, any point P(r,θ), about any symmetry axis Symmetrical point Its pixel value satisfies τ is the tolerance factor, T ss (I) is the description of spatial symmetry, N R is the number of regions R, N E is the number of random comparisons in region R, E(·) is the binary judgment function of weak texture symmetric points. When a pair of symmetric points is within the tolerance threshold, they are judged as a pair of weak texture points. is the regional mean in order to reduce the effect of specular reflection on the results;

[0053] (2.2) The color gradient of the weak texture area is described as:

[0054]

[0055] Any point P(x, y) in image I, T cg (I) represents the color gradient, N represents the number of pixels in the window, and W (x，y) Represents a window centered on pixel (x, y), I(u, v) represents the grayscale value of the pixel in the grayscale image, Represents the average value of pixels within the window range;

[0056] (2.3) Calculate the weak texture index T less (I) is defined as:

[0057] The weak texture index is divided into 5 levels from strong to weak, and when the index is (0.8-1.0), it is extremely strong texture, (0.6-0.8) is strong texture, (0.4-0.6) is moderate texture, (0.2-0.4) is weak texture, and (0.0-0.2) is extremely weak texture; the spatial symmetry and color gradient are given different weights, respectively, α and β.

[0058] Preferably, as Figure 4 shown, in the step (3), a weak texture gradient reservation factor Γ(·) is introduced, and the value of ε is adaptively adjusted, which is represented as:

[0059] T less (I)=α*T ss (I)+β*T cg (I) (4)

[0060] Where, |W k | is the number of pixels in the window, is the second-order differential of the pixel point, is the average value of the second-order differential of the pixels in W k ; the residual of the gradient amplitude and the average value of each pixel in the window W k reflects the difference between local pixels, avoiding introducing significant differences between individuals; μ k and are the mean and variance of I, is the mean of p i in the window W k .

[0061] Preferably, in the step (3), a smaller value of ε is used in the image with high weak texture degree, reducing the constraint on a k , and obtaining better gradient reservation results, and a larger value of ε is used in the image with low weak texture degree, obtaining smaller a k , avoiding introducing too much high-frequency noise.

[0062] Preferably, the method further comprises solving the above-mentioned constructed mathematical model, and screening the feature point matching according to the solving result, to obtain a medical endoscope continuous frame image feature point matching method (as Figure 5 shown).

[0063] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be ROM / RAM, a magnetic disc, an optical disc, a memory card, etc. Therefore, corresponding to the method of the present application, the present application also simultaneously includes an endoscopic weak texture image enhancement device, which is usually represented in the form of a functional module corresponding to each step of the method. The device includes:

[0064] A detection module configured to detect a specular reflection region in the endoscopic image;

[0065] An evaluation index acquisition module configured to obtain an evaluation index of the weak texture image by calculating the overall spatial symmetry and color gradient of the image;

[0066] A minimum cost function acquisition module configured to express the pixels of the output image as a weighted average of neighboring pixels according to the local linear relationship of the image pixels and the general linear model of the filter q=aI+b. In the area where the edge information changes, the linear coefficient changes with the image content, and the output should be consistent with the gradient information of the original image as a constraint. To solve a and b, a minimum cost function is used:

[0067]

[0068] Wherein, ε is a regularization parameter, the smaller the ε is, the lower the image smoothing degree is, the more the gradient is retained, but noise is introduced; on the contrary, the higher the image smoothing degree is, the less the gradient is retained.

[0069] The present application is based on the endoscopic image obtained in the clinical operation, and is used to enhance the weak texture image caused by the filling of the solution in the operation, the motion blur and the loss of focal length. An image RGB three-channel intensity offset optimization model is constructed, the specular reflection region in the endoscopic image is obtained, and a fast matching model is used to complete the missing information of the specular reflection region of the image. The weak texture endoscopic image is enhanced by using the texture information of the image through an adaptive gradient retention method. Finally, an endoscopic weak texture image enhancement scheme is generated. The present application provides a new idea for feature extraction for weak texture endoscopic image feature matching, surgical tool recognition and positioning, and internal environment surface three-dimensional reconstruction. The problems of few feature points in the weak texture endoscopic image and large difference in feature point density between the weak texture region and the strong texture region are solved.

[0070] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiment are still within the protection scope of the technical solution of the present application.

Claims

1. Endoscopic weak texture image enhancement method, characterized by: It includes the following steps: (1) Detecting the specular reflection area in the endoscopic image; (2) By calculating the overall spatial symmetry and color gradient of the image, the evaluation index of the weak texture image is obtained; (3) Based on the local linear relationship of image pixels and the general linear model of the filter q = aI + b, the pixels of the output image are represented as a weighted average of neighboring pixels; in the area where the edge information changes, the linear coefficient changes with the image content, and the output should be consistent with the gradient information of the original image as a constraint. To solve a and b, the cost function is minimized: Among them, ε is the regularization parameter, a k and b k is the linear coefficient corresponding to pixel k in window w, I is the input image, i is the image pixel index, and p is the output image; In step (1), the color balance ratio is calculated as: Among them, P 95 (·) is 95%, c G and c B is the blue and green channel component of the pixel point, and the grayscale intensity c E Set to c E =0.3·c R +0.6·c G +0.1·c B , c R The red channel component is obtained by analyzing the mirror reflection area of ​​the entire image, determining the mirror reflection area and saving it in a binary mask; and completing the missing information of the mirror reflection area of ​​the image based on the fast matching model. The step (2) comprises the following sub-steps: (2.1) The spatial symmetry of the weak texture region is described as: Among them, in the region R, any point P(r,θ), about any symmetry axis Symmetrical point Its pixel value satisfies τ is the tolerance factor, T ss (I) is the description of spatial symmetry, N R is the number of regions R, N E is the number of random comparisons in region R, E(·) is the binary judgment function of weak texture symmetry points, is the regional mean; (2.2) The color gradient of the weak texture area is described as: Any point P(x, y) in image I, T cg (I) represents the color gradient, N represents the number of pixels in the window, and W (x，y) Represents a window centered on pixel (x, y), and I(u, v) represents the grayscale value of the pixel in a grayscale image; (2.3) Calculate the weak texture index T less (I) is defined as: T less (I)=α*T ss (I)+β*T cg (I) (4) The weak texture index is divided into 5 levels, from strong to weak. When the index is (0.8-1.0), it is very strong texture, (0.6-0.8) is strong texture, (0.4-0.6) is medium texture, (0.2-0.4) is weak texture, and (0.0-0.2) is very weak texture. Different weights are assigned to spatial symmetry and color gradient, namely α and β. In step (3), a weak texture gradient retention factor Γ(·) is introduced to adaptively adjust the ε value, which is expressed as: Among them, |W k | is the number of pixels in the window, is the second-order differential at the pixel point, W k The second-order differential mean of the inner pixels; The minimized cost function is:

2. The method for enhancing endoscopic weak texture images according to claim 1, characterized in that: The method further includes solving the constructed mathematical model and screening the feature point matching according to the solution result.

Images