Image enhancement method and system based on image decomposition and spectral transformation

Abstract

The present invention discloses an image enhancement method and system based on image decomposition and spectral transformation, which relates to the field of image processing technology, including: path separation of the original image, which is divided into a noise layer image and a base layer image; detail extraction of the base layer image, which is divided into a structure layer image and a detail layer image; truncated adaptive brightness stretching of the structure layer image; de-highlighting and detail color processing of the detail layer image; detail enhancement of the detail layer image after detail color processing; weighted fusion of the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image; equal-proportion fusion of the processed base layer image and the noise layer image, and output of the result image. The present invention is particularly suitable for endoscopic images in medicine, and can adaptively amplify the details in the endoscopic image, eliminate the influence of noise, and solve the problem of over-brightness or over-darkness of local modules.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to an image enhancement method and system based on image decomposition and spectral transformation. Background Art

[0002] At present, the main algorithms for endoscopic image enhancement are: methods based on machine learning, methods based on retinex theory, and methods based on homomorphic filtering.

[0003] Based on multispectral imaging methods, there are currently three main technologies in endoscopic image enhancement technology: NBI, FICE, and I-Scan. They all achieve real-time processing by enhancing vascular features. NBI is a hardware image enhancement technology that uses narrow-band light illumination. It uses optical filters of specific wavelengths to narrow the range of the three bands of blue, green, and red, where blue is (415±15) nm, green is (540±10) nm, and red is (600±10) nm. The three bands penetrate the mucosa to different depths, so NBI technology allows blood vessels to be clearly displayed. However, its band is relatively single and the hardware is complex. FICE and I-Scan are both software enhancement technologies. FICE reconstructs color images by calculating the reflection intensity at several specific wavelengths to enhance the target of interest. It requires strict calibration of the endoscope system before use, and the color of the enhanced image is very different from the real color; while I-Scan technology can not only select different wavelength combinations to display images, but also introduces two methods: surface enhancement and contrast enhancement, but the algorithm is relatively complex.

[0004] The method based on the retinex theory, the Retinex algorithm was proposed by American physicists. The basis of the Retinex theory is the color constancy of the human visual system. The color perception of the human visual perception system has the characteristic of "preconceived", that is, when the light source conditions change, the color information received by the retina will also be rejected by the human brain. The basis of the Retinex theory is that the original image can be decomposed into an illuminated image and a reflected image. People can turn multiplication operations into simple addition operations by operating in the logarithmic domain, and then single-scale retinex algorithms, multi-scale retinex algorithms, and multi-scale retinex algorithms for color restoration have emerged. However, these methods all have certain defects: halo phenomenon is prone to occur in the transition area between strong light and shadow, mainly because the Gaussian operator cannot estimate the illumination well in the transition area. The poor processing of relatively bright images is mainly due to the fact that the logarithmic processing compresses the display range of the bright area, resulting in weakening of its details.

[0005] Based on the deep learning method, with the rise of artificial intelligence in recent years, more and more algorithms for image enhancement using neural networks have been used, and deep learning for image enhancement has gradually become the mainstream. The network is flexible and powerful, the training effect is obvious, and the enhancement efficiency is high. However, for endoscopic images, the preliminary preparation will be more troublesome. It is difficult to find enough images with special lesions and vascular contours and corresponding images without special lesions and vascular contours to meet the training requirements of deep learning, and the amount of calculation will be very large.

[0006] Therefore, how to enhance images, especially to amplify details and eliminate noise of endoscopic images in medicine, is a problem that technical personnel in this field need to solve urgently. Summary of the invention

[0007] In response to the problems in the prior art of endoscopic images such as noise amplification along with contrast, detail loss, and local modules being too bright or too dark, an innovative image enhancement method based on image decomposition and spectral transformation was proposed, which is particularly suitable for image enhancement of endoscopic vascular images.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] An image enhancement method based on image decomposition and spectral transformation comprises the following steps:

[0010] Step 1: perform path separation on the input original image, and divide the original image into a noise layer image and a base layer image;

[0011] Step 2: extract details from the base layer image, and divide the base layer image into a structure layer image and a detail layer image;

[0012] Step 3, performing truncated adaptive brightness stretching on the structure layer image;

[0013] Step 4: performing de-highlighting and detail color processing on the detail layer image;

[0014] Step 5, obtaining a scale factor α, and performing detail enhancement on the detail layer image after detail color processing;

[0015] Step 6: weighted fusion of the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image;

[0016] Step 7: Fuse the processed base layer image and the noise layer image in equal proportion and output a result image.

[0017] Optionally, the method for performing path separation on the input original image is:

[0018] Step 1.1, perform global noise estimation on the original image and obtain global noise parameters;

[0019] Step 1.2: Use the global noise parameter in a total variation structure texture decomposition method to obtain a noise layer image and a base layer image.

[0020] Optionally, a weighted least squares method is used to extract details from the base layer image.

[0021] Optionally, when performing truncated adaptive brightness stretching on the structure layer image, the image is converted into an HSI space, and truncated adaptive brightness stretching is performed on the I channel.

[0022] Optionally, the method for performing de-highlighting processing on the detail layer image is:

[0023] Preprocess and enhance the original image;

[0024] Convert the preprocessed and enhanced image from RGB to CIE-XYZ space to obtain brightness Y;

[0025] According to the brightness Y, the color brightness y is obtained;

[0026] Extract the area where the brightness Y is greater than the color brightness y, which is the highlight area;

[0027] Based on the highlighted point area and the detail layer image, a de-highlighted detail layer image is obtained.

[0028] Optionally, the detail color processing method is:

[0029] For the de-highlighted detail layer image, the red component of the R channel is suppressed through the histogram modification technology, and the green component of the G channel and the blue component of the B channel are enhanced through the S-shaped curve.

[0030] Optionally, the scale factor α is obtained based on the structure layer image and the truncated structure layer image after adaptive brightness stretching, and the specific formula is:

[0031]

[0032] Among them, std represents the standard deviation of the image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I structure Represents a structure layer image.

[0033] Optionally, the formula for weighted fusion of the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching is: I′ base =I′ structure +α·I′ detail , where I′base represents the processed base layer image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I′ detail Represents the detail layer image after detail color processing.

[0034] Optionally, the formula for performing equal-proportion fusion of the processed base layer image and the noise layer image is: I′=I′ base +I noise , where I represents the output image after proportional fusion, I′ base Represents the processed base layer image, I noise Represents the noise layer image.

[0035] The present invention also provides an image enhancement system based on image decomposition and spectral transformation, comprising:

[0036] A path separation module is used to perform path separation on the input original image, dividing the original image into a noise layer image and a base layer image;

[0037] A detail extraction module, used to extract details from the base layer image, and divide the base layer image into a structure layer image and a detail layer image;

[0038] A brightness stretching module, used for performing truncated adaptive brightness stretching on the structure layer image;

[0039] A color processing module, used for performing de-highlighting and detail color processing on the detail layer image;

[0040] A detail enhancement module is used to obtain a scale factor α and perform detail enhancement on the detail layer image after detail color processing;

[0041] A first image fusion module is used to perform weighted fusion on the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image;

[0042] The second image fusion module is used to fuse the processed base layer image and the noise layer image in equal proportion and output a result image.

[0043] It can be seen from the above technical solutions that the present invention discloses an image enhancement method and system based on image decomposition and spectral transformation, which has the following beneficial effects compared with the prior art:

[0044] The present invention combines the global noise estimation method set by the original image with the relative total variation method to extract the noise layer, ensuring that the image enhancement is performed under noise suppression, effectively preventing noise amplification. The weighted least squares method is used to decompose the base layer into a structure layer and a detail layer, with a small amount of calculation and obvious decomposition effect, which is convenient for subsequent detail enhancement. The structure layer adopts truncated adaptive brightness stretching to prevent the highlights in some areas from being stretched too bright or the stretching of some dark areas from being blurred. The detail layer is improved in three channels based on the removal of highlights, which solves the interference of highlights in the endoscopic image, enhances the characteristics and information of the blood vessels, and enriches the details to the greatest extent. Further, a weighted fusion detail layer enhancement (scale) factor is proposed to ensure the adaptive magnification of the relevant details of the endoscopic image. The method of the present invention is used for image processing, which is particularly suitable for endoscopic images in medicine, can adaptively magnify the details in the endoscopic image, eliminate the influence of noise, and solve the problem of excessive brightness or darkness of the local module. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 is a flow chart of the method steps of the present invention;

[0047] Figure 2 It is a schematic diagram of image structure change of the present invention;

[0048] Figure 3 It is a schematic diagram of the system structure of the present invention;

[0049] Figure 4 This is an original diagram of an embodiment of the present invention;

[0050] Figure 5 An image processed by an embodiment of the present invention;

[0051] Figure 6 This is an original diagram of another embodiment of the present invention;

[0052] Figure 7 This is an image processed according to another embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses an image enhancement method based on image decomposition and spectral transformation. The method steps are shown in Figure 1 , the structural change process of the image can be seen in Figure 2 The specific method is:

[0055] Step 1: Perform path separation on the input original image and divide the original image into a noise layer image and a base layer image. Specifically:

[0056] Step 1.1: Perform global noise estimation on the original image to obtain global noise parameters.

[0057] The image noise estimation algorithm based on arithmetic mean is used. Because the image edge structure has a strong second-order difference characteristic, the image is sensitive to the noise statistics of the Laplace mask, that is, the convolution operation is performed through the kernel composed of two Laplace masks. The set global noise parameter is obtained by the following formula:

[0058]

[0059] Where * represents the convolution operator, W and H represent the width and height of the image, respectively, k is the endoscopic image noise suppression factor, which is used to control the degree of detail, and I c represents the input image, N s represents the filter kernel, (x, y) represents an image pixel.

[0060] Step 1.2: Use the global noise parameter in a total variation structure texture decomposition method to obtain a noise layer image and a base layer image.

[0061] In the total variation method, the input image is regarded as a superposition of a base image and a noise image, and the formula is:

[0062]

[0063] Among them I c (x,y) represents the original image, represents the base layer image, Represents the noise layer image.

[0064] Combined with TV regularization, the base layer is obtained by minimizing the following objective function. The objective function consists of two parts. The first is the difference term adapted to the texture component, and the second is the regularization term based on the total change, which is used to limit the details of the image. ▽ represents the gradient operator, and λ here c That is, the parameter ε obtained by estimating the global noise of the endoscope c The objective function is:

[0065]

[0066] Step 2: Use weighted least squares method to extract details from the base layer image, and divide the base layer image into a structure layer image and a detail layer image. Specifically:

[0067] The base layer is filtered by weighted least squares to obtain the structure layer, and then the detail layer is obtained by subtracting the structure layer from the base layer. Extracting the detail layer based on the weighted least squares method can extract good detail information while maintaining the original image structure. Compared with the artifacts that are prone to occur in bilateral filtering and the complexity of guided filtering guided image selection, the weighted least squares method is applied to endoscopic image enhancement and the effect will be better.

[0068] Step 3, performing truncated adaptive brightness stretching on the structure layer image;

[0069] The structure layer and detail layer are obtained by weighted least squares method, and the structure layer is further truncated for adaptive brightness stretching. Conventional brightness stretching may cause local over-enhancement in endoscopic images. The cumulative density function obtained by using the shearing histogram can adaptively determine the parameter α. At the same time, due to the problem of detail and edge loss at the edges of bright areas of some dark images, this is caused by the use of too low gamma values ​​for high-intensity pixels, so a truncation value is added to ensure that γ is limited to a certain range. The truncated adaptive brightness stretching function is improved on the function based on gamma correction, changing the manually set α to adaptive. At the same time, the image is converted to HSI space and processed in the I channel without affecting the change of color, saturation and other information. The transformation formula for truncated adaptive brightness stretching of the image is:

[0070]

[0071] Where T{I(u,v)} represents the truncated image after adaptive brightness stretching, round is the rounding function, I(u,v) represents the input image, and I max (u,v) represents the maximum pixel coordinate of the input image, γ represents the transformation parameter, and the calculation formula of γ is:

[0072] γ=max(τ,1-C W (i)),

[0073] Among them, τ is the lower limit value set to ensure that a very small value will not be used to calculate and cause ambiguity. The weighted cumulative distribution function (CDF) is:

[0074]

[0075] Where i represents the intensity level, I max is the maximum intensity level, the weighted probability density sum (PDF) is:

[0076]

[0077] The weighted distribution histogram distribution function is:

[0078]

[0079] Where P max and P min is the maximum and minimum value of the clipping histogram, α = c(i), c(i) is defined as follows:

[0080]

[0081] The method of automatically calculating the gamma value using the CDF obtained from the input image, where the probability density is: h c (i) To control the degree of enhancement, the design shall be:

[0082]

[0083] h(i) is the original histogram, h c (i) is the shear histogram, T c As the shear limit value, it is calculated based on the average value of strength, the formula is as follows:

[0084]

[0085] Where L represents the gray level of the image.

[0086] Finally, the enhanced I channel is returned and converted into RGB space to obtain the enhanced structure layer image.

[0087] Step 4: performing de-highlighting and detail color processing on the detail layer image;

[0088] Step 4.1: De-highlight the detail layer image.

[0089] In order to prevent the contrast of the highlight areas in the endoscopic image from being enhanced in the detail layer, highlight removal is performed before detail layer processing.

[0090] Step 4.1.1: Perform preprocessing and enhancement on the original image to make the reflective area more obvious, the non-reflective area less obvious, and reduce related interference factors. The preprocessing enhancement uses a simple nonlinear filter:

[0091]

[0092] Step 4.1.2, using the idea that the brightness Y (luminance) of the reflective pixel is greater than its chromatic brightness y (chromatic luminance), convert the pre-processed enhanced image from RGB to CIE-XYZ space to obtain the brightness Y;

[0093] Step 4.1.3, obtaining color brightness y according to brightness Y;

[0094]

[0095] Step 4.1.4, extract the area where the brightness Y is greater than the color brightness y, which is the highlight area;

[0096] Step 4.1.5: Based on the highlighted point area and the detail layer image, a de-highlighted detail layer image is obtained.

[0097] Step 4.2, detail color processing, for the de-highlighted detail layer image, the red component of the R channel is suppressed through the histogram modification technology, and the green component of the G channel and the blue component of the B channel are enhanced through the S-shaped curve.

[0098] For endoscopic images, separating the three-channel images shows that the vascular features of the R channel are the least obvious, while the vascular features and details of the G and B channels are richer. After inspection, blue light is most suitable for enhancing superficial mucosal structures and detecting tiny mucosal changes. Green light is relatively more suitable for enhancing thick blood vessels in the middle layer of the mucosa. Therefore, the blue and green components are more advantageous for extracting endoscopic image information. The red component is suppressed by the histogram modification technology, where the red channel stretching function is as follows:

[0099] T(i)=(i max -s)(i / i max ) γ ,

[0100] where i max represents the maximum intensity of the input image, and s represents the suppression factor that controls the degree of suppression.

[0101] Among them, s and γ determine the function trend. The green and blue components of the endoscopic image are contrast enhanced using the S-shaped curve. In order to enhance the contrast of the perceived image within a limited dynamic range, the high-intensity and low-intensity areas are compressed and the medium-intensity areas are stretched. The S-shaped mapping function is:

[0102] T(i)=i max 1 / (1+e a(b-i) ),

[0103] Where a and b are the parameters chosen for best performance.

[0104] Step 5, obtaining a scale factor α, and performing detail enhancement on the detail layer image after detail color processing;

[0105] The scaling factor α is obtained based on the structure layer image and the truncated structure layer image after adaptive brightness stretching. The specific formula is:

[0106]

[0107] Among them, std represents the standard deviation of the image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I structure Represents a structure layer image.

[0108] Step 6: weighted fusion of the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image;

[0109] The weighted fusion function is:

[0110] I′ base =I′ structure +α·I′ detail ,

[0111] Among them, I′ base represents the processed base layer image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I′ detail Represents the detail layer image after detail color processing.

[0112] Step 7: Fuse the processed base layer image and the noise layer image in equal proportion to ensure that no detail information is lost, and output the result image.

[0113] I′=I′ base +I noise ,

[0114] Among them, I represents the output result image after proportional fusion, I′ base Represents the processed base layer image, I noise Represents the noise layer image.

[0115] The embodiment of the present invention further discloses an image enhancement system based on image decomposition and spectral transformation, comprising:

[0116] A path separation module is used to perform path separation on the input original image, dividing the original image into a noise layer image and a base layer image;

[0117] A detail extraction module, used to extract details from the base layer image, and divide the base layer image into a structure layer image and a detail layer image;

[0118] A brightness stretching module, used for performing truncated adaptive brightness stretching on the structure layer image;

[0119] A color processing module, used for performing de-highlighting and detail color processing on the detail layer image;

[0120] A detail enhancement module is used to obtain a scale factor α and perform detail enhancement on the detail layer image after detail color processing;

[0121] A first image fusion module is used to perform weighted fusion on the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image;

[0122] The second image fusion module is used to fuse the processed base layer image and the noise layer image in equal proportion and output a result image.

[0123] The above method and steps are used to enhance the endoscopic images in medicine. The results are shown in Figure 4-Figure 7 , showing two groups of input images and their corresponding enhancement results, it can be seen that the use of the method of the present invention to enhance the endoscopic image can better adaptively amplify the image details, eliminate the influence of noise, and overcome the problem of local over-brightness or over-darkness.

[0124] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0125] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image enhancement method based on image decomposition and spectral transformation, characterized in that: The following steps are involved: Perform path separation on the input original image, and divide the original image into a noise layer image and a base layer image; Extracting details from the base layer image, dividing the base layer image into a structure layer image and a detail layer image; Performing truncated adaptive brightness stretching on the structure layer image; Performing de-highlighting processing and detail color processing on the detail layer image; Obtaining a scale factor α, and performing detail enhancement on the detail layer image after detail color processing; The detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching are weightedly fused to obtain a processed base layer image; Fusing the processed base layer image and the noise layer image in equal proportions, and outputting a result image; The method for performing de-highlighting processing on the detail layer image is: Preprocess and enhance the original image; Convert the preprocessed and enhanced image from RGB to CIE-XYZ space to obtain brightness Y; According to the brightness Y, the color brightness y is obtained; Extract the area where the brightness Y is greater than the color brightness y, which is the highlight area; Based on the highlighted point area and the detail layer image, a de-highlighted detail layer image is obtained; The method of detail color processing is: For the de-highlighted detail layer image, the red component of the R channel is suppressed by the histogram modification technology, and the green component of the G channel and the blue component of the B channel are enhanced by the S-shaped curve; The scale factor α is obtained based on the structure layer image and the truncated structure layer image after adaptive brightness stretching, and the specific formula is: Among them, std represents the standard deviation of the image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I structure Represents a structure layer image.

2. The image enhancement method based on image decomposition and spectral transformation according to claim 1, characterized in that: The method for performing path separation on the input original image is: Perform global noise estimation on the original image to obtain global noise parameters; The global noise parameter is used in a total variation structure texture decomposition method to obtain a noise layer image and a base layer image.

3. The image enhancement method based on image decomposition and spectral transformation according to claim 1, characterized in that: The weighted least square method is used to extract details from the base layer image.

4. The image enhancement method based on image decomposition and spectral transformation according to claim 1, characterized in that: When performing truncated adaptive brightness stretching on the structure layer image, the image is converted into the HSI space, and truncated adaptive brightness stretching is performed on the I channel.

5. The image enhancement method based on image decomposition and spectral transformation according to claim 1, characterized in that: The formula for weighted fusion of the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching is: base =I′ structure +α·I′ detail , where I′ base represents the processed base layer image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I′ detail Represents the detail layer image after detail color processing.

6. The image enhancement method based on image decomposition and spectral transformation according to claim 1, characterized in that: The formula for fusing the processed base layer image and the noise layer image in equal proportion is: I′=I′ base +I noise , where I represents the output image after proportional fusion, I′ base Represents the processed base layer image, I noise Represents the noise layer image.

7. An image enhancement system based on image decomposition and spectral transformation, characterized in that: include: A path separation module is used to perform path separation on the input original image, dividing the original image into a noise layer image and a base layer image; A detail extraction module, used to extract details from the base layer image, and divide the base layer image into a structure layer image and a detail layer image; A brightness stretching module, used for performing truncated adaptive brightness stretching on the structure layer image; A color processing module, used for performing de-highlighting and detail color processing on the detail layer image; A detail enhancement module is used to obtain a scale factor α and perform detail enhancement on the detail layer image after detail color processing; A first image fusion module is used to perform weighted fusion on the detail layer image after detail enhancement and the structure layer image after truncated adaptive brightness stretching to obtain a processed base layer image; A second image fusion module, used for fusing the processed base layer image and the noise layer image in equal proportions, and outputting a result image; The method for performing de-highlighting processing on the detail layer image is: Preprocess and enhance the original image; Convert the preprocessed and enhanced image from RGB to CIE-XYZ space to obtain brightness Y; According to the brightness Y, the color brightness y is obtained; Extract the area where the brightness Y is greater than the color brightness y, which is the highlight area; Based on the highlighted point area and the detail layer image, a de-highlighted detail layer image is obtained; The method of detail color processing is: For the de-highlighted detail layer image, the red component of the R channel is suppressed by the histogram modification technology, and the green component of the G channel and the blue component of the B channel are enhanced by the S-shaped curve; The scale factor α is obtained based on the structure layer image and the truncated structure layer image after adaptive brightness stretching, and the specific formula is: Among them, std represents the standard deviation of the image, I′ structure Represents the truncated structure layer image after adaptive brightness stretching, I structure Represents a structure layer image.

Images