Method for enhancing imaging quality of handheld fundus camera

Abstract

The invention discloses a method for enhancing the imaging quality of a handheld fundus camera. The method specifically comprises the following steps: collecting a sample image for preprocessing; constructing an image quality enhancement model, wherein the image quality enhancement model comprises a balanced illumination module, a super-resolution module and a blood vessel enhancement module; the quality problems caused by light rays such as exposure, light leakage and artifacts in the image are relieved through the illumination balancing module, and the image after illumination balancing is obtained; training a super-resolution network, and outputting a fundus image with high resolution; then, a blood vessel enhancement module is used for improving the overall contrast of the image, and pixels of the image are dynamically adjusted in a contrast-limited adaptive histogram equalization mode; and finally, performing macular retention to complete image quality enhancement. The situation that downstream tasks based on the result are seriously interfered due to poor shooting quality of the handheld fundus camera is effectively relieved, the automatic image quality enhancement function is achieved, and the production efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a method for enhancing the imaging quality of a handheld fundus camera. Background Art

[0002] Fundus images are crucial for ophthalmologists to diagnose patient diseases, and their quality directly impacts diagnostic efficiency. Traditionally, fundus images have been captured using desktop fundus cameras. These high-quality images provide doctors with clear and accurate information about fundus structures, effectively assisting in disease diagnosis. However, desktop fundus cameras are bulky and difficult to carry, limiting their application in specific scenarios, such as community screening and medical care in remote areas. Advances in medical technology and evolving needs have led to the emergence of handheld fundus cameras. These cameras offer advantages such as portability and flexibility, enabling rapid acquisition of fundus images in a wider range of scenarios, significantly improving the accessibility and convenience of fundus examinations. However, due to their portability, handheld fundus cameras are susceptible to interference during the imaging process, resulting in poor image quality. Reflections in fundus images are almost unavoidable. Furthermore, many factors can affect fundus image quality, such as optic disc overexposure, dark area artifacts, conjunctival glare, vascular blur, and poor image resolution.

[0003] In practical medical applications, high-quality fundus images are crucial for early disease detection, accurate diagnosis, and effective treatment. For example, in diagnosing retinopathy, clear vascular images help doctors observe the morphology, direction, and condition of blood vessels. In glaucoma screening, accurate fundus images help assess the health of the optic nerve. Improving image quality facilitates ophthalmologists' diagnoses and directly impacts the tasks derived from fundus images, thereby improving ophthalmologists' work efficiency. Therefore, improving the quality of fundus images captured by handheld fundus cameras is of great importance.

[0004] Although several image processing techniques have been applied to fundus image enhancement, these methods are mostly targeted at specific problems, such as single-shot reflection removal or simple contrast adjustment, and their effectiveness is limited. A systematic and comprehensive solution to the various quality issues found in images captured by handheld fundus cameras is lacking, hindering the effective improvement of overall image quality to meet the needs of medical diagnosis. Therefore, developing a comprehensive and effective image quality enhancement method for poor image quality from handheld fundus cameras is of great practical significance and has broad application prospects. Summary of the Invention

[0005] The first objective of this invention is to address the issues raised in the background art by proposing a method for enhancing the image quality of handheld fundus cameras. This method, based on deep learning technology, automatically enhances fundus images by effectively correlating low-quality and high-quality fundus images based on illumination, clarity, and contrast.

[0006] The present invention adopts the following technical solutions: The present invention comprises the following steps: Collect sample images and construct a dataset after preprocessing; Constructing an image quality enhancement model; the image quality enhancement model includes a light balance module (LBM), a super resolution module (SRM), and a vessel enhancement module (VEM); wherein the light balance module (LBM) is used to process an input image to obtain an image with overall balanced light and input it into the super resolution module (SRM); the super resolution module (SRM) is used to improve the resolution of the image and input it into the vessel enhancement module (VEM); the vessel enhancement module (VEM) is used to increase the contrast of the vascular area in the image and then output the final image after macula-sparing processing; Use the dataset to train the image quality enhancement model; The trained image quality enhancement model is used to process the actual fundus image to improve the quality of the actual fundus image.

[0007] Furthermore, the sample image is a high-quality image HQ, and a corresponding low-quality image LQ is obtained after preprocessing, and the high-quality image HQ and the low-quality image LQ are used together as a data set.

[0008] In the present invention, the low-quality image LQ is defined as a fundus image containing defects such as optic disc overexposure, dark area artifacts, conjunctival reflection, vascular blur and poor image resolution; the high-quality image HQ is defined as a fundus image without the above defects.

[0009] Furthermore, the balanced illumination module adopts a Cofe-Net network.

[0010] Furthermore, the super-resolution module adopts a Real-ESRGAN network, including a generator G and a discriminator D; The generator G includes a shallow feature extraction layer, a deep feature extraction layer, and an upsampling layer; the shallow feature extraction layer includes a 3×3 convolution layer and a LeakyReLU activation function, and the deep feature extraction layer uses multiple RRDB (Residual-in-Residual Dense Block) blocks, each of which includes a Dense Layer and a LeakyReLU, where each Dense Layer consists of a 3×3 convolution layer and a LeakyReLU activation function; the upsampling layer expands the height and width of the original image to 4 times, outputs a super-resolution image, and achieves resolution improvement; The discriminator D adopts the U-Net structure, which consists of 8 layers of 3×3 convolutional layers and LeakyReLU activation function. The number of channels doubles after each layer. The last layer outputs multi-scale discrimination results through Sigmoid. The size of each layer output is [64, 64, 128, 256, 512, 1024, 2048, 1]. fake High-quality images and real real High-quality images are required to go through the above process, and finally the loss of both is calculated through BCE_loss.

[0011] Furthermore, the vessel enhancement module performs a contrast-limited adaptive histogram equalization (CLAHE) algorithm, comprising the following steps: (1) Block processing: Divide the input image into non-overlapping sub-blocks of equal size, and each sub-block contains N pixels; (2) Calculate the sub-block histogram: Calculate the histogram of each sub-block; (3) Calculate the clipping threshold: set a threshold and clip the histogram; (4) Pixel redistribution: redistribute the redundant pixels to other parts of the histogram; (5) Histogram equalization: equalize the cropped histogram; (6) Pixel grayscale value reconstruction: Use bilinear interpolation to reconstruct the grayscale value of the pixel.

[0012] Furthermore, the macula-preserving processing includes the following steps: locating the fovea area on the image after blood vessel enhancement by using a macula detection algorithm, generating a mask and performing reasonable filling, and finally outputting the enhanced image.

[0013] A second object of the present invention is to provide an imaging quality enhancement system for handheld fundus images that implements the above method, comprising the following modules: Data preprocessing module, used to collect images and perform preprocessing; The image quality enhancement module enhances the quality of the preprocessed image through the trained image quality enhancement model.

[0014] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the above method.

[0015] A fourth object of the present invention is to provide a computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the above method is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The fundus image quality enhancement method presented in this paper utilizes deep learning technology, effectively improving downstream tasks (such as fundus image vessel extraction, fundus image-based optic cup and disc detection, and fundus image quality classification), reducing imaging costs and increasing production efficiency. Furthermore, the present invention incorporates a macula-preserving strategy to further enhance image processing accuracy.

[0017] The present invention can enhance the image quality based on various fundus images and has certain versatility.

[0018] 3) The present invention also has the function of restoring some blood vessels after improving dark area artifacts. Given a fundus image of poor quality, it can restore dark area details, improve overall image clarity, suppress exposure, and to a certain extent restore vascular structures obscured by reflections. It effectively alleviates the problem that fundus images taken by mobile fundus imagers cannot be used due to various factors, provides a structural reference for invisible vascular areas, and has good potential application value.

[0019] 4) The present invention overcomes the defect of existing image super-resolution algorithms that are extremely prone to introducing noise, avoids the generation of non-existent vascular structures when processing fundus images, and thus effectively reduces the misdiagnosis rate during use. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention.

[0021] Figure 2 It is a network model structure diagram of the present invention.

[0022] Figure 3 This is the structural diagram of the Cofe-Net network.

[0023] Figure 4 These are the different effects of the modules of the present invention on image enhancement.

[0024] Figure 5This is a diagram explaining the principle of the capillary fuzzy algorithm CBA of the present invention.

[0025] Figure 6 This is a diagram showing the low-quality image enhancement effect of the present invention, wherein the first and third rows are low-quality fundus images, and the second and fourth rows are enhanced fundus images. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, the implementation process of the present invention is as follows: Step 1: Collect sample images, perform preprocessing, and build a data set. The specific process is as follows: High-quality fundus images HQ were collected and processed using the capillary blurring algorithm (CBA) to obtain the corresponding low-quality images LQ. The images were resized to 512×512 and then subjected to data enhancement processing, including adding rotation angle, Gaussian blur, brightness and contrast adjustment, to form a dataset.

[0028] The principle diagram of the capillary fuzzy algorithm CBA is as follows Figure 5 As shown, the following steps are included: (1) Generate a mask image: First, extract the vascular structure in the fundus image and generate a mask image M. This mask image accurately captures the vascular structure through corrosion and expansion operations.

[0029] (2) Generate mask difference region: Corrode the blood vessels obtained by masking twice and then dilate them twice to obtain the mask image Md; subtract the two mask images to obtain the mask difference region Mb.

[0030] (3) Apply Gaussian blur: Apply Gaussian blur to the corresponding mask difference area Mb in the high-quality image HQ to generate the final low-quality image LQ.

[0031] Step 2: Build an image quality enhancement model; Figure 2 As shown in the figure, an image quality enhancement model is constructed, including a balanced illumination module LBM, a super-resolution module SRM and a vessel enhancement module VEM; Step 3: Balance Lighting; Process the input image through the Balance Lighting module input , and obtain an image with overall balanced light; the balanced illumination module adopts the Cofe-Net network, and its network structure is as follows Figure 3 shown.

[0032] Step 4: Feature extraction: The generator G learns high-quality image physiological information features by processing the dataset obtained in step 1 through the super-resolution module.

[0033] The generator uses a U-Net architecture, consisting of shallow feature extraction layers, deep feature extraction layers, and upsampling layers. The shallow feature extraction layer consists of a 3×3 convolutional layer with a LeakyReLU activation function. The deep feature extraction layer uses 23 Residual-in-Residual Dense Blocks (RRDBs) as the backbone. Each RRDB contains a dense layer and a LeakyReLU. Each dense layer consists of a 3×3 convolutional layer with a LeakyReLU activation function, with a fixed number of 64 channels. The convolutional layer consists of a convolution with an input dimension of 3, an output dimension of 64, and a stride of 2; batch normalization followed by a Swish activation function; a convolution with an input dimension of 64, an output dimension of 128, and a stride of 2; a 3×3 max pooling layer with a stride of 2; a convolution with an input dimension of 128, an output dimension of 256, and a stride of 1; and a convolution with an input dimension of 256, an output dimension of 512, and a stride of 1. After that, residual scaling and batch normalization are performed to avoid blurry textures. The upsampling layer uses a progressive method, that is, a mixture of interpolation and convolution layers, to expand the height and width to 4 times the original image respectively, and finally passes through the output layer, that is, the convolution layer with kernel_size of 3 and output_channel of 2048 and the normalized output. fake Image, represents the generated high-quality image.

[0034] Step 5: Module training: The super-resolution module is trained using the loss function between high-quality images HQ and low-quality images LQ. The generator G attempts to generate low-quality images that are closer to the features of high-quality images to deceive the discriminator D. The discriminator continuously learns to distinguish between true and false high-quality images, thereby obtaining the final super-resolution module.

[0035] The discriminator D adopts a VGG-like structure, consisting of 8 layers of 3×3 convolutional layers and LeakyReLU activation functions. The number of channels is doubled after each layer, and the last layer outputs multi-scale discrimination results through Sigmoid. The first layer is a convolution with an input dimension of 3, an output dimension of 64, and a step size of 1, plus a ReLu activation function; the second layer is a convolution with an input dimension of 64, an output dimension of 64, and a step size of 2, plus a ReLu activation function; the third layer is a convolution with an input dimension of 64, an output dimension of 128, and a step size of 1, plus spectral normalization. Normalization) and a ReLu activation function; the fourth layer is a convolution with an input dimension of 128, an output dimension of 256, and a stride of 1, plus spectral normalization and a ReLu activation function; the fifth layer is a convolution with an input dimension of 256, an output dimension of 512, and a stride of 1, plus spectral normalization and a ReLu activation function; the sixth layer is a convolution with an input dimension of 512, an output dimension of 1024, and a stride of 1, plus spectral normalization and a ReLu activation function; the seventh layer is a convolution with an input dimension of 1028, an output dimension of 2048, and a stride of 1, plus spectral normalization and a ReLu activation function; the last layer is a convolution with an input dimension of 2048, an output dimension of 1, and a stride of 1. For the generator G Generated fake high-quality images and real real High-quality images are required to go through the above process, and finally the loss of both is calculated through BCE_loss.

[0036] Step 6: Use the super-resolution module obtained after training in step 5 to test the actual image, and then process the image obtained in step 3 to obtain a high-resolution image.

[0037] Step 7: Dynamically adjust the image pixels using contrast-limited adaptive histogram equalization to increase the contrast of the blood vessel area, including the following steps: (1) Block processing: Divide the input image into non-overlapping sub-blocks of equal size, and each sub-block contains N pixels; (2) Calculate the sub-block histogram: Calculate the histogram of each sub-block; (3) Calculating the clipping threshold: Setting a threshold to clip the histogram; specifically, in this embodiment, since the image is magnified 4 times during processing, it is clipped into a small area of ​​4*4=16; (4) Pixel redistribution: redistribute the redundant pixels to other parts of the histogram; (5) Histogram equalization: equalize the cropped histogram; (6) Pixel grayscale value reconstruction: Use bilinear interpolation to reconstruct the grayscale value of the pixel.

[0038] Step 8: Yellow spot preservation: Locate the original image through yellow spot detection algorithm input The foveal area is masked and finally transferred to the enhanced high-quality image obtained in step 7. The combined parts are fused using the Poisson fusion algorithm to obtain the final enhanced image. output The proposed macula detection algorithm transforms the optic disc and macula detection problem from a traditional pixel classification task (classifying pixels as background, optic disc center, or macula center) to a pixel-level regression task, where the goal is to regress the distance of each pixel to the nearest optic disc or macula center. In this method, a fully convolutional neural network (F-CNN) is used for distance regression, using a U-Net-like fully convolutional neural network architecture. The input is a retinal image, and the output is a normalized distance map for each pixel. ;in, is defined as follows: For each pixel position , define the dual distance graph , indicating the distance from the pixel to the center of the visual disk and macular center Minimum distance: The double distance map is then normalized to values ​​in the range [0,1] .

[0039] The L2 loss function is used to optimize the fully convolutional neural network, with the goal of minimizing the difference between the predicted distance map and the true distance map; the distance map output by the fully convolutional neural network is processed by the Laplacian of Gaussian operator (LoG) to extract the two most significant extreme points as candidate locations of the optic disc and macula.

[0040] Finally, the optic disc and macula were distinguished by local intensity analysis: the optic disc area is generally brighter than the surrounding area, and the macula area is generally darker. Specific embodiment: To further verify the effectiveness of the method of the present invention, each module is first split and the image is processed separately. Figure 6 The first and third rows are used as low-quality images for testing. Figure 6 The second and fourth rows are the enhancement results.

[0042] Depend on Figure 6 As can be seen, this method effectively enhances low-quality fundus images, significantly reducing equipment resource costs and improving production efficiency. Furthermore, this method can restore some blood vessels after improving dark area artifacts, providing structural reference for invisible vascular regions, thus possessing significant potential application value.

[0043] The above specific embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for enhancing the imaging quality of a handheld fundus camera, characterized in that: The method comprises the following steps: Collect sample images and construct a dataset after preprocessing; Constructing an image quality enhancement model; the image quality enhancement model includes a balanced illumination module (LBM), a super-resolution module (SRM), and a vascular enhancement module (VEM); wherein the balanced illumination module (LBM) is used to process an input image to obtain an image with overall balanced lighting and input it into the super-resolution module (SRM); the super-resolution module (SRM) is used to improve the resolution of the image and input it into the vascular enhancement module (VEM); the vascular enhancement module (VEM) is used to increase the contrast of the vascular region in the image and then output a final image after macula-sparing processing; Use the dataset to train the image quality enhancement model; The trained image quality enhancement model is used to process the actual fundus image to improve the quality of the actual fundus image.

2. The method for enhancing the quality of a handheld fundus camera according to claim 1, wherein: The preprocessing includes: size adjustment, capillary blurring and data enhancement.

3. The method for enhancing the quality of a handheld fundus camera according to claim 2, wherein: The capillary blurring comprises the following steps: Generate mask image: First, extract the vascular structure in the resized image to generate a mask image M; Generate mask difference region: Corrode the blood vessel twice using mask image M and then dilate it twice to obtain mask image Md; subtract the two mask images to obtain mask difference region Mb; Apply Gaussian blur: Apply Gaussian blur to the corresponding mask difference area Mb in the resized image to generate the corresponding low-quality image.

4. The method for enhancing the imaging quality of a handheld fundus camera according to claim 1, wherein: The balanced illumination module adopts the Cofe-Net network.

5. The method for enhancing the imaging quality of a handheld fundus camera according to claim 1, wherein: The super-resolution module adopts the Real-ESRGAN network, including the generator G and the discriminator D; The generator G includes a shallow feature extraction layer, a deep feature extraction layer and an upsampling layer; the shallow feature extraction layer includes a 3×3 convolution layer and a LeakyReLU activation function, and the deep feature extraction layer uses multiple RRDB blocks, each RRDB block includes a Dense Layer and a LeakyReLU, where each Dense Layer consists of a 3×3 convolution layer and a LeakyReLU activation function; after processing by the deep feature extraction layer, it is input to the upsampling layer through residual scaling and batch-free normalization, and the upsampling layer expands the height and width to 4 times the original image respectively, outputs a super-resolution image, and achieves resolution improvement; The discriminator D adopts a U-Net structure, which consists of 8 layers of 3×3 convolutional layers and a LeakyReLU activation function. The number of channels doubles with each layer, and the last layer outputs multi-scale discrimination results through Sigmoid.

6. The method for enhancing the imaging quality of a handheld fundus camera according to claim 1, wherein: The vessel enhancement module performs a contrast-limited adaptive histogram equalization (CLAHE) algorithm, comprising the following steps: Block processing: Divide the input image into non-overlapping sub-blocks of equal size, and each sub-block contains N pixels; Calculate sub-block histogram: calculate the histogram of each sub-block; Calculate the clipping threshold: set a threshold to clip the histogram; Pixel redistribution: redistribute excess pixels to other parts of the histogram; Histogram equalization: equalize the cropped histogram; Pixel grayscale value reconstruction: Use bilinear interpolation to reconstruct the grayscale value of pixels.

7. The method for enhancing the imaging quality of a handheld fundus camera according to claim 1, wherein: The macula preservation processing includes the following steps: locating the fovea area of ​​the input image by using a macula detection algorithm, generating a mask to extract the macula area and overlay it on the image after blood vessel enhancement, transitioning the overlay edge by using a Poisson fusion algorithm, and finally outputting the enhanced image.

8. A system for enhancing the imaging quality of a handheld fundus camera, which implements the method according to any one of claims 1 to 7, characterized in that: Includes the following modules: A data preprocessing module, used to collect and preprocess images from a handheld fundus camera; The image quality enhancement module enhances the quality of the preprocessed image through the trained image quality enhancement model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor, characterized in that: The memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images