A pyramid conditional normalization flow method for low-quality fundus image enhancement

Abstract

This invention discloses a pyramid conditional normalized flow method for enhancing low-quality fundus images. Step S1: Prepare a training dataset. Randomly select 10,000 high-quality images labeled "Good" from the EyeQ dataset and use image quality degradation methods such as optical transmission interference, image blurring, and retinal artifacts to generate corresponding low-quality fundus images to form image pairs as the training dataset. Step S2: Build a network model. Map the high-quality fundus images to latent variables and combine them with the low-quality fundus images as input. The reversible pyramid flow module decouples different frequency components of the fundus images. The retinal structure conditional module is used to extract fundus structures, and the extracted fundus structures serve as conditions to guide the training of the reversible pyramid flow module. Step S3: Train the network model. Input the collected low-quality retinal images into the constructed network method, load the pre-trained network parameters, and perform image enhancement processing to obtain the enhanced image.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, specifically to an enhancement network model (PCFlow) applied to low-quality fundus images. Background Technology

[0002] Retinal imaging enables the visualization of capillaries, arteries, and veins in the fundus, making them crucial for diagnosing and evaluating a range of ophthalmic pathologies. However, the quality of captured retinal images is easily compromised by complex lighting conditions, dust, and blinking movements during fundus imaging, leading to misdiagnosis of serious eye diseases due to overexposure and underexposure, uneven light distribution, and low contrast. Therefore, image enhancement techniques for low-quality fundus images are of significant research importance.

[0003] Currently, fundus image enhancement algorithms can be divided into two main categories: traditional fundus image enhancement algorithms and deep learning-based fundus image enhancement algorithms.

[0004] 1. Traditional fundus image enhancement algorithms mainly include methods based on histogram equalization and retinex theory. Histogram equalization-based enhancement methods are simple and efficient, effectively enhancing the contrast of fundus images. However, they heavily rely on the global histogram of the image, potentially leading to the loss of detailed information. Retinex-based enhancement methods generally include SSR (single-scale Retinex), MSR (multi-scale Retinex), and other improved algorithms based on Retinex theory. This type of method decomposes the image into a brightness map and a reflectance map, where the reflectance map reflects the image's intrinsic properties and can be used to generate the final enhanced image. However, such methods focus on image brightness rather than features beneficial for clinical disease diagnosis, such as fundus structure and detail features, which may lead to undesirable color and structural distortions.

[0005] 2. Deep learning-based image enhancement algorithms have been widely used in image processing tasks such as image deblurring, image denoising, and image enhancement. Currently, deep learning-based methods are also widely applied in the field of fundus image enhancement. For example, a bidirectional generative adversarial network (StillGAN) with structural and illumination constraints treats low-quality and high-quality images as two distinct domains and introduces local structural and illumination constraints to learn global features and local details. However, this method focuses only on generating realistic images, neglecting lesion areas important for clinical applications. To address this issue, subsequent approaches have included clinically oriented fundus enhancement networks, importance-guided semi-supervised learning frameworks, Transformer-based adversarial networks, and end-to-end optimized teacher-student frameworks. These methods have achieved significant progress compared to traditional methods. However, most of these methods primarily focus on improving the visual quality of images, aiming to learn deterministic pixel-to-pixel mappings, which may produce artifacts and reduce diagnostic accuracy. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention proposes a pyramid conditional normalized flow model. This model uses retinal structure as a conditional factor to constrain model optimization and employs a pyramid structure to construct a reversible coupling layer to identify various frequency components of retinal features while preserving retinal anatomy and pathological details.

[0007] To address the shortcomings of existing technologies, this invention proposes an image enhancement network model for learning the complex relationship between low-quality fundus images and their corresponding high-quality versions. The technical solution employed is as follows:

[0008] Step S1: Prepare the training dataset. Randomly select 10,000 high-quality images labeled "Good" from the EyeQ dataset as label images, and use image quality degradation methods such as light transmission interference, image blurring, and retinal artifacts to generate corresponding low-quality fundus images as input images.

[0009] Step S2: Establish a network model by mapping high-quality fundus images to latent variables and combining them with low-quality fundus images as input. The high-frequency filter in the retinal structure conditional module extracts the retinal structure, which is then passed through a network consisting of 8 RRDB modules. This network is then used as the conditional guide for training the reversible pyramid flow module. Simultaneously, the reversible pyramid flow module decomposes the input image into feature sets of different scales. The coupling layer in the pyramid flow module fuses multi-scale pyramid features using multi-scale affine parameters, employs a reversible 1×1 convolutional layer for feature fusion, and finally activates the standard layer with normalized activation to obtain the enhanced image.

[0010] Step S3: Train the network model. Combine the dataset obtained in step S1 according to the batch size as input, set the initial learning rate, learning rate decay rate, training times, weight initialization method, and loss function of the network model. Optimize the network parameters using the Adam optimizer by minimizing the negative log-likelihood loss function.

[0011] Step S4: Image enhancement. Input the collected low-quality fundus images into the constructed network model, load the pre-trained network parameters, and perform image enhancement processing to obtain the enhanced image.

[0012] Beneficial effects:

[0013] (1) This invention proposes a novel Pyramid Conditional Flow (PCFlow) model that learns the complex relationship between low-quality fundus images and their corresponding high-quality versions, rather than a deterministic pixel-to-pixel mapping. PCFlow is able to restore images with comfortable visual quality, focusing on enhancing valuable clinical information in the results, and achieving better performance in both quantitative metrics and visual quality.

[0014] (2) This invention proposes a retinal structure condition module, which uses the retinal structure as a condition factor to constrain the optimization of PCFlow, and uses a pyramid structure to construct a reversible coupling layer to identify the frequency components of retinal features, while preserving the anatomical structure and pathological features of the retina for clinical use.

[0015] (3) The network model proposed in this invention achieves state-of-the-art performance in fundus image enhancement tasks. On the DRIVE, REFUGE, and Subtrain-EyeQ datasets, it ranks first in both SSIM and PSNR compared to other state-of-the-art methods (Setiawan, DCP, RetinexNet, COFNet, SCRNet, StillGAN, T-MAGE-Net, ARCNet, PCENet, CEFNet). Furthermore, on the RETFORM dataset, which includes both real low-quality and high-quality paired fundus images, it still ranks first in both SSIM and PSNR compared to other methods (COFNet, SCRNet, RealSR, ACRNet, RFormer, PCENet). Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the image enhancement system for low-quality fundus images according to the present invention;

[0017] Figure 2 This is a flowchart of a structure-pyramid conditional flow processing method for enhancing low-quality fundus images according to the present invention;

[0018] Figure 3 This is a schematic diagram illustrating the qualitative comparison results of the method of this invention with other advanced methods on the DRIVE dataset;

[0019] Figure 4 This is a schematic diagram illustrating the qualitative comparison results of the method of this invention with other advanced methods on the REFGUE dataset;

[0020] Figure 5 This is a schematic diagram illustrating the qualitative comparison results of the method of this invention with other advanced methods on the eyeQ dataset;

[0021] Figure 6 This is a schematic diagram illustrating the qualitative comparison results of the method of this invention with other advanced methods on the RETFORM dataset;

[0022] Figure 7 This is a schematic diagram that provides a direct comparison of the enhancement effect of the method of the present invention on the results of blood vessel segmentation with other advanced methods. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0024] This invention proposes a pyramid conditional normalized flow (PCFlow) method for enhancing low-quality fundus images. Specifically, it uses retinal structure as a conditional factor to constrain the optimization of PCFlow and employs a pyramid structure to construct reversible coupling layers for identifying each frequency component of retinal features, while preserving retinal anatomy and pathological features for clinical application. The network model structure is as follows: Figure 1 As shown:

[0025] According to an embodiment of the present invention, the network mainly includes an Invertible Pyramid Coupling Module and a Retinal Structure Conditional Module. The Invertible Pyramid Coupling Module aims to independently identify each frequency component of the retinal features. The pyramid layer decomposes the input into different feature sets. The pyramid coupling layer first fuses multi-scale pyramid features using multi-scale affine parameters, then performs further feature fusion using a reversible 1×1 convolutional layer, and finally activates a normalization layer for normalization activation. The Retinal Structure Conditional Module uses a reasonable retinal structure as a conditional factor for the normalization flow to constrain the final image enhancement result, specifically including the following steps:

[0026] S1. Prepare the training dataset;

[0027] A total of 29,032 retinal images were collected from public sources and hospitals. Then, 10,000 images labeled "Good" were randomly selected from the EyeQ dataset as the training dataset, capturing high-quality fundus images. For high-quality fundus images without corresponding low-quality labels, we synthesized degraded images using optical transmission interference, image blurring, and retinal artifacts.

[0028] S2. Build the network model;

[0029] The network in this example is mainly divided into two parts: a reversible pyramid coupling module and a retinal structure condition module.

[0030] (1) Reversible pyramid coupling module: In this example, the present invention first provides an input image I and a decomposition level L. The pyramid layer decomposes the input into different feature sets, as shown in Equation 1. D() and U() are downsampling and upsampling operators, respectively. i Let represent the pyramid features of layer i. After obtaining these decomposed features, they are used as input to the pyramid coupling layer. The pyramid coupling layer first fuses the multi-scale pyramid features using multi-scale affine parameters, then performs further feature fusion using a reversible 1×1 convolution, while the activation normalization layer serves as a normalization layer within the process step to normalize activation, promoting stable and effective learning through consistent statistical properties between the data. This process is shown in Equation 2, where A represents the affine coupling operator, C represents the reversible 1×1 convolution layer, and N is the activation normalization layer.

[0031] P i =D i (P)-U(D i+1 (P) )(1)

[0032] 0=N(C(A(P i(2)

[0033] Affine coupling operators are used to decompose input features along the channel dimension to achieve invertible linear non-mapping. One of the decomposed sub-features retains its original properties, while the other undergoes an affine transformation under its control, which can be formally written as Equation (3), where x i0 and x i1 It is a feature of segmentation, y i0 and y i1 The output is the i-th level, and s() and t() are affine parameters. The logarithmic determinant of the Jacobian matrix in equation (4) can be estimated as shown in equation (5), and the affine parameters are estimated from the two linear layers. m and n represent the positions of s. The pyramid coupling layer is a pyramid flow of K stacked layers, with K=3.

[0034] y i0 =x i0

[0035]

[0036] (2) Retinal Structure Conditioning Module: In this example, the present invention first extracts the high-frequency components of the fundus image based on a high-pass filter with a Gaussian kernel, and embeds them into the model to extract the retinal structure. The process can be represented as Equation (5). H represents the high-pass filter, G represents the Gaussian filter, and m and σ represent the mean and variance of the Gaussian kernel, respectively. After extracting the retinal structure, it will be used as part of the input of the affine coupling layer in the pyramid coupling module to guide the model optimization. The specific process can be represented as Equation (6). p represents the input feature, A represents the affine coupling operator, and θ is the feature converter that provides scaling and shifting for the affine coupling layer during forward and backward propagation. The network consists of 8 RRDB modules:

[0037] H(I)=IG(m,σ) (5)

[0038] f=A(θ(H(I),p)) (6)

[0039] (3) Loss Function: In this example, the present invention uses the negative log-likelihood loss function, applying L1 loss and VGG loss to the enhanced fundus image during backpropagation. The L1 loss is used to enhance the feature details of the enhanced fundus image result and is defined by equation (7), where x is the real fundus image and g(x) is the generated image:

[0040] Loss l1 =|g(x)-x| (7)

[0041] The VGG loss is used to constrain the high-level structural features of the obtained results, defined as Equation (8), where VGG represents the VGG-16 network, which is pre-trained on the ImageNet dataset, and i represents the number of layers in the selected feature output:

[0042] Loss vgg =|VGG i (g(x))-VGG i (x)| (8)

[0043] The overall loss function is given by equation (9), where λ1, λ2, and λ3 are hyperparameters used to balance these loss functions:

[0044] Loss total =λ1(-log(p(x)))+λ2Loss l1 +λ3Loss vgg (9)

[0045] S3, Train the network model;

[0046] This model was implemented using the PyTorch framework trained on an NVIDIA RTX 3090Ti GPU. Furthermore, we used an initial learning rate of 2×102. -5 The Adam optimizer was used to optimize the model. The batch size and patch size were set to 24 and 512×512, respectively. The weights λ1, λ2, λ3, and λ4 were set to 1, 0.1, 0.5, and 0.01, respectively.

[0047] S4. Prepare the test dataset;

[0048] This example uses the DRIVE, REFUGE, Subtrain-EyeQ, and RETFORM datasets to evaluate the model. The DRIVE dataset contains 40 fundus images with a resolution of 512×512, the REFUGE dataset contains 400 images with a resolution of 1024×1024, the Subtrain-EyeQ dataset contains an additional 500 images from the EyeQ dataset, and the RETFORM dataset consists of 120 pairs of real low-quality and high-quality fundus images. For the dataset containing only high-quality fundus images, a degradation model is used to generate corresponding degraded low-quality fundus images.

[0049] Results analysis;

[0050] (1) Results on the DRIVE dataset, REFUGE test dataset, and Subtrain-EyeQ dataset. Qualitative comparisons of the method of this invention and state-of-the-art methods on the DRIVE dataset, REFUGE test dataset, and Subtrain-EyeQ dataset are shown below. Figure 2 , Figure 3 and Figure 4 In the table, "input" refers to the original image being input. The quantitative comparison results are shown in Table 1 (the best value for each comparison indicator is displayed in bold). Qualitatively, compared with other advanced methods, the method proposed in this invention guarantees better image quality, fewer artifacts, clearer vascular details, and is closer to high-quality fundus images. It also demonstrates advantages in detail enhancement and preservation of more complete retinal vessels. Quantitatively, compared with other advanced methods, the method proposed in this invention achieves the best values ​​in both Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) indicators. These quantitative advantages correspond to the qualitative advantages, demonstrating the superiority of this method.

[0051] Table 1. Quantitative comparison of different methods on the DRIVE, REFUGE, and Subtest-EyeQ datasets.

[0052]

[0053] (2) Results on the RETFORM dataset. Qualitative comparisons of the method of this invention and other state-of-the-art methods on the RETFORM dataset, which includes both real low-quality and high-quality paired fundus images, are shown. Figure 5 The quantitative comparison results are shown in Table 2 (the best value for each comparison metric is shown in bold). Qualitative comparison results show that the method of this invention can restore finer and clearer details of the retinal structure, which is beneficial for clinical diagnosis. Quantitative comparison results show that the method of this invention achieves best performance on the RETFORM dataset, obtaining the best values ​​in both Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. These leading quantitative comparison results demonstrate the superior performance of the method of this invention in fundus image enhancement tasks.

[0054] Table 2 shows a quantitative comparison of different methods on the RETFORM dataset.

[0055]

[0056] (3) Results in Clinical Applications. Quantitative comparisons of the method of this invention and other advanced methods in vessel segmentation tasks are shown in Table 3 (the best value for each comparison indicator is shown in bold). The results show that the method of this invention achieves optimal values ​​in AUC, accuracy (ACC), and intersection-over-union (IoU), providing the best results for vessel segmentation in images and yielding more refined and complete vessels. Qualitative comparisons of the method of this invention and other advanced methods in optic disc / optic cup detection tasks are shown in... Figure 6 The quantitative comparison results are shown in Table 4. Qualitative comparison results show that the image enhanced using the method of this invention is beneficial for the detection of the optic disc and optic cup, with clearer fundus structures and smaller artifact areas. Quantitative comparison results show that the method of this invention achieves optimal values ​​in both mIoU and Dice. These leading quantitative comparison indicators correspond to the leading qualitative comparison results, demonstrating the superiority of this method and proving its effectiveness for clinical observation.

[0057] Table 3. Quantitative comparison of blood vessel segmentation results of different methods on the DRIVE training dataset.

[0058]

[0059] Table 4. Quantitative comparison of spectral disc / spectral cup detection results of different methods on the REFGUE training dataset.

[0060]

[0061] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes will be obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A pyramid conditional normalized flow method for enhancing low-quality fundus images, characterized in that, Includes the following steps: Step S1: Prepare the training dataset. Randomly select 10,000 high-quality images labeled "Good" from the EyeQ dataset as label images, and use light transmission interference, image blurring, and retinal artifact image quality degradation methods to generate corresponding low-quality fundus images as input images. Step S2: Establish a network model by mapping high-quality fundus images to latent variables and combining them with low-quality fundus images as input. The high-frequency filter in the retinal structure conditional module extracts the retinal structure, which is then passed through a network consisting of 8 RRDB modules. This network is then used as the conditional guide for training the reversible pyramid flow module. Simultaneously, the reversible pyramid flow module decomposes the input image into feature sets of different scales. The coupling layer in the pyramid flow module fuses multi-scale pyramid features using multi-scale affine parameters, employs a reversible 1×1 convolutional layer for feature fusion, and finally activates the standard layer with normalized activation to obtain the enhanced image. Step S3: Train the network model. Combine the dataset obtained in step S1 according to the batch size as input, set the initial learning rate, learning rate decay rate, training times, weight initialization method, and loss function of the network model. Optimize the network parameters using the Adam optimizer by minimizing the negative log-likelihood loss function. Step S4: Image enhancement. The collected low-quality fundus images are input into the constructed network model, the pre-trained network parameters are loaded, and image enhancement processing is performed to obtain the enhanced image. The network model in step 2 includes a reversible pyramid flow module and a retinal structure condition module. The purpose of the reversible pyramid flow module is to independently identify each frequency component of the retinal features. The pyramid layer decomposes the input into different feature sets. The pyramid coupling layer first fuses the multi-scale pyramid features by using multi-scale affine parameters, then uses a reversible 1 × 1 convolutional layer for further feature fusion, and finally activates the normalization layer for normalization activation. The retinal structure condition module uses a reasonable retinal structure as a condition factor for the normalized flow to constrain the final image enhancement result. The retinal structure condition module: a high-pass filter based on a Gaussian kernel can extract high-frequency components of fundus images and embed them into the model to extract retinal structures. The process can be represented as Equation (5). Indicates a high-pass filter. Indicates a Gaussian filter. and Let represent the mean and variance of the Gaussian kernel, respectively. After extracting the retinal structure, it will be used as part of the input of the affine coupling layer in the pyramid coupling module to guide model optimization. The specific process can be represented as shown in equation (6), where p represents the input feature. Represents the affine coupling operator. It is a feature converter that provides scaling and shifting for the affine coupling layer during forward and backward propagation, and is a network consisting of 8 RRDB modules: ； Loss function: Using the negative log-likelihood loss function, during backpropagation, L1 loss and VGG loss are applied to the enhanced fundus image. L1 loss is used to enhance the feature details of the fundus image result; VGG loss is used to constrain the high-level structural features of the obtained result.

2. The pyramid conditional normalized flow method for enhancing low-quality fundus images according to claim 1, characterized in that: The reversible pyramid coupling module: given an input image and decomposition level The pyramid layer decomposes the input into different feature sets, as shown in Equation (1). and These are downsampling and upsampling operators. express After obtaining these decomposed features, they are used as input to the pyramid coupling layer. The pyramid coupling layer first fuses the multi-scale pyramid features by using multi-scale affine parameters, and then performs further feature fusion using reversible 1 × 1 convolution. The activation normalization layer serves as a normalization layer within the process steps to normalize activation. Stable and effective learning is promoted through consistent statistical properties between data. This process is shown in Equation (2), where A represents the affine coupling operator. This represents a reversible 1×1 convolutional layer. It is to activate the standardization layer. 。 3. The pyramid conditional normalized flow method for enhancing low-quality fundus images according to claim 2, characterized in that: The affine coupling operator is used to decompose the input features along the channel dimension to achieve an invertible linear non-mapping, where one of the decomposed sub-features retains its original properties, while the other undergoes an affine transformation under its control. This transformation can be formally written as Equation (3), where and It is a feature of segmentation. and yes Level output, and The affine parameters are given by equation (4), where the logarithmic determinant of the Jacobian matrix can be estimated as shown in equation (5). The affine parameters are estimated from two linear layers. and express The location of the pyramid coupling layer is Layered pyramid flow, settings =3, 。 4. The pyramid conditional normalized flow method for enhancing low-quality fundus images according to claim 1, characterized in that: Step 3, which optimizes the network parameters using the Adam optimizer by minimizing the negative log-likelihood loss function, specifically includes: It is defined as equation (7). These are real fundus images. Here is the generated image: Defined as Equation (8), VGG represents the VGG-16 network, which was pre-trained on the ImageNet dataset. Indicates the number of layers in the selected feature output: The overall total loss function is given by equation (9). , , These are hyperparameters used to balance these loss functions: 。

Images