An ophthalmic sign preserving fundus image enhancement method

Abstract

The present application belongs to the technical field of medical image processing, and specifically relates to an ophthalmic sign-maintained fundus image enhancement method. The method of the present application adopts a CycleGAN-based image enhancement network model, which comprises a blood vessel segmentation network and a conversion enhancement network composed of two generators and two discriminators. Considering the particularity of fundus images, the model of the present application introduces a blood vessel segmentation loss and a structure damage as a constraint in addition to using a conversion loss, so that the enhanced fundus image can better retain the fundus blood vessel details and pathological morphological structure. Experimental results show that the present method can effectively enhance the fundus image, retain the fundus signs, improve the image quality, and make the enhanced fundus image more conducive to clinical medical diagnosis. The SSGAN model is superior to the existing model in terms of objective evaluation indexes and subjective evaluation results of ophthalmologists, can effectively realize fundus image enhancement, and has a wide application prospect.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method for enhancing fundus images while preserving ophthalmic signs in fundus images. Background Technology

[0002] Fundus images, generated by fundus cameras, allow observation of important fundus tissues such as the macula and optic disc, and are commonly used and crucial for the examination and diagnosis of various ophthalmic diseases. Current fundus imaging equipment widely used in ophthalmic examinations offers advantages such as fast imaging speed and low cost; however, the resulting fundus images suffer from uneven overall illumination distribution, eyelid and eyelash occlusion, marginal artifacts, and poor contrast of ophthalmic signs, affecting the diagnostic efficiency and accuracy of clinicians and further hindering the development of downstream fundus image analysis tasks such as fundus vessel segmentation and foveal localization. While new, high-end fundus imaging equipment faces challenges such as high cost, low adoption rate, and long imaging time, it provides high-fidelity imaging of ophthalmic signs, with image quality highly suitable for ophthalmologists. Therefore, converting and enhancing ordinary fundus images captured by traditional equipment into high-quality, true-color, high-resolution fundus images, thereby effectively integrating the advantages of both imaging methods, has significant research significance and practical application value.

[0003] Currently, methods for enhancing fundus images are mainly divided into two categories: one is enhancement methods based on traditional digital image processing, and the other is enhancement methods based on deep learning.

[0004] Enhancement methods based on traditional digital image processing techniques mainly include histogram-based methods, such as histogram equalization. [1] Dynamic histogram equalization [2] and contrast-limited adaptive histogram equalization [3][4] Transform analysis-based methods, such as wavelet transform. [5] Non-subsampled contour wave transform [6][7] ; and methods based on Retinex theory [8][9] Early fundus image enhancement tasks were mostly direct applications of traditional digital image enhancement techniques, such as those by Jintasuttasak et al. [4] A contrast-limited adaptive histogram equalization method is proposed. [3] Enhancement of color fundus images, E. Daniel et al. [5] Wavelet transform was used to enhance the fundus images. Meanwhile, some studies have improved upon traditional methods to address the characteristics of medical images, such as those by Hsu et al. [1] A histogram equalization method for medical images based on color preservation is proposed, which enhances the image while better preserving its original colors.

[0005] However, these methods only modify images based on their statistical features and can only process the image as a whole. They do not take into account the importance of feature preservation in medical images and are prone to problems such as color distortion, amplification of local noise, and loss of features.

[0006] Most deep learning-based methods are supervised learning, requiring paired images as the training set. However, collecting and constructing paired original / enhanced image datasets is extremely difficult, especially for medical images. Zhu JY et al. proposed Cycle-GAN.

[10] This provides an effective solution for the transformation and enhancement between unpaired images. The network can learn features from the original image domain on unpaired datasets and transform them to the target image domain, leading to numerous fundus image enhancement methods based on recurrent generative adversarial networks. Yoo et al.

[11] They collected publicly available fundus images online and invited ophthalmologists to categorize the images into two classes: those with artifacts and those without. Using a recurrent generative adversarial network (RGAN), trained on this unpaired dataset, they largely achieved artifact removal from fundus images. However, because the RGAN lacks sufficient constraints on the detailed features within the image, it can only learn the style differences between the two domains and the global features of the image, resulting in insufficient preservation of local details in the transformed image. Noting this problem, Wan C et al.

[12] Introducing an attention mechanism into recurrent generative adversarial networks (GANs) allows the network to focus on more detailed image features. (Ma Yuhui et al.)

[13] The proposed StillGAN network introduces two loss constraints—brightness and structure—into recurrent generative adversarial networks, enabling the generation of more uniformly bright medical images. However, these methods do not take into account the special characteristics of fundus images and fail to focus on specific features of the fundus. When enhancing the optic disc and areas with lesions, problems such as lesion distortion can easily occur.

[0007] However, the above studies failed to consider the unique characteristics of fundus images. The enhancement process typically focused only on the subjective effect to the human eye rather than on fundus signs that are crucial for ophthalmologists' diagnoses. This resulted in lesion distortion in the enhanced images, affecting the accuracy of clinical diagnosis. Fundus signs are vital diagnostic evidence for ophthalmologists. For example, the length, width, and curvature of fundus vessels provide crucial information for diagnosing diabetic retinopathy and glaucoma. Preserving key signs such as vascular morphology as accurately as possible is a fundamental requirement for fundus image enhancement models.

[0008] Therefore, to overcome the above defects, this invention proposes a recurrent generative adversarial network model SSGAN (Segmentation and Structure constrained GAN) as a novel feature-preserving fundus image enhancement method. It introduces two new constraint functions, segmentation loss and structure loss, so that the enhanced fundus image can more accurately preserve the morphology and structural details of fundus vessels and better preserve the lesion morphology of the original image. Summary of the Invention

[0009] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fundus image enhancement method that can effectively preserve fundus signs.

[0010] The fundus image enhancement method for preserving ophthalmic signs provided by this invention employs the SSGAN image enhancement network model based on CycleGAN; the SSGAN model includes a blood vessel segmentation network and two generators (G... A G B Two discriminators (D) A D B A conversion enhancement network consisting of ) and ; where:

[0011] The generator (G) A G B The generator G is used to transform the input image to the image domain corresponding to the generator index. A Transform the input image to the image domain A, generator G B The input image is transformed to the image domain B; each generator uses an encoder-decoder architecture with residual blocks, the encoder extracts features from the input image to obtain a feature map, the feature map is input into the residual block for transformation to obtain a new feature representation, and finally the decoder reconstructs these features into a new image as the generator output;

[0012] The encoder consists of three convolutional layers. The first layer uses a 7×7 convolutional kernel to perform a convolution operation on the image with a stride of 1. The remaining two layers use 3×3 convolutional kernels consecutively to perform convolution operations on the image with a stride of 2. Each convolutional layer is followed by an instance normalization layer and a ReLU activation function, which respectively normalize the individual image and introduce non-linear factors. The final output is a feature map with 256 channels.

[0013] The decoder and encoder have symmetrical structures and both consist of three convolutional layers. The first two layers use 3×3 convolutional kernels to perform transpose convolutions on the feature maps with a stride of 2. Each transpose convolution is followed by an instance normalization layer and a ReLU activation function. The last layer uses 7×7 convolutional kernels to perform convolutions with a stride of 1, restoring the features to an image with three RGB channels. The tanh activation function is then used to amplify the feature effect and accelerate the convergence speed.

[0014] The residual block uses a residual network with nine convolutional layers to process the features.

[0015] The discriminator (D) A D B This is used to determine whether the input image belongs to the image domain corresponding to the discriminator, using discriminator D. A For example, D A It needs to be able to distinguish whether the input image is a real image in the image domain A or a generator G. A The generated fake image, D, too. B It needs to be able to distinguish whether the input image is a real image in the image domain B or a generator G. B The generated fake image;

[0016] The discriminator uses PatchGANs.

[14] The structure allows images to be reduced to 70×70 matrix blocks through convolution operations. This matrix is ​​then used as the final evaluation criterion to determine whether the input image is a true image in the corresponding domain. This method requires fewer parameters compared to using the entire image as the evaluation criterion, and it can produce images with better results.

[0017] The blood vessel segmentation network is a U-Net model pre-trained on fundus blood vessel images with segmentation annotations, used to segment vascular features from the input image; the segmentation results can be used to calculate blood vessel segmentation loss and quantify the difference in blood vessel distribution between the original image and the enhanced image.

[0018] The vascular segmentation network model can be divided into two symmetrical processes: downsampling and upsampling, each with five layers. In the downsampling process, each layer undergoes two convolutions and a nonlinear transformation. The result is then further downsampled using max pooling and cropped before being directly fed into the upsampling process via skip connections. In the upsampling process, each layer deconvolves the input feature map, and the deconvolution result is concatenated with the image obtained from the skip connections. The concatenated result is then subjected to two convolutions to extract features. Finally, two 1×1 convolutional kernels are used to divide the image's feature channels into two: the segmented target and the background, resulting in the final segmented image. The segmentation loss calculated using this output is used to optimize the generation of the enhanced image, allowing the generator in the circular transformation network to focus more on the vascular details of the fundus image, ensuring that the enhanced image retains the original image's fundus vascular texture details well.

[0019] The transformation network consists of two generators and a discriminator. During training, in addition to the basic transformation loss, the model also uses two new constraint functions, segmentation loss and structure loss, to address the problem of insufficient constraints in the recurrent generative adversarial network, thereby preserving the distribution of retinal blood vessels and other pathological signs during the enhancement of fundus images.

[0020] The transformation loss is a loss function used in the process of image generation and discrimination by the transformation network. It consists of three parts: adversarial loss, cycle consistency loss, and identical mapping loss. The transformation loss is defined as follows:

[0021]

[0022] Where λ1 and λ2 are the weights of the cycle consistency loss and the identical mapping loss.

[0023] Among them, combating losses It is the loss resulting from the adversarial game between the two sets of generators and discriminators during training, and is defined as follows:

[0024]

[0025] The following uses generator G as an example. A Discriminator D A To illustrate the adversarial loss described above, consider the following example: Given an image b in domain B, the generator G... A We need to generate an image G that is similar to the image in domain A. A (b), while the discriminator D A Then it is necessary to distinguish the image G that the generator forged. A (b) Distinguish between real image samples in image domain A and make a judgment on this. Similarly, given image a in domain A, generator G... BWe need to generate an image G that is similar to the image in domain B. B (a), while the discriminator D B Then it is necessary to distinguish the image G that the generator forged. B (a) Distinguish between real image samples in image domain a and make a judgment on this. E represents the expected value.

[0026] During training, the discriminator needs to maximize this loss function, while the generator needs to minimize it. They optimize each other in this adversarial process.

[0027]

[0028] Cycle consistency loss Used to constrain the similarity between the input image and the generated image of the generator, specifically defined as follows:

[0029]

[0030] This can be divided into cycle consistency in domain A and cycle consistency in domain B. For domain A, each image a in domain A, after passing through generator G... B Transform into an image G in domain B B (a) After that, it goes through generator G A Transform back to domain A to get G A (G B When (a) is passed, the image should be consistent with image a. Similarly, for domain B, each image b in domain B, after passing through generator G... A Transform into an image G in domain A A (b) After that, it goes through generator G B Transform back to domain B to get G B (G A (b) In this case, the image should be consistent with image b.

[0031] Equal mapping loss Also used to constrain the generator to preserve the features of the original image, unlike the cycle consistency loss, the equal mapping loss constrains each generator individually, and its definition is as follows:

[0032]

[0033] The congruence mapping loss constrains the generator to generate an image as similar as possible to the input image when the received input image is already a true sample in the corresponding image domain. For example, for generator G... A For example, when the input image is image a in domain A, the image G that passes through... A (a) should be as consistent as possible with image a.

[0034] To address the issues that existing technologies cannot maintain the original morphology of fundus signs in the original image, resulting in lesion degeneration, blurred optic disc area, and even loss of vascular information, this invention also uses structural loss and segmentation loss to strengthen the constraints on the generator.

[0035] (1) Structural loss

[0036] Ideally, the enhanced fundus image should exhibit the same pathological features as the original image and be clearer, facilitating ophthalmologist observation and diagnosis. Therefore, constraints related to image structure should be added to the generator. Structural Similarity (SSIM) is one such constraint.

[15] Structural similarity is a commonly used image evaluation metric that can reasonably assess the similarity between two images from three perspectives: brightness, contrast, and global structure. Compared to mean squared error (MSE), structural similarity better reflects the structural differences between two images and is closer to how the human visual system measures the difference between two images. Therefore, this invention proposes structural loss based on structural similarity to quantify the structural difference between the original image and the enhanced image, ensuring that the overall structural details of the original image are better preserved in the enhanced image. The definition of structural loss is as follows:

[0037]

[0038] In the above formula, SSIM represents the calculation of structural similarity, and its calculation formula is as follows:

[0039]

[0040] Where, μ x With μ y These are the mean values ​​of the x and y values ​​of the image, respectively, and σ is the mean value of the image. xy The covariance of the images x and y and These are the variances of the images x and y, respectively. The constant term C1 = (K1L) 2 C2 = (K2L) 2 K1 and K2 are control parameters, and L is the dynamic range of grayscale. For the normalized image, L = 1.

[0041] (2) Segmentation loss

[0042] The segmentation loss is a loss function defined using the segmentation results output by the Vascular Segmentation Network (VSN). It is used to constrain the consistency of vascular segmentation results in fundus images before and after enhancement / degradation. The definition of the segmentation loss is as follows:

[0043]

[0044] The Vessel Segmentation Network (VSN) is a pre-trained network with fixed parameters, similar to a vessel feature extractor. This network can be used during the training of a transformation network to extract vessel features from both the original and transformed images and output corresponding vessel segmentation results. The SSIM in the segmentation loss uses a sliding window calculation method (specifically: the entire fundus image is uniformly divided into H'*W' blocks, the SSIM values ​​between corresponding blocks in the original and transformed images are calculated, and finally, the average of these SSIM values ​​is calculated to obtain the final SSIM value between the original and transformed images. Note that when dividing the fundus image into multiple H'*W' blocks, the horizontal and vertical overlap pixels between blocks are M and N, respectively. In the specific implementation, H'*W' is 448*448, M=16, and N=16). By calculating the structural similarity between the vessel segmentation results of the original and transformed images, the degree to which the generator preserves vessel features is quantified, ensuring that the generated image contains vessel features consistent with the original image.

[0045] Therefore, the final objective function of the model used in this invention is:

[0046]

[0047] Where α and β are the weights of the structural loss and the segmentation loss, respectively.

[0048] The specific procedure for enhancing fundus images while preserving ophthalmic signs is as follows:

[0049] (1) Create a training set;

[0050] (2) Train the SSGAN network model using the above training set;

[0051] (3) Export the trained optimal model;

[0052] (4) Create a test set;

[0053] (5) Use the network model to transform and enhance the test set to obtain the corresponding results.

[0054] The dataset used for training is a fundus image dataset, which can be either narrow-angle or wide-angle fundus images.

[0055] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0056] (1) The present invention can train the model using various unpaired low-quality and high-quality fundus image datasets, including wide-angle fundus images, which solves the problem that there is currently no method to enhance wide-angle fundus images and the difficulty in collecting low-quality and high-quality paired datasets of fundus images.

[0057] (2) The blood vessel segmentation loss and structural loss introduced in the generator and discriminator parts of the training conversion network in this invention can make the enhanced fundus image more accurately retain the morphological and structural details of fundus blood vessels, solve the problem of ignoring the structural features of fundus blood vessels and lesion areas in the prior art, better preserve the lesion morphology of the original image, enhance the original fundus image, and assist in clinical medical diagnosis.

[0058] (3) The model structure of this invention is clear and concise. It can enhance fundus images while achieving better preservation of fundus features. Through training, the SSGAN model outperforms existing models in both objective evaluation indicators and subjective evaluation results by ophthalmologists. It can effectively enhance fundus images and has broad application prospects. Attached Figure Description

[0059] Figure 1 This is a diagram showing the overall structure of the SSGAN model used in this invention.

[0060] Figure 2 This is a comparison chart of low-quality and high-quality fundus images in the dataset constructed during a specific implementation of the present invention. The first row shows low-quality fundus images, and the second row shows high-quality fundus images.

[0061] Figure 3 This image shows a comparison of the enhancement effects of the present invention and Cycle-GAN on normal fundus images. In the image, A represents the original image, B represents the image enhanced by Cycle-GAN, and C represents the result of the present technical solution.

[0062] Figure 4 This image shows a comparison of the enhancement effects of the present invention and Cycle-GAN on fundus images with lesions. In the image, A represents the original image, B represents the image enhanced by Cycle-GAN, and C represents the result of the present invention.

[0063] Figure 5 This image shows a comparison of the enhancement effects of the present invention and Cycle-GAN on the optic disc region of fundus images. In the image, A represents the original image, B represents the image enhanced by Cycle-GAN, and C represents the result of the present technical solution. Detailed Implementation

[0064] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0065] Figure 1 This paper illustrates the overall structure of the SSGAN model used in this invention, a fundus image enhancement method for preserving ophthalmic signs, employing the SSGAN fundus image enhancement network model based on CycleGAN. The SSGAN model includes two generators (G... A GB Two discriminators (D) A D B A transformation network consisting of a transformation network and a blood vessel segmentation network.

[0066] 1. The generator is used to transform the input image to the image domain corresponding to the generator index, such as G. A The input image is transformed to image domain A. The generator uses an encoder-decoder architecture with residual blocks. The encoder extracts features from the input image to obtain a feature map. The feature map is input into the residual network for transformation to obtain a new feature representation. Finally, the decoder reconstructs these features into a new image as the generator output. The encoder consists of three convolutional layers. The first layer uses a 7×7 convolutional kernel to perform a convolution operation with a stride of 1 on the image. The remaining two layers use 3×3 convolutional kernels to perform a convolution operation with a stride of 2 on the image. Each convolutional layer is followed by an instance normalization layer and a ReLU activation function, which respectively normalize the individual image and introduce nonlinear factors. The final output is a feature map with 256 channels. The decoder and encoder have symmetrical structures, both consisting of three convolutional layers. The first two layers use 3×3 kernels to perform transposed convolutions on the feature maps with a stride of 2. Each transposed convolution is followed by an instance normalization layer and a ReLU activation function. The last layer uses 7×7 kernels to perform convolutions with a stride of 1, restoring the features to an image with RGB channels. A tanh activation function is then used to amplify the feature effect and accelerate convergence. The residual network portion uses a residual network with nine convolutional layers to refine the features.

[0067] 2. The discriminator is used to determine whether the input image belongs to the image domain corresponding to the discriminator, in order to determine D. A For example, D A It needs to be able to distinguish whether the input image is a real image in the image domain A or a generator G. A The generated fake images are characterized by the use of PatchGANs.

[14] The structure allows images to be reduced to 70×70 matrix blocks through convolution operations. This matrix is ​​then used as the final evaluation criterion to determine whether the input image is a true image in the corresponding domain. This method requires fewer parameters compared to using the entire image as the evaluation criterion, and it can produce images with better results.

[0068] 3. The described vascular segmentation network is a pre-trained U-Net model on a fundus vascular image with segmentation annotations. It is used to segment vascular features from the input image. The segmentation result can be used to calculate the vascular segmentation loss and quantify the difference in vascular distribution between the original and enhanced images. The vascular segmentation network model consists of two symmetrical processes: downsampling and upsampling, each with five layers. In the downsampling process, each layer undergoes two convolutions and nonlinear transformations. The resulting image is then further downsampled using max pooling and cropped before being directly fed into the upsampling process via skip connections. In the upsampling process, each layer deconvolves the input feature map. The deconvolution result is then concatenated with the image obtained from the skip connections. The concatenated result undergoes two more convolutions to extract features. Finally, two 1×1 convolution kernels are used to divide the image's feature channels into two: the segmented target and the background, resulting in the final segmented image. The segmentation loss calculated using this output will be used to optimize the generation of the transformed and enhanced image, allowing the generator in the ring-shaped transform network to focus more on the vascular details of the fundus image, ensuring that the enhanced image retains the fundus vascular texture details of the original image well.

[0069] Specifically, the transformation network in the SSGAN model used in this invention consists of two generators and a discriminator. In addition to the basic transformation loss, the model also uses two new constraint functions, segmentation loss and structure loss, to solve the problem of insufficient constraints in the recurrent generative adversarial network, thereby maintaining the distribution of fundus blood vessels and other pathological signs during fundus image enhancement.

[0070] The transformation loss is a loss function used in the image generation and discrimination process of the transformation network. It consists of three parts: adversarial loss, cycle consistency loss, and identical mapping loss. The formula for the transformation loss is defined as follows:

[0071]

[0072] Where λ1 and λ2 are the weights of the cycle consistency loss and the identical mapping loss, usually λ1 = 10 and λ2 = 5.

[0073] Among them, combating losses It is the loss of the two sets of generators and discriminators in a game-like adversarial process during training, and its formula is defined as follows:

[0074]

[0075] The following uses generator G as an example. A Discriminator D A To illustrate the adversarial loss described above, consider the following example: Given an image b in domain B, the generator G... AWe need to generate an image G that is similar to the image in domain A. A (b), while the discriminator D A Then it is necessary to distinguish the image G that the generator forged. A (b) Distinguish between real image samples in image domain A and make a judgment on this. During training, the discriminator needs to maximize this loss function, while the generator needs to minimize it. They optimize each other in an adversarial process, i.e.:

[0076]

[0077] Among them, cycle consistency loss The formula used to constrain the similarity between the input image and the generated image is defined as follows:

[0078]

[0079] This can be divided into cycle consistency in domain A and cycle consistency in domain B. For domain A, each image a in domain A, after passing through generator G... B Transform into an image G in domain B B (a) After that, it goes through generator G A Transform back to domain A to get G A (G B When (a) is passed, the image should be consistent with image a. Similarly, for domain B, each image b in domain B, after passing through generator G... A Transform into an image G in domain A A (b) After that, it goes through generator G B Transform back to domain B to get G B (G A (b) In this case, the image should be consistent with image b.

[0080] The congruence mapping loss Similarly used to constrain the generator to preserve the features of the original image, unlike the cycle consistency loss, the equal mapping loss constrains each generator individually, and its formula is defined as follows:

[0081]

[0082] The congruence mapping loss constrains the generator to generate an image as similar as possible to the input image when the received input image is already a true sample in the corresponding image domain. For example, for generator G... A For example, when the input image is image a in domain A, the image G that passes through... A (a) should be as consistent as possible with image a.

[0083] To address the issues that existing technologies cannot maintain the original morphology of fundus signs in the original image, resulting in lesion degeneration, blurred optic disc area, and even loss of vascular information, this invention also uses structural loss and segmentation loss to strengthen the constraints on the generator.

[0084] (1) Structural loss

[0085] Ideally, the enhanced fundus image should exhibit the same pathological features as the original image and be clearer, facilitating ophthalmologist observation and diagnosis. Therefore, constraints related to image structure should be added to the generator. Structural Similarity (SSIM) is one such constraint.

[15] Structural similarity is a commonly used image evaluation metric that can reasonably assess the similarity between two images from three perspectives: brightness, contrast, and global structure. Compared to mean squared error (MSE), structural similarity better reflects the structural differences between two images and is closer to how the human visual system measures the difference between two images. Therefore, this invention proposes structural loss based on structural similarity to quantify the structural difference between the original and enhanced images, ensuring that the overall structural details of the original image are better preserved in the enhanced image. The formula for structural loss is defined as follows:

[0086]

[0087] In the above formula, SSIM represents the calculation of structural similarity, and its calculation formula is as follows:

[0088]

[0089] Where, μ x With μ y These are the mean values ​​of the x and y values ​​of the image, respectively, and σ is the mean value of the image. xy The covariance of the images x and y and These are the variances of the images x and y, respectively. The constant term C1 = (K1L) 2 C2 = (K2L) 2 Let K1 = 0.01, K2 = 0.03, and L be the dynamic range of grayscale. For the normalized image, let L = 1.

[0090] (2) Segmentation loss

[0091] The segmentation loss is a loss function defined using the segmentation results output by the Vessel Segmentation Network (VSN). It is used to constrain the consistency of vessel segmentation results in fundus images before and after enhancement / degradation. The formula for the segmentation loss is defined as follows:

[0092]

[0093] The Vessel Segmentation Network (VSN) is a pre-trained network with fixed parameters, similar to a vessel feature extractor. This network can be used during the training of a transformation network to extract vessel features from both the original and transformed images and output corresponding vessel segmentation results. The SSIM segmentation loss uses a sliding window calculation method, quantifying the generator's preservation of vessel features by calculating the structural similarity between the segmentation results of the original and transformed images. This ensures that the generated image contains vessel features consistent with the original image.

[0094] Therefore, the final objective function of the model used in this invention is:

[0095]

[0096] Here, α and β are the weights of the structural loss and the segmentation loss, respectively. Typically, α = 5 and β = 100.

[0097] Figure 2 This invention provides a comparison of low-quality and high-quality fundus images within the constructed dataset. Figure 3-5 The images shown here are comparisons between images enhanced using the model trained in this invention and those enhanced using Cycle-GAN. The enhancement effects are demonstrated for normal fundus images, fundus images with lesions, and the optic disc region in fundus images.

[0098] like Figure 3 As shown, Cycle-GAN can remove some of the eyelid and eyelash artifacts, but a black hole appeared on the left side of the optic disc, and the optic disc area also showed morphological changes and unclear vascular patterns. The overall color was reddish. The image enhanced by the SSGAN model proposed in this technical solution effectively removed the artifacts of the imaging instrument and weakened the occlusion of eyelids and eyelashes. It enhanced the details of vascular patterns in the center and around the optic disc, and the small blood vessels at the edge of the image were also clearly displayed. It better preserved the pathological signs of the original image, and the overall color was uniform.

[0099] like Figure 4 As shown, the image reveals vitrectomyctic lesions in the optic disc region. Cycle-GAN-enhanced images cannot effectively preserve the morphology of the optic disc and surrounding areas, resulting in unclear optic disc boundaries. The images enhanced by the SSGAN model proposed in this invention show clear optic disc boundaries, reveal obvious vitrectomyctic lesions, and clearly display the details of vascular crossings within and around the optic disc, which is more conducive to clinical diagnosis.

[0100] like Figure 5As shown, the optic disc is a crucial structure in the fundus. All optic nerves in the retina must pass through the optic disc to connect to the visual center in the brain. The blood vessels in the fundus also mainly originate from the optic disc, making it an important object of observation during medical diagnosis. In Cycle-GAN-enhanced images, the optic disc area suffers from overexposure and blurred central optic disc vessels. The image enhanced by the SSGAN model proposed in this technical solution is smooth overall, with clear optic disc boundaries and enhanced vessels in the center and surrounding areas of the optic disc. The visibility of the central optic disc vessels is higher than in the original image, and other fundus features outside the optic disc are not significantly distorted. From a clinical diagnostic perspective, this provides the best visual effect.

[0101] This invention addresses the problems of neglecting fundus pathological signs and vascular morphological details in some traditional methods and deep learning-based fundus image enhancement methods. It proposes a fundus image enhancement method based on recurrent generative adversarial networks (RGANs). By introducing two new constraint functions, segmentation loss and structural loss, the problem of insufficient constraints in RGANs is solved. This method preserves the distribution of fundus vessels and other pathological signs during fundus image enhancement, resulting in well-preserved ophthalmic signs in the enhanced fundus image, making it more suitable for clinical medical diagnosis.

[0102] The above descriptions are preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0103] References:

[0104] [1]Hsu, Wei-Yen, and Ching-Yao Chou. "Medical image enhancement using modified color histogram equalization." Journal of Medical and BiologicalEngineering 35.5(2015):580-584.

[0105] [2] M.Abdullah-Al-Wadud, MHKabir, MAADewan, and O.Chae, "A dynamichistogram equalization for image contrast enhancement," IEEETrans.Consum.Electron., vol.53, no.2, pp.593–600, May 2007.

[0106] [3]K.Zuiderveld,“Contrast limited adaptive histogram equalization,”inGraphics Gems IV.New York,NY,USA:Academic,1994,pp.474–485.

[0107] [4]Jintasuttisak,Thani,and SathitIntajag."Color retinal imageenhancement by Rayleigh contrast-limited adaptive histogram equalization."2014 14th International Conference on Control,Automation and Systems(ICCAS2014).IEEE,2014.

[0108] [5]E.Daniel and J.Anitha,"Retinal image enhancement using waveletdomain edge filtering and scaling,"2014 International Conference onElectronics and Communication Systems(ICECS),2014,pp.1-6,doi:10.1109 / ECS.2014.6892670.

[0109] [6]Swaminathan,Anand,et al."Contourlet transform-based sharpeningenhancement of retinal images and vessel extraction application."Biomedizinische Technik / Biomedical Engineering 58.1(2013):87-96.

[0110] [7]Lee,Chien-Cheng,et al."Enhancement of blood vessels in retinalimaging using the nonsubsampled contourlet transform."MultidimensionalSystems and Signal Processing 23.4(2012):423-436.

[0111] [8]D.J.Jobson,Z.-U.Rahman,and G.A.Woodell,“A multiscale retinex forbridging the gap between color images and the human observation of scenes,”IEEE Trans.Image Process.,vol.6,no.7,pp.965–976,Jul.1997.

[0112] [9]Vázquez,S.G.,et al."Using retinex image enhancement to improve theartery / vein classification in retinal images."International Conference ImageAnalysis and Recognition.Springer,Berlin,Heidelberg,2010.

[0113]

[10] Zhu J Y,Park T,Isola P,et al.Unpaired Image-to-Image Translationusing Cycle-Consistent Adversarial Networks[J].IEEE,2017.

[0114]

[11] Yoo,Tae Keun,Joon Yul Choi,and Hong Kyu Kim."CycleGAN-based deeplearning technique for artifact reduction in fundus photography."Graefe'sArchive for Clinical and Experimental Ophthalmology 258.8(2020):1631-1637.

[0115]

[12] Wan C,Zhou X,You Q,Sun J,Shen J,Zhu S,Jiang Q,Yang W.RetinalImage Enhancement Using Cycle-Constraint Adversarial Network.Front Med(Lausanne).2022 Jan 12;8:793726.doi:10.3389 / fmed.2021.793726.PMID:35096883;PMCID:PMC8789669.

[0116]

[13] Ma Y,Liu J,Liu Y,et al.Structure and Illumination Constrained GANfor Medical Image Enhancement[J].IEEE Transactions on Medical Imaging,2021,PP.

[0117]

[14] Li C,Wand M.Precomputed real-time texture synthesis withmarkovian generative adversarial networks.In:Leibe B,Matas J,Sebe N,WellingM,editors.Computer Vision-ECCV 2016.Amsterdam:Springer InternationalPublishing(2016).p.702–716.

[0118]

[15] Z.Wang,A.C.Bovik,H.R.Sheikh,and E.P.Simoncelli,“Image qualityassessment:From error visibility to structural similarity,”IEEE Trans.ImageProcess.,vol.13,no.4,pp.600–612,Apr.2004。

Claims

1. A method for enhancing fundus images while preserving ophthalmic signs, characterized in that, An image augmentation network model based on CycleGAN, denoted as SSGAN, is used. The SSGAN model consists of a blood vessel segmentation network and two generators. Two discriminators The conversion enhancement network consists of: The generator This is used to transform the input image to the image domain corresponding to the generator index. Transform the input image to image domain A, generator The input image is transformed to the image domain B; each generator uses an encoder-decoder architecture with residual blocks. The encoder extracts features from the input image to obtain a feature map, which is then input into the residual block for transformation to obtain a new feature representation. Finally, the decoder reconstructs these features into a new image, which is then output as the generator. The discriminator It is used to determine whether the input image belongs to the image domain corresponding to the discriminator, so that the discriminator... For example, It needs to be able to distinguish whether the input image is a real image in image domain A or a generator image. The generated fake images, too, It needs to be able to distinguish whether the input image is a real image in image domain B or a generator image. The generated fake image; The blood vessel segmentation network is a U-Net model pre-trained on fundus blood vessel images with segmentation annotations, used to segment vascular features from the input image; the segmentation results can be used to calculate blood vessel segmentation loss and quantify the difference in blood vessel distribution between the original image and the enhanced image. The blood vessel segmentation network model consists of two symmetrical processes: downsampling and upsampling; each process has five layers. In the downsampling process, each layer undergoes two convolutions and nonlinear transformations. The results are then downsampled using max pooling and cropped and copied before being directly fed into the upsampling process via skip connections. In each layer of the upsampling process, the input feature map is deconvolved. The deconvolution result is then concatenated with the image obtained from the skip connections. The concatenated result is then convolved twice more to extract features. Finally, two... The convolutional kernel of a certain size divides the feature channels of the image into two, namely the segmented target and the background, to obtain the final output segmented image. The segmentation loss calculated using this output is used to optimize the generation of the transformation and enhancement image, so that the generator in the ring transformation network can pay more attention to the vascular details of the fundus image, ensuring that the enhanced image retains the fundus vascular texture details of the original image well. The transformation network consists of two generators and a discriminator; during training, the losses used include: transformation loss, segmentation loss, and structural loss; The structural loss is imposed on the generator by adding constraints related to image structure. Based on structural similarity (SSIM), the structural loss is given to quantify the structural difference between the original image and the enhanced image, ensuring that the overall structural details of the original image can be better preserved in the enhanced image. The structural loss is defined as follows: ，（5） In the above formula, The formula for calculating structural similarity is as follows: ， （6） in, and These are images and The mean, For image and covariance, and These are images and The variance; where the constant term , , To adjust parameters, To determine the dynamic range of grayscale, take the following values ​​for the normalized image: ; The segmentation loss is obtained using a blood vessel segmentation network ( The loss function defined by the output segmentation results is used to constrain the consistency of vascular segmentation results in fundus images before and after enhancement / degradation. The segmentation loss is defined as follows: ， （7） Vascular segmentation network ( This is a pre-trained network with fixed parameters, similar to a blood vessel feature extractor. This network is used during the training of the transformation network to extract blood vessel features from the original and transformed images and output the corresponding blood vessel segmentation results; the segmentation loss... Using a sliding window calculation method, the degree to which the generator preserves vascular features is quantified by calculating the structural similarity between the vascular segmentation results of the original image and the transformed image. This can encourage the generated image to contain vascular features consistent with the original image. Therefore, the final objective function of the adopted model is: ，（8） in, and These are the weights for structural loss and segmentation loss.

2. The method for enhancing fundus images while preserving ophthalmic signs according to claim 1, characterized in that, The discriminator middle: The encoder consists of three convolutional layers, with the first layer using... The convolution kernel performs a convolution operation on the image with a stride of 1, and the remaining two layers use... The convolution kernel performs a convolution operation on the image with a stride of 2; each convolutional layer is followed by an instance normalization layer and a ReLU activation function, which respectively normalize the individual image and introduce non-linear factors; the final output is a feature map with 256 channels. The decoder and encoder have symmetrical structures, both consisting of three convolutional layers, with the first two layers using... The convolution kernel performs a transpose convolution operation on the feature map with a stride of 2. After each transpose convolution, an instance normalization layer and a ReLU activation function are connected. The last layer uses... The convolution kernel is used to perform a convolution with a stride of 1, which restores the features to an image with three RGB channels, and the tanh activation function is used to amplify the feature effect and speed up the convergence speed. The residual block uses a residual network with 9 convolutional layers to process the features.

3. The method for enhancing fundus images while preserving ophthalmic signs according to claim 2, characterized in that, The discriminator uses a PatchGANs structure and reduces the image size through convolution operations. A matrix of a certain size is used as the final evaluation criterion to determine whether the input image is a true image in the corresponding domain.

4. The method for enhancing fundus images while preserving ophthalmic signs according to claim 1, characterized in that, The transformation loss is a loss function used in the process of image generation and discrimination by the transformation network. It consists of three parts: adversarial loss, cycle consistency loss, and identical mapping loss. The transformation loss is defined as follows: ，（1） in, and where represents the weights of the cycle consistency loss and the equality mapping loss; (1.1) Combating Losses It is the loss resulting from the adversarial game between the two sets of generators and discriminators during training, and is defined as follows: ， （2） Its meaning is: the image in the given domain B generator We need to generate an image similar to the image in domain A. The discriminator Then it is necessary to distinguish the images forged by the generator. Distinguish between real image samples in image domain A and make a judgment accordingly; similarly, given images in domain A... generator We need to generate an image similar to the image in domain B. The discriminator Then it is necessary to distinguish the images forged by the generator. Image domain To distinguish between real image samples and make a judgment on this, Indicates the expected value; During training, the discriminator needs to maximize this loss function, while the generator needs to minimize it. They optimize each other in this adversarial process. ； (1.2) Cyclic consistency loss Used to constrain the similarity between the input image and the generated image of the generator, specifically defined as follows: ， （3） It is divided into cycle consistency in domain A and cycle consistency in domain B. For domain A, each image in domain A... After passing through the generator Convert to image in domain B Then, it goes through the generator. Transform back to domain A to get At that time, the image should be the same as the image. Maintain consistency; similarly, for domain B, the images in each domain B... After passing through the generator Convert to an image in domain A Then, it goes through the generator. Transform back to domain B to get At that time, the image should be the same as the image. Maintain consistency; (1.3) Equivalent mapping loss Also used to constrain the generator to preserve the features of the original image, unlike the cycle consistency loss, the equal mapping loss constrains each generator individually, and its definition is as follows: ， （4） When the identical mapping loss constraint generator receives an input image that is already a real sample in the corresponding image domain, the generated image should be as similar as possible to the input image.

5. The method for enhancing fundus images while preserving ophthalmic signs according to claim 4, characterized in that, The specific process is as follows: (1) Create a training set; (2) Train the SSGAN network model using the above training set; (3) Export the trained optimal model; (4) Create a test set; (5) Use the network model to transform and enhance the test set to obtain the corresponding results.

6. The method for enhancing fundus images while preserving ophthalmic signs according to claim 5, characterized in that, The dataset used for training is a fundus image dataset, including narrow-angle fundus images or wide-angle fundus images.

Images