Cataract eye fundus image enhancement method based on weak supervised learning

Abstract

The invention provides a two-stage multi-scale attention cataract eye fundus image enhancement method based on weak supervised learning, a similar real cataract image synthesis stage is provided, domain transformation is carried out by using a CycleGAN network to synthesize paired similar real cataract images, the problem that a large amount of paired data is difficult to obtain is solved, and through the learning of the network, the real cataract eye fundus image enhancement efficiency is improved. And more degradation factors can be covered. In order to further extract fundus image details and reduce generation of artifacts in the enhancement process, a multi-scale attention cataract fundus enhancement stage is provided to learn more useful cataract image enhancement features, and application in visual perception tasks is facilitated.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and medical artificial intelligence, and relates to image restoration technology, specifically a cataract fundus image enhancement method based on weakly supervised learning. Background Art

[0002] Different from natural image enhancement, fundus image enhancement algorithms need to better preserve the structural details of the image to enhance the accuracy of subsequent diagnosis. Therefore, the enhancement algorithms for natural images cannot be directly transferred to fundus image enhancement tasks. In view of this, researchers have gradually begun to pay attention to fundus image enhancement tasks. Few-sample learning (FSL) has received increasing attention in this field because it can obtain image category information from relatively few samples.

[0003] Fundus image enhancement is the basis of visual perception and serves a series of high-level semantic understanding tasks, such as segmentation and detection. This technology focuses on restoring the degraded information in the captured image and generating images as close to the real world as possible to meet the application requirements of high-level semantic understanding tasks. By comparing the cataract image and the fundus image enhanced by the network model of the present invention, it can be seen that the retina of cataract patients is relatively blurred and the image readability is poor. After enhancement, the image contains richer information and retains the image structure, which is conducive to the application in further high-level semantic understanding tasks.

[0004] Current cataract fundus image enhancement methods usually use supervised learning methods to restore image details. However, it is very time-consuming and laborious to obtain paired cataract retinal images. Although the artificial synthesis method can obtain a large number of paired images, it is impossible to cover all the degradation factors of cataract images using only the synthesis function, which makes the network's generalization ability insufficient in real cataract degradation scenes and limits its application in further high-level semantic understanding tasks. Due to the lack of supervision constraints, unsupervised learning methods are prone to lose image details during the enhancement process. Summary of the invention

[0005] In view of the above problems, the present invention proposes a two-stage multi-scale attention cataract fundus image enhancement method based on weakly supervised learning. By proposing a realistic cataract image synthesis stage, the CycleGAN network is used for domain conversion to synthesize paired realistic cataract images, which solves the problem of difficulty in obtaining a large amount of paired data. Moreover, through network learning, more degradation factors can be covered. In order to further extract fundus image details and reduce the generation of artifacts during the enhancement process, a multi-scale attention cataract fundus enhancement stage is proposed to learn more useful cataract image enhancement features, which is helpful for application in visual perception tasks.

[0006] To achieve the above object, the technical solution of the present invention is:

[0007] A cataract fundus image enhancement method based on weakly supervised learning, the method comprising the following steps:

[0008] Step 1: Use cataract fundus image simulation to process the clear fundus image to obtain a simulated cataract-like fundus synthetic image; the cataract detection panel first performs Euclidean distance transformation on the input synthetic image and the real cataract image, and then processes the result through Gaussian filtering;

[0009] Step 2: Perform bidirectional domain conversion on the synthetic image and the real cataract image through the CycleGAN network to obtain a real-like cataract image;

[0010] Step 3: Use the high-frequency extraction module H(·) to extract the high-frequency component information of the real cataract image;

[0011] Step 4: Input the high-frequency component information extracted in step 3 into the multi-scale attention module of the cataract fundus image enhancement network, so that the network model can learn more useful cataract image enhancement features, and train the cataract fundus image enhancement network with the mean square error loss, color loss, total discrete pixel change loss, and structural consistency loss of the real cataract image as the total loss;

[0012] Step 5: Input the cataract fundus image to be enhanced into the trained network to obtain the corresponding enhanced cataract fundus image.

[0013] Based on the above scheme, this method is suitable for cataract fundus image enhancement to solve the problem that the existing methods ignore the semantic understanding task requirements in visual perception, resulting in limited application in semantic understanding tasks. Cataract-like fundus images are simulated and processed, and then the CycleGAN network is used for bidirectional domain conversion to generate images that are closer to the real cataract situation. The high-frequency component information of the real cataract image is extracted by the high-frequency extraction module and input into the multi-scale attention module, which can prompt the network model to focus on the key features in the image, especially the high-frequency details related to cataracts; the multi-scale attention module can further enhance the network's ability to capture features of different scales, thereby learning more useful cataract image enhancement features. A comprehensive loss function including mean square error loss, color loss, total discrete pixel change loss and structural consistency loss is used to train the cataract fundus image enhancement network, which not only focuses on pixel-level differences, but also takes into account the color fidelity, structural similarity and overall visual effects of the image, thereby ensuring that the enhanced image reaches a better state in multiple dimensions. This method uses weakly supervised learning, which has lower requirements for labeled data than fully supervised learning and can be more flexibly applied to practical scenarios. This is particularly important for fields such as cataract fundus images, where it is difficult to obtain large amounts of high-quality labeled data, and helps reduce data labeling costs and improve the practicality and scalability of the method.

[0014] Furthermore, the clear fundus image is processed by cataract fundus image simulation in step 1 to obtain a simulated cataract-like fundus synthetic image C(s c ), which is specifically expressed by the following formula:

[0015] C(s c )=α·s c *g B (r B ,σ B )+β·J*g L (r L ,σ L )·(L c -s c ) (1)

[0016] Among them, C(s c ) represents a simulated cataract-like fundus synthetic image; s c represents a clear fundus image, where c represents the r, g, and b channels of the image; α and β represent the weight of the clear fundus image and the noise from the cataract, respectively; * represents the convolution operation; g B and g L denote the Gaussian filters for clear fundus image smoothing and cataract detection panel, respectively; g(r,σ) denotes a Gaussian filter with r as radius and σ as space constant; J denotes cataract detection panel, Lc Indicates c Maximum strength.

[0017] Furthermore, step 2 is specifically as follows: through two GAN networks, respectively learning the mapping from the synthetic image domain s to the real cataract image domain r and the mapping from the real cataract image domain r to the synthetic image domain s.

[0018] Furthermore, in step 3, the high-frequency extraction module is used to extract the high-frequency component information of the real cataract image. Specifically, the high-frequency extraction module g(·) first extracts the low-frequency component of the real cataract image through a low-pass Gaussian filter, and then subtracts the low-frequency component from the real cataract image to obtain the high-frequency component of the real cataract image. The process of the high-frequency extraction module H(·) is expressed by the following formula:

[0019] H(I)=II*g p (r p ,σ p ) (2)

[0020] Where I represents the real cataract image, g p (r p ,σ p ) represents a p is the radius, σ p is a Gaussian filter with a spatial constant.

[0021] Furthermore, the learning process of the multi-scale attention module in step 4 can be expressed as:

[0022] M i =F i +f MSAB (LN(F i )) (3)

[0023] F i+1 =M i +f GDFB (LN(M i )) (4)

[0024] F i and F i+1 Represent the input and output features of the multi-scale attention module, M i represents the extracted multi-scale features, f MSAB (·) indicates MSAB module, f GDFB (·) denotes the GDFB module, and LN(·) denotes the layer normalization operation.

[0025] Furthermore, in step 4, the mean square error loss, color loss, total discrete pixel change loss, and structural consistency loss of the real cataract image are used as the total loss to train the cataract fundus image enhancement network, specifically including:

[0026] Step 4.1: In the stage of synthesizing realistic cataract images, the loss function in the CycleGAN network is used to train the stage of synthesizing realistic cataract images. The loss function is expressed as L CycleGAN , the calculation process is shown as follows:

[0027]

[0028] Among them, S and R represent the source domain and target domain respectively. represents the mapping function from the S domain to the R domain, represents the mapping function from R domain to S domain; D S and D R represent the discriminators of the source domain and the target domain respectively, L GAN and L cyc They represent adversarial loss and cycle consistency loss respectively, and λ is the weight;

[0029] Step 4.2: To maintain the structural information of the fundus, the structural loss L in GFENet is also used R , the total loss function is expressed as L total , the calculation process is shown as follows:

[0030]

[0031] Among them, S G Represents the structure diagram of the source domain; R G A structural diagram representing the target domain; Represents the true value of the source domain structure graph; Represents the true value of the target domain structure graph;

[0032] Step 4.3: In the cataract image enhancement stage of the multi-scale attention module, the mean square error loss L is used MSE To constrain the structural changes of the image, the color loss L color To constrain the color brightness change of the image, the total discrete pixel change loss L TV To constrain the edge of the image, the structural consistency loss L SSIM Restore the brightness and structure of the image; the sum of the losses in this step is used as the total loss function, and the total loss function can be expressed as:

[0033]

[0034] Among them, rgb represents r, g, b channels, maxc rgbRepresents the maximum value of the three RGB channels in the predicted image; c represents the predicted image; y represents the real image; structural similarity index

[0035] The beneficial effects of the present invention include:

[0036] The present invention proposes a cataract fundus image enhancement method based on weakly supervised learning, including a two-stage network, which is used to generate realistic cataract images and cataract image enhancement, respectively. In the first stage of the network, CycleGAN is used to perform domain conversion to synthesize paired realistic cataract images. In addition, in order to further extract the detail information of the fundus image and reduce the generation of artifacts during the image enhancement process, a multi-scale attention module is proposed in the second stage of the network to learn more useful cataract image enhancement features, which is helpful for application in visual perception tasks. Experimental results verify that the method has achieved good results in the three perception tasks of cataract fundus image enhancement, fundus vascular segmentation and fundus disease detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an overall flow chart of a cataract fundus image enhancement method based on weakly supervised learning;

[0038] Figure 2 Schematic diagram of the multi-level feature processing flow in the network. DETAILED DESCRIPTION

[0039] The embodiments of the present invention are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0040] The present invention provides a cataract fundus image enhancement method based on weakly supervised learning, which aims to enhance the image restoration performance by extracting richer information from multi-level features. Specifically, in order to retain key feature information, a realistic cataract image synthesis stage is proposed, which uses a CycleGAN network for domain conversion to synthesize paired realistic cataract fundus image synthesis stages. In order to further extract fundus image details and reduce the generation of artifacts during the enhancement process, a multi-scale attention cataract fundus enhancement stage is introduced to learn more useful cataract image enhancement features, which is helpful for application in visual perception tasks.

[0041] Example 1

[0042] A cataract fundus image enhancement method based on weakly supervised learning, the specific steps are:

[0043] Step 1: Use the cataract fundus image simulation synthesis function C(·) to process the clear fundus image and generate a simulated cataract-like fundus synthesis image C(s c ) to shorten the domain distance with the real cataract fundus image. It is specifically expressed by the following formula:

[0044] C(s c )=α·s c *g B (r B ,σ B )+β·J*g L (r L ,σ L )·(L c -s c ) (1)

[0045] Among them, C(s c ) represents a simulated cataract-like fundus synthetic image; s c represents a clear fundus image, where c represents the r, g, and b channels of the image; α and β represent the weight of the clear fundus image and the noise from the cataract, respectively; * represents the convolution operation; g B and g L denote the Gaussian filters for clear fundus image smoothing and cataract detection panel, respectively; g(r,σ) denotes a Gaussian filter with r as radius and σ as space constant; J denotes cataract detection panel, L c Indicates c In this embodiment, the value of α is set to 0.8, and the value of β is randomly selected between 0.6 and 0.8. For a clear image, r B The value of is randomly selected from 1, 3, 5, and 7, σ B The value of is randomly selected as an integer between 10 and 30; r L The value of is randomly selected from 1, 3, 5, and 7; L The value of is randomly selected as an integer between 10 and 30;

[0046] For the cataract detection panel, the input synthetic image and the real cataract image are first subjected to Euclidean distance transformation, and then the result is processed by Gaussian filtering.

[0047] Step 2: Perform bidirectional domain conversion on the synthetic image and the real cataract image through the CycleGAN network to further simulate the cataract fundus image that is closer to the real degeneration and obtain the real cataract image; specifically: through two GAN networks, learn the mapping from the synthetic image domain s to the real cataract image domain r and the mapping from the real cataract image domain r to the synthetic image domain s respectively.

[0048] Step 3: According to the Retinex theory and cataract imaging model, the high-frequency extraction module H(·) is used to extract rich structural information of the realistic cataract image; according to the retinal theory, the light reflected by the cataract is a low-frequency component in the fundus image, while the light reflected by the retina is a high-frequency component in the fundus image. Therefore, the high-frequency component in the fundus image is more robust to cataracts and contains a large number of domain-invariant structural features. Specifically: the high-frequency extraction module first extracts the low-frequency component of the fundus image through a low-pass Gaussian filter, and then subtracts the low-frequency component from the realistic cataract image to obtain the high-frequency component of the realistic cataract image as the input for the next stage; the process of the high-frequency extraction module H(·) is expressed by the following formula:

[0049] H(I)=II*g p (r p ,σ p ) (2)

[0050] Where I represents the real cataract image, g p (r p ,σ p ) represents a p is the radius, σ p is a Gaussian filter with a spatial constant.

[0051] Step 4: Input the high-frequency component information extracted in step 3 into the multi-scale attention module of the cataract fundus image enhancement network, so that the network model can learn more useful cataract image enhancement features, and train the cataract fundus image enhancement network with the mean square error loss, color loss, total discrete pixel change loss, and structural consistency loss of the real cataract image as the total loss;

[0052] The multi-scale attention module can make more full use of the features of the real cataract images synthesized in the previous stage and learn richer fundus image features. The learning process can be expressed as:

[0053] M i =F i +f MSAB (LN(F i )) (3)

[0054] F i+1 =M i +f GDFB (LN(M i )) (4)

[0055] F i and F i+1 Represent the input and output features of the multi-scale attention module, M i represents the extracted multi-scale features, f MSAB(·) indicates MSAB module, f GDFB (·) denotes the GDFB module and LN(·) denotes the layer normalization operation.

[0056] In the stage of synthesizing realistic cataract images, the loss function in the CycleGAN network is used to train the realistic cataract image synthesis stage. The loss function can be expressed as L CycleGAN , the calculation process is shown as follows:

[0057]

[0058] Among them, S and R represent the source domain and target domain respectively. represents the mapping function from the S domain to the R domain, represents the mapping function from R domain to S domain; D S and D R represent the discriminators of the source domain and the target domain respectively, L GAN and L cyc They represent adversarial loss and cycle consistency loss respectively, and λ is the weight;

[0059] In addition, in order to maintain the structural information of the fundus, the structural loss L in GFENet is also used R The total loss function is expressed as L total , the calculation process is shown as follows:

[0060]

[0061] Among them, S G Represents the structure diagram of the source domain; R G A structural diagram representing the target domain; Represents the true value of the source domain structure graph; Represents the true value of the target domain structure graph;

[0062] In the multi-scale attention module cataract fundus image enhancement stage, in order to maintain the content information of the fundus image, retain the image structure, color brightness and other information during the restoration process, and reduce the artifacts generated during the restoration process, the mean square error loss L is used. MSE To constrain the structural changes of the image, the color loss L color To constrain the color brightness change of the image, the total discrete pixel change loss L TV To constrain the edge of the image, generate a smoother image, reduce the generation of image artifacts, and reduce the structural consistency loss L SSIM Restore the brightness and structure of the image. The sum of the losses in this step is used as the total loss function, and the total loss function is expressed as:

[0063] L multi =L TVΔ(cy)+L MSE ||cy||2+L Color ||maxc rgb -maxy rgb ||+L SSIM ssim((c,y)) (7)

[0064] Among them, λ represents the hyperparameter, rgb represents r, g, b channels, maxc rgb represents the maximum value of the three RGB channels in the predicted image; c represents the predicted image; y represents the real image; the structural similarity index ssim = 1-ssim.

[0065] Step 5: Load the model in step 4 and apply it to restore the fundus image of the unseen cataract. Input the cataract fundus image to be enhanced into the trained network to obtain the corresponding enhanced cataract fundus image.

[0066] Example 2

[0067] This embodiment uses Windows system as the development environment, Pycharm as the development platform, Python as the development language, and adopts a cataract fundus image enhancement network based on weakly supervised learning of the present invention to complete image enhancement for cataract fundus images. It includes the following steps:

[0068] Step 1: Divide the dataset into a query set and a support set, preprocess the original data in the dataset, and construct paired support-query image pairs as training data;

[0069] Step 2: Construct a cataract fundus image enhancement network based on weakly supervised learning, and input the preprocessed training data into the cataract fundus image enhancement network (such as Figure 1 shown);

[0070] Step 3: Use mean square error loss to constrain image structure changes, use color loss to constrain image color brightness changes, use total discrete pixel change loss to constrain image edges, generate smoother images, reduce image artifacts, and use structural consistency loss to restore image brightness and structure. The above-mentioned sums are used as the total loss function for training to obtain the final trained cataract fundus image enhancement network;

[0071] Step 4: Take the cataract fundus image to be enhanced as input, load the model saved after training in step 3, and obtain the enhanced image corresponding to the input cataract fundus image and the corresponding evaluation index.

[0072] This example uses the Natural Image Quality Evaluation (NIQE) method as the evaluation metric. This is a no-reference image quality score metric that calculates image A. The smaller the score, the better the perceived quality.

[0073] According to the above steps, the present invention is compared with the CycleGAN model, CofeNet model, EnlightenGAN model, ArcNet model, PCENet model and GFENet model on two different test sets (Retina and ODIR-5k) under the experimental setting. As can be seen from Table 1, the performance of the method proposed in the present invention on these two common test sets is basically better than other methods.

[0074] Table 1 Quantitative comparison on Retina and ODIR-5K datasets. The best results are marked in bold.

[0075]

[0076] The present invention proposes a cataract fundus image enhancement method based on weakly supervised learning, aiming to solve the problem that the existing methods are limited in application in such tasks due to ignoring the semantic understanding task requirements in visual perception. An innovative stage of quasi-realistic cataract image synthesis is introduced, and a bidirectional domain conversion network is used to synthesize cataract fundus images close to reality, thereby addressing the challenge that artificially synthesized paired images are difficult to fully simulate real-world degradation factors, thereby affecting the generalization performance of the model. In addition, in order to more finely extract the detailed features of the fundus image and reduce the image artifacts that may occur during the enhancement process, the present invention designs a cataract fundus image enhancement module with a multi-scale attention mechanism. This design aims to deeply learn and utilize the key features of the fundus image to enhance its expressiveness and accuracy in visual perception tasks. The effectiveness of the present invention has been verified in three key visual perception tasks: cataract fundus image enhancement, fundus vascular segmentation, and fundus disease detection, proving the potential and value of the invention in improving fundus image processing and analysis.

[0077] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A cataract fundus image enhancement method based on weakly supervised learning, characterized in that: The method comprises the following steps: Step 1: Use cataract fundus image simulation to process the clear fundus image to obtain a simulated cataract-like fundus synthetic image; the cataract detection panel first performs Euclidean distance transformation on the input synthetic image and the real cataract image, and then processes the result through Gaussian filtering; Step 2: Perform bidirectional domain conversion on the synthetic image and the real cataract image through the CycleGAN network to obtain a real-like cataract image; Step 3: Use the high-frequency extraction module H(·) to extract the high-frequency component information of the real cataract image; Step 4: Input the high-frequency component information extracted in step 3 into the multi-scale attention module of the cataract fundus image enhancement network, so that the network model can learn more useful cataract image enhancement features, and train the cataract fundus image enhancement network with the mean square error loss, color loss, total discrete pixel change loss, and structural consistency loss of the real cataract image as the total loss; Step 5: Input the cataract fundus image to be enhanced into the trained network to obtain the corresponding enhanced cataract fundus image.

2. The method for enhancing cataract fundus images based on weakly supervised learning according to claim 1, characterized in that: In step 1, the clear fundus image is processed by cataract fundus image simulation to obtain a simulated cataract-like fundus synthetic image C(s c ), which is specifically expressed by the following formula: C(s c )=α·s c *g B (r B ,s B )+β·J*g L (r L ,s L )·(L c -s c ) (1) Among them, C(s c ) represents a simulated cataract-like fundus synthetic image; s c represents a clear fundus image, where c represents the r, g, and b channels of the image; α and β represent the weight of the clear fundus image and the noise from the cataract, respectively; * represents the convolution operation; g B and g L denote the Gaussian filters for clear fundus image smoothing and cataract detection panel, respectively; g(r, v) denotes a Gaussian filter with r as radius and σ as space constant; J denotes cataract detection panel, L c Indicates c Maximum strength.

3. A cataract fundus image enhancement method based on weakly supervised learning according to claim 1 or 2, characterized in that: Step 2 is specifically as follows: through two GAN networks, respectively learn the mapping from the synthetic image domain s to the real cataract image domain r and the mapping from the real cataract image domain r to the synthetic image domain s.

4. The method for enhancing cataract fundus images based on weakly supervised learning according to claim 3, characterized in that: Step 3 uses a high-frequency extraction module to extract high-frequency component information of the real cataract image. Specifically, the high-frequency extraction module H(·) first extracts the low-frequency component of the real cataract image through a low-pass Gaussian filter, and then subtracts the low-frequency component from the real cataract image to obtain the high-frequency component of the real cataract image. The process of the high-frequency extraction module H(·) is expressed by the following formula: H(I)=I-I*g p (r p ,σ p ) (2) Where I represents the real cataract image, g p (r p , σ p ) represents a p is the radius, σ p is a Gaussian filter with a spatial constant.

5. The method for enhancing cataract fundus images based on weakly supervised learning according to claim 1, characterized in that: The learning process of the multi-scale attention module in step 4 can be expressed as: M i =F i +f MS4B (LN(F i )) (3) F i+1 =M i +f GDFB (LN(M i )) (4) F i and F i+1 Represent the input and output features of the multi-scale attention module, M i represents the extracted multi-scale features, f MS4B (·) indicates MSAB module, f GDFB (·) denotes the GDFB module and LN(·) denotes the layer normalization operation.

6. The method for enhancing cataract fundus images based on weakly supervised learning according to claim 5, characterized in that: In step 4, the mean square error loss, color loss, total discrete pixel change loss, and structural consistency loss of the real cataract image are used as the total loss to train the cataract fundus image enhancement network, specifically including: Step 4.1: In the stage of synthesizing realistic cataract images, the loss function in the CycleGAN network is used to train the stage of synthesizing realistic cataract images. The loss function is expressed as L CycleGAN , the calculation process is shown as follows: Among them, S and R represent the source domain and target domain respectively. represents the mapping function from the S domain to the R domain, represents the mapping function from R domain to S domain; D S and D R represent the discriminators of the source domain and the target domain respectively, L GAN and L cyc They represent adversarial loss and cycle consistency loss respectively, and λ is the weight; Step 4.2: To maintain the structural information of the fundus, the structural loss L in GFENet is also used R , the total loss function is expressed as L total , the calculation process is shown as follows: Among them, S G Represents the structure diagram of the source domain; R G A structural diagram representing the target domain; Represents the true value of the source domain structure graph; Represents the true value of the target domain structure graph; Step 4.3: In the cataract image enhancement stage of the multi-scale attention module, the mean square error loss L is used MSE To constrain the structural changes of the image, the color loss L color To constrain the color brightness change of the image, the total discrete pixel change loss L TV To constrain the edge of the image, the structural consistency loss L SSIM Restore the brightness and structure of the image; the sum of the losses in this step is used as the total loss function, and the total loss function can be expressed as: L multi =L TV Δ(cy)+L MSE ||cy||2+L Color ||maxc rgb -maxy rgb ||+L SSIM ssim((c,y)) (7) Among them, λ represents the hyperparameter, rgb represents r, g, b channels, maxc rgb represents the maximum value of the three RGB channels in the predicted image; c represents the predicted image; y represents the real image; the structural similarity index ssim = 1-ssim.

Images