Method for removing reflection of inner conjunctiva in eye fundus image

Abstract

The invention discloses a method for removing reflection of inner conjunctiva in an eye fundus image. The method specifically comprises the following steps: collecting a sample picture for preprocessing; constructing a reflection removal model, wherein the reflection removal model comprises a reflection detection module, a blood vessel retention module and a reflection elimination module; processing the sample picture through a light reflection detection module and a blood vessel retention module, obtaining light reflection positioning features and blood vessel features, and then obtaining fusion features; generating a target picture by using a reflection elimination module; and then model training and reasoning are carried out to obtain an eye fundus image with inner conjunctival reflection removed. According to the method, under the condition that the fundus image with the reflective light is given, the reflective area can be positioned, the reflective light is removed, the vascular structure covered by the reflective light is recovered to a certain extent, the problem that the fundus image shot by a mobile fundus imager is not easy to use due to the reflective light of the inner conjunctiva is effectively solved, the automatic reflective light removal function is achieved, and the efficiency is improved. The fundus image imaging cost is effectively reduced, and the production efficiency is improved.

Description

Technical Field

[0001] The invention belongs to the field of medical image processing, and in particular relates to a method for removing inner conjunctival reflection in fundus images. Background Art

[0002] Fundus images are of great value as the basis for ophthalmologists to diagnose patients' diseases. Therefore, the quality of fundus images will directly affect the efficiency of diagnosis. In the past, fundus images were often captured by desktop fundus cameras, which had good image quality. Due to their portability, handheld fundus cameras have gradually shown their advantages, but the image quality is poor, and reflections in fundus images are almost inevitable. Therefore, improving the quality of fundus images is of great value.

[0003] One of the biggest factors affecting the quality of fundus images is the reflection of the inner conjunctiva, which is usually caused by the reflection of light on the inner conjunctiva, especially when the angle between the light source and the imaging sensor is inappropriate. Although the reflection of the inner conjunctiva can be reduced by designing a precise optical structure and an appropriate shooting angle, this type of reflection often exists in the fundus image and cannot be eliminated well due to the structure of the instrument itself and the experience of the photographer. Therefore, a method is needed to remove the reflection of the inner conjunctiva in the fundus image, thereby effectively improving the quality of the fundus image, affecting the derivative tasks based on the fundus image, and improving the work efficiency of ophthalmologists. Summary of the invention

[0004] The first purpose of the present invention is to solve the problem in the background technology and propose a method for removing the inner conjunctival reflection in the fundus image. The method is based on deep learning technology and realizes the automatic reflection removal function of the fundus image by establishing an effective association among the reflection image, the reflection positioning image and the blood vessel detection image.

[0005] The technical solution adopted by the present invention is as follows:

[0006] The present invention comprises the following steps:

[0007] 1) Data preprocessing: Collect sample images, perform preprocessing, and construct a data set; the sample images include reflective images and their corresponding non-reflective images;

[0008] 2) constructing a reflection removal model; the reflection removal model includes a reflection detection module, a blood vessel preservation module and a reflection elimination module;

[0009] 3) Feature extraction: The data sets are processed separately by the reflection detection module and the blood vessel preservation module to obtain the reflection positioning features and blood vessel features, and the fusion features are obtained by fusing the reflection image with the reflection positioning features and blood vessel features;

[0010] 4) Reflection elimination: first encode the fusion features through the reflection elimination module to obtain the reflection distribution features; then decode the reflection distribution features to generate the target image; wherein the reflection elimination module includes a first reflection elimination block, a first downsampling layer, a first reflection elimination block, a second downsampling layer, a first upsampling layer and a second upsampling layer connected in sequence, the first reflection elimination block is connected to the second upsampling layer through a jump connection, and the second reflection elimination block is connected to the first upsampling layer through a jump connection;

[0011] 5) Model training: Use the loss function to train the reflection removal model to obtain the final reflection removal model;

[0012] 6) Model reasoning: Use the reflection removal model obtained in step 5) to test the actual image.

[0013] Furthermore, the step 1) is specifically as follows: the collected image is a reflective image X, which is preprocessed to a size of 256×256, and then data enhancement processing is performed on the reflective image X, including: adding a rotation angle, Gaussian blur, brightness and contrast adjustment, to form a data set;

[0014] The step 3) is specifically as follows: construct a reflection detection module RDM (Reflection Detection Model) and a vessel preservation module VPM (Vessel Preservation Model) respectively, use the reflection detection module RDM to obtain reflection positioning features, and use the vessel preservation module VPM to obtain vessel features. The reflection detection module mainly relies on the reflection detection network RDN (Reflection Detection Network), and the network uses six different convolution modules, including an initial convolution layer composed of 3×3 convolution kernels, a deep convolution layer composed of multiple depth-separable convolution blocks, an SE layer composed of a global average pooling layer, a fully connected layer, and a Sigmoid activation function, a network block layer composed of 2 1×1 convolutions and 1 3×3 convolutions, a global average pooling layer, and a fully connected layer; and the vessel detection module mainly relies on the vessel detection network VDN (Vessel Detection Network), and the network consists of two Unet networks and a Sigmoid activation function. The feature maps output by the convolution modules in the two modules are downsampled to the same size as the last feature map, and the features are stacked in the channel direction to obtain the final fused feature X'.

[0015] The step 4) is specifically implemented as follows: the feature encoder of the reflection removal block RRB (Reflection Removal Block) in the reflection removal module RRM (Reflection Removal Model) is used; the first reflection removal block and the second reflection removal block have the same structure, both of which include a feature encoder and a feature decoder; wherein the feature encoder uses two 3×3 convolutional layers, Gelu, Sigmoid activation functions, and an adaptive average pooling layer, and uses the Kernel attention mechanism to capture pixel-level features; the encoder is used to extract the features of X' at each channel level at different scales, and finally the Sigmoid activation function and the residual block are used to combine the features of the encoder at different stages with the vascular features of the corresponding scale. Get the final reflection distribution characteristics.

[0016] The feature decoder uses four consecutive convolution modules, each of which consists of a 3×3 convolution layer, a batch normalization layer, a sigmoid activation layer, and a dropout layer. The decoder is used to decode the reflective distribution features and generate the final image.

[0017] The step 5) is specifically as follows: using perceptual loss, structural loss, and smooth regression loss between generating the target image and the non-reflective image, combined with vascular loss to guide the model training.

[0018] The second object of the present invention is to provide a system for removing the inner conjunctival reflection in the fundus image for implementing the above method, comprising the following modules:

[0019] Data preprocessing module, used to collect and preprocess images;

[0020] The reflection removal module removes reflections from preprocessed images using a trained reflection removal model.

[0021] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the above method.

[0022] A fourth object of the present invention is to provide a computing device, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the above method is implemented.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1) The present invention uses a deep learning method to detect reflective areas, and uses the distribution of blood vessels in the fundus image as a constraint, combined with a deep convolutional network for reflection removal, which greatly reduces the impact of reflections, effectively improves the quality of fundus images, reduces imaging costs, and improves production efficiency.

[0025] 2) The present invention can perform certain reflection removal on both low-quality data sets and high-quality data sets, and has certain versatility.

[0026] 3) The present invention also has the function of restoring part of the blood vessels after the reflective covering area is removed, providing a structural reference for the invisible blood vessel area, and has good potential application value.

[0027] 4) The present invention can locate the reflective area, remove the reflection, and restore the vascular structure covered by the reflection to a certain extent when given a fundus image with reflection, effectively alleviating the problem that the fundus image taken by the mobile fundus imager is not easy to use due to the reflection of the inner conjunctiva. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the present invention.

[0029] Figure 2 It is a network model structure diagram of the present invention.

[0030] Figure 3 It is a rendering of the reflection detection network RDN of the present invention, wherein (a) is a fundus image with reflection, (b) is a detected reflection area, and (c) is the effect after removing the reflection.

[0031] Figure 4 It is a structural diagram of the vascular loss function of the present invention.

[0032] Figure 5 This is a de-reflection effect diagram of the present invention, wherein the first and third rows are reflective diagrams, and the second and fourth rows are de-reflection effect diagrams. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown, the implementation process of the present invention is as follows:

[0035] Step 1: Collect sample images, perform preprocessing, and build a data set. The specific process is as follows:

[0036] Collect the reflection images X and their corresponding non-reflection images T, preprocess them to a size of 256×256, and then perform data enhancement on the reflection images X, including adding rotation angle, Gaussian blur, brightness and contrast adjustment, to form a data set.

[0037] Step 2: Construct a reflection removal model; Figure 2 As shown, the reflection removal model includes a reflection detection module, a blood vessel preservation module and a reflection elimination module;

[0038] Step 3: Extract features. The specific process is: use the reflection detection network RDN to obtain the reflection positioning features, and use the blood vessel detection network VDN to obtain the blood vessel features.

[0039] The reflective detection network RDN specifically uses the EfficientNet-b3 network, which contains six different convolutional modules, including an initial convolutional layer composed of 3×3 convolutional kernels, a deep convolutional layer composed of multiple depth-separable convolutional blocks, an SE layer composed of a global average pooling layer, a fully connected layer, and a Sigmoid activation function, a network block layer composed of two 1×1 convolutions and one 3×3 convolution, a global average pooling layer, and a fully connected layer;

[0040] The initial convolutional layer consists of a module: a convolution with an input dimension of 3, an output dimension of 40, and a stride of 2; BatchNormalization connected to a Swish activation function; a convolution with an input dimension of 40, an output dimension of 40, and a stride of; Batch Normalization connected to a Swish activation function; a 3×3 maximum pooling with a stride of 2. The deep convolutional layer consists of five MB convolutions, the first of which has an input dimension of 40, an output of 24, and a stride of 1; the second has an input dimension of 24, an output of 40, and a stride of 1; the third has an input dimension of 40, an output of 80, and a stride of 1; the fourth has an input dimension of 80, an output of 112, and a stride of 1; the fifth has an input dimension of 112, an output of 192, and a stride of 1. After passing through the SE layer, the network block layer, and the global average pooling layer, it reaches the fully connected layer with an output dimension of 256.

[0041] Figure 3 This is a rendering of the reflective detection network RDN of the present invention.

[0042] The vascular detection network VDN consists of two Unet networks and a Sigmoid activation function. The first Unet has an input dimension of 3, a type of 1, and 8, 16, and 32 layers; the output features are combined with the original image in the first dimension and then input into the second Unet. The second Unet has an input dimension of 4, a type of 1, and 8, 16, and 32 layers.

[0043] Finally, we downsample the feature maps output by the convolution modules in the two modules to the same size of 256 as the last feature map, and stack the features in the channel direction to obtain the final fusion feature X';

[0044] Step 4: Perform feature encoding. The specific process is as follows: use the feature encoder in the reflection removal block RRB in the reflection removal module RRM; the feature encoder uses two 3×3 convolution layers, Sigmoid activation function, and adaptive average pooling layer, and uses the kernel attention mechanism to capture pixel-level features; use the encoder to extract the features of X' at each channel level at different scales, and finally use the Sigmoid activation function and residual block to combine the features of the encoder at different stages with the vascular features V at the corresponding scale s n , obtain the final reflection distribution characteristics.

[0045] The first convolution has an input dimension of 4, an output dimension of 8, and a stride of 1; the second convolution has an input dimension of 8, an output dimension of 4, and a stride of 1. After the Sigmoid activation function and the adaptive average pooling layer, the kernel attention mechanism is used to obtain the features of each pixel. Kernel attention consists of four convolution layers, three Gelu activation functions, and a Sigmoid function. The first convolution has an input dimension of 4, an output dimension of 3, and a stride of 1; the second convolution has an input dimension of 3, an output dimension of 6, and a stride of 1; the third convolution has an input dimension of 6, an output dimension of 6, and a stride of 1; the fourth convolution has an input dimension of 6, an output dimension of 3, and a stride of 1. The input fusion feature X' is combined with the vascular features in four dimensions of 256, 128, 64, and 32 respectively. Then perform downsampling operation to obtain the reflection distribution characteristics.

[0046] Step 5: Use the decoder to decode the reflective distribution features and generate the final image The feature decoder uses four consecutive convolutional modules, each of which consists of a 3×3 convolutional layer, a batch normalization layer, a sigmoid activation layer, and a dropout layer.

[0047] The first convolution has an input dimension of 16, an output dimension of 32, and a step size of 1; the second convolution has an input dimension of 32, an output dimension of 64, and a step size of 1; the third convolution has an input dimension of 64, an output dimension of 128, and a step size of 1; the fourth convolution has an input dimension of 128, an output dimension of 256, and a step size of 1. The input reflective distribution features are upsampled to twice the original size through convolution at each scale, and then output after passing through the batch normalization layer, the Sigmoid activation layer, and the dropout layer.

[0048] Step 6: Use the main loss function (perceptual loss, structural loss, smooth regression loss) between the generated image and the real image, and combine the vascular loss guidance model designed by the present invention for training; the learning rate of the reflective positioning network is set to 1×10 -5, the learning rate of the reflection removal network is set to 2×10 -4 , and both use the cosine annealing learning strategy to adjust the learning rate. The optimizer is Adam, with parameters β1=0.9 and β2=0.999.

[0049] like Figure 4 As shown in the figure, the vascular loss is composed of two functions: SSIM (Structural Similarity) and DICE. The weight ratio of the main loss function to the vascular loss is 10:1, and the ratio of SSIM to DICE in the vascular loss is 10:1. The vascular loss is mainly the result of eliminating the reflection of the non-reflection image T and its corresponding reflection image X through the network (VPRRM) designed by the present invention. Input VDN, obtain two vascular structure images, calculate the loss, and obtain the final vascular loss.

[0050] Step 7: Send actual cases into the model for testing.

[0052] To further verify the effectiveness of the method of the present invention, this example uses Figure 5 The first and third rows are used as reflective images for testing. Figure 5 The second and fourth rows are the de-reflection results.

[0053] Depend on Figure 5 It can be seen that the present invention realizes the effective reflection removal function based on the reflection picture, effectively reduces the equipment resource cost, and improves the production efficiency. At the same time, the method has the function of restoring part of the blood vessels after removing the reflection coverage area, providing a structural reference for the invisible blood vessel area, and has great potential application value.

[0054] The above specific implementation modes are used to explain the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for removing inner conjunctival reflection in fundus images, characterized in that: The removal method comprises the following steps: 1) Data preprocessing: Collect sample images, perform preprocessing, and construct a data set; the sample images include reflective images X and their corresponding non-reflective images T; 2) constructing a reflection removal model; the reflection removal model includes a reflection detection module, a blood vessel preservation module and a reflection elimination module; 3) Feature extraction: The data sets are processed separately by the reflection detection module and the blood vessel preservation module to obtain the reflection positioning features and blood vessel features. The fusion feature X' is obtained by fusing the reflection image X with the reflection positioning features and blood vessel features. 4) Reflection elimination: Using the reflection elimination module, first encode the fusion feature X' to obtain the reflection distribution feature; then decode the reflection distribution feature to generate the target image ; Wherein, the reflection elimination module includes a first reflection elimination block, a first down-sampling layer, a first reflection elimination block, a second down-sampling layer, a first up-sampling layer and a second up-sampling layer connected in sequence, the first reflection elimination block is connected to the second up-sampling layer through a jump connection, and the second reflection elimination block is connected to the first up-sampling layer through a jump connection; 5) Model training: using target images The reflection removal model is trained by the loss function between the reflection-free image T and the reflection-free image T to obtain the final reflection removal model; 6) Model reasoning: Use the reflection removal model obtained in step 5) to test the actual image.

2. The method for removing inner conjunctival reflection in fundus images according to claim 1, characterized in that: Step 1) The preprocessing includes: adjusting the size of the sample image, and then performing data enhancement processing on the reflective image; wherein the data enhancement processing includes: adding a rotation angle, Gaussian blur, brightness and contrast adjustment.

3. The method for removing inner conjunctival reflection in fundus images according to claim 1, characterized in that: The reflective detection module includes the following convolution modules: an initial convolution layer composed of 3×3 convolution kernels, a deep convolution layer composed of multiple depth-separable convolution blocks, an SE layer composed of a global average pooling layer, a fully connected layer, and a Sigmoid activation function, a network block layer composed of 2 1×1 convolutions, 1 3×3 convolution, a global average pooling layer, and a fully connected layer; the blood vessel detection module is composed of two Unet networks and a Sigmoid activation function.

4. The method for removing inner conjunctival reflection in fundus images according to claim 3, characterized in that: In step 3), the feature maps output by the convolution modules in the reflection detection module and the blood vessel detection module are downsampled to the same size as the last feature map, and the features are stacked in the channel direction to obtain the final fusion feature X'.

5. The method for removing inner conjunctival reflection in fundus images according to claim 1, characterized in that: In the reflection removal module, the first reflection removal block and the second reflection removal block have the same structure, both including a feature encoder and a feature decoder; wherein the feature encoder uses two 3×3 convolutional layers, Gelu, Sigmoid activation functions, and an adaptive average pooling layer, and uses the Kernel attention mechanism to capture pixel-level features; the encoder is used to extract the features of X' at each channel level at different scales, and finally the Sigmoid activation function and the residual block are used to combine the features of the encoder at different stages with the vascular features V at the corresponding scale s n , obtain the final reflection distribution characteristics.

6. The method for removing inner conjunctival reflection in fundus images according to claim 5, characterized in that: The feature decoder uses four consecutive convolution modules, each of which consists of a 3×3 convolution layer, a batch normalization layer, a sigmoid activation layer, and a dropout layer. The decoder is used to decode the reflective distribution features and generate the final target image. .

7. The method for removing inner conjunctival reflection in fundus images according to claim 1, characterized in that: Step 5) The loss function includes a main loss function and a vascular loss function, wherein the main loss function is composed of perceptual loss, structural loss and smooth regression loss, and the vascular loss function is composed of SSIM and DICE.

8. A system for removing inner conjunctival reflection in fundus images, which implements the method according to any one of claims 1 to 7, characterized in that: Includes the following modules: Data preprocessing module, used to collect and preprocess images; The reflection removal module removes reflections from preprocessed images using a trained reflection removal model.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor, characterized in that: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 7 is implemented.

Images