Eye medical image shielding removing method and device and storage medium

Abstract

The invention provides an eye medical image de-occlusion method and device and a storage medium, and relates to the field of image processing, and the method comprises the steps: constructing a training set; carrying out feature extraction by using deformation convolution on the basis of the VQGAN; model training: coding a training sample into a continuous vector based on a CNN Encoder module; coding the continuous vector into a discrete vector based on a CodeBook module; random noise is added to the discrete vector, and a shielded discrete vector is generated; the shielded discrete vector is reconstructed by using a Transform module, and an unshielded discrete vector is generated; based on a CNN Decoder module, the uncovered discrete vector is decoded into an image again; and image shielding removal: receiving an eye medical image with eyelash shielding, and inputting the trained eyelash shielding repair model to obtain an eye medical image without eyelash shielding. According to the invention, the VQGAN model is improved for the eyelash occlusion problem of the eye medical image, feature extraction is carried out by using deformation convolution, and the restoration capability of the eye medical image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an ocular medical image de-occlusion method, device and storage medium. Background Art

[0002] When the human eye is naturally open, it is difficult to avoid the image being affected by eyelash occlusion due to physiological structure, equipment lighting, and improper operation of personnel. In addition, the morphology of eyelashes is complex and diverse. In the ophthalmic medical industry, the problem of eyelash occlusion in ocular medical images will not only interfere with the clinical diagnosis of ophthalmologists, but also affect the performance of many subsequent image analysis tasks, such as tear film breakup time detection, meibomian gland deficiency analysis, lipid layer thickness measurement, or other computer-aided diagnosis. Eyelash occlusion is not only one of the main interfering noises in ocular medical images, but some eyelashes will also be misdetected during subsequent image feature extraction, which seriously affects the accuracy of image analysis. Therefore, the research on eyelash removal occlusion methods for ocular medical images is very critical. Removing eyelash occlusion from ocular medical images is one of the core issues to improve the performance of ocular medical image analysis tasks or improve the analysis and diagnosis capabilities of ocular diseases (such as dry eye diagnosis).

[0003] Since the eyelash region usually has complex interactions and overlaps with other parts of the eye, it becomes very difficult to accurately segment and identify eyelashes from images. In addition, the special morphological, color and texture characteristics of eyelashes also make automatic identification and analysis more challenging. At present, the methods used to remove eyelash occlusion in ocular medical images mainly include traditional methods based on image gradients, lighting information and morphological features and image processing methods based on deep learning. Traditional methods lack the ability to understand the high-dimensional features of eyelash occlusion, and the removal effect is general. Compared with traditional image processing methods, deep learning-based methods can discover high-dimensional features in image structure and automatically extract the most descriptive and significant features, thereby generating images with higher quality than traditional methods. Deep learning includes strong supervised learning, weak supervised learning, unsupervised learning and self-supervised learning. Strong supervised learning requires a large number of paired annotated images to constitute training samples. On the one hand, manual or automatic labeling of data (image / text) is a time-consuming and expensive process. On the other hand, for medical images, paired images with and without eyelashes are more difficult to obtain, and obtaining sufficient data is a challenge in itself. Unsupervised learning methods, such as autoencoders, only reduce the dimension and do not contain more semantic features. They are not very helpful for downstream tasks and generally cannot complete the difficult eyelash occlusion removal task. Although weakly supervised learning does not require paired matching data or data annotation, it still requires obtaining two sets of images with and without eyelashes to form samples, and data acquisition is also relatively difficult.

[0004] In summary, the existing strong supervision, unsupervised and weak supervision learning methods and traditional image processing methods have certain limitations in removing eyelash occlusion from ocular medical images, and their application in the detection and recognition of ocular medical images has been greatly restricted. Therefore, how to design a more convenient and efficient solution to the eyelash occlusion problem in ocular medical images has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a method, device and storage medium for de-occlusion of ocular medical images. The VQGAN model is improved to solve the eyelash occlusion problem of ocular medical images, and deformable convolution is used for feature extraction to improve the ability to repair ocular medical images.

[0006] The first aspect of the present invention discloses a method for de-occluding an eye medical image, comprising: constructing a training set: obtaining an eye image without eyelash occlusion as a training sample; constructing an eyelash occlusion repair model based on VQGAN: the eyelash occlusion repair model comprises a CNN Encoder module, a CodeBook module, a Transformer module and a CNN Decoder module; wherein the CNN Encoder module performs feature extraction based on deformable convolution; model training: encoding the training sample into a continuous vector based on the CNN Encoder module; encoding the continuous vector into a discrete vector based on the CodeBook module; adding random noise to the discrete vector to generate a covered discrete vector; reconstructing the covered discrete vector using the Transformer module to generate an uncovered discrete vector; re-decoding the uncovered discrete vector into an image based on the CNN Decoder module; thereby completing the training of the eyelash occlusion repair model; image de-occlusion: receiving an eye medical image with eyelash occlusion, inputting the trained eyelash occlusion repair model, and obtaining an eye medical image without eyelash occlusion.

[0007] In this technical solution, deformable convolution can be closer to the shape and size of the object when sampling, and is more robust, which standard convolution cannot do. Compared with traditional fixed-window convolution, deformable convolution can effectively deal with collective graphics because its local receptive field is learnable and oriented to the entire image. Deformed convolution can adaptively determine the size of the receptive field and focus on the areas of the image that need attention. Compared with traditional fixed-shape convolution kernels, deformable convolution can better capture the features of objects of different shapes and sizes, thereby achieving better performance in tasks such as object detection and segmentation.

[0008] According to the ocular medical image deocclusion method disclosed in the present invention, preferably, the steps of receiving an ocular medical image with eyelash occlusion, inputting a trained eyelash occlusion restoration model, and obtaining an ocular medical image without eyelash occlusion specifically include: the ocular medical image with eyelash occlusion is passed through a trained CNN Encoder module to generate a continuous vector of the image with eyelashes; the continuous vector of the image with eyelashes enters a trained CodeBook module to generate a discrete vector of the image with eyelashes; the discrete vector of the image with eyelashes enters a trained Transformer module to generate a discrete vector of the image without eyelashes; the discrete vector of the image without eyelashes enters a trained CNN Decoder module to generate an ocular medical image without eyelash occlusion.

[0009] According to the ocular medical image de-occlusion method disclosed in the present invention, preferably, the calculation process of the CNN Encoder module specifically includes: obtaining training samples or feature maps of training samples; calculating sampling positions: for each sampling point in the output position and the convolution kernel, according to the fixed convolution kernel position and the offset corresponding to the fixed convolution kernel position, calculating the sampling position after offset adjustment; bilinear interpolation: estimating the feature value of the sampling point from the nearest pixel value through bilinear interpolation or other interpolation methods; deformation convolution calculation: using the sampling point after offset adjustment to perform convolution calculation to generate pixel values ​​on the output feature map, and the calculation result of each output position is the weighted sum of the convolution kernel on the input feature map area after offset adjustment; convolution calculation: performing convolution calculation on the result of the deformation convolution calculation; repeating the above calculation process several times to complete the feature extraction process; outputting a continuous vector: obtaining a continuous vector through a fully connected layer.

[0010] According to the ocular medical image de-occlusion method disclosed in the present invention, preferably, the steps of calculating the sampling position, bilinear interpolation, deformation convolution calculation and convolution calculation are repeated 3-7 times.

[0011] According to the ocular medical image deocclusion method disclosed in the present invention, preferably, before inputting the training samples into the eyelash occlusion repair model for model training, it also includes: checking the information entropy, contrast, clarity, Laplacian gradient, and grayscale difference function SMD of the training sample images to exclude images that do not meet the training requirements.

[0012] According to the eye medical image deocclusion method disclosed in the present invention, preferably, it also includes: after obtaining the eye medical image without eyelash occlusion, performing image quality assessment; if the assessment result does not meet expectations, modifying the model parameters and retraining.

[0013] According to the ocular medical image de-occlusion method disclosed in the present invention, preferably, the evaluation indicators of image quality evaluation include: a structural similarity index, a learning-perceived image block similarity index and an image generation evaluation index.

[0014] The second aspect of the present invention discloses an ocular medical image de-occlusion device, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement an ocular medical image de-occlusion method as any of the above technical solutions.

[0015] A third aspect of the present invention discloses a computer-readable storage medium storing a program code, wherein the program code is used to implement a method for de-occluding an ocular medical image according to any of the above technical solutions.

[0016] The beneficial effects of the present invention include at least: the present invention performs model training based on self-supervised learning, and does not require labeling of a large number of training samples. The training materials of the present invention are easy to obtain, and it is not necessary to obtain paired images with eyelash occlusion and images without eyelash occlusion of the same patient; the present invention is aimed at the problem of inaccurate feature extraction when the CNN network processes ocular medical images, especially for non-grid eyelashes. The feature extraction process of the traditional CNN Encoder is improved: the use of deformable convolution can be closer to the shape and size of the object when sampling, and is more robust, while standard convolution cannot do this. Compared with the traditional fixed window convolution, the deformable convolution can effectively deal with the collection of graphics, because its local receptive field is learnable and oriented to the whole picture. The deformable convolution can adaptively determine the size of the receptive field and pay attention to the areas where the image needs to be paid attention to. Compared with the traditional fixed shape convolution kernel, the deformable convolution can better capture the features of objects of different shapes and sizes, thereby obtaining excellent performance in the image processing task of removing eyelash occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic flow chart of a method for de-occlusion of an eye medical image according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram of a model training process according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of an image restoration process according to an embodiment of the present invention is shown.

[0020] Figure 4 A schematic block diagram of an eye medical image de-occlusion device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] like Figure 1 As shown, according to one embodiment of the present invention, the eye medical image de-occlusion method disclosed in the present invention includes: step S101, constructing a training set: obtaining an eye image without eyelash occlusion as a training sample; step S102, constructing an eyelash occlusion repair model: the eyelash occlusion repair model includes a CNN Encoder module, a CodeBook module, a Transformer module and a CNN Decoder module; wherein the CNN Encoder module performs feature extraction based on deformable convolution; step S103, model training: encoding the training sample into a continuous vector based on the CNN Encoder module; encoding the continuous vector into a discrete vector based on the CodeBook module; adding random noise to the discrete vector to generate a covered discrete vector; reconstructing the covered discrete vector using the Transformer module to generate an uncovered discrete vector; re-decoding the uncovered discrete vector into an image based on the CNN Decoder module; thereby completing the training of the eyelash occlusion repair model; step S104, image de-occlusion: receiving an eye medical image with eyelash occlusion, inputting the trained eyelash occlusion repair model, and obtaining an eye medical image without eyelash occlusion.

[0024] According to the above embodiment, preferably, step S104 specifically includes: the eye medical image with eyelashes occluded generates a continuous vector of the image with eyelashes through the trained CNN Encoder module; the continuous vector of the image with eyelashes enters the trained CodeBook module to generate a discrete vector of the image with eyelashes; the discrete vector of the image with eyelashes enters the trained Transformer module to generate a discrete vector of the image without eyelashes; the discrete vector of the image without eyelashes enters the trained CNNDecoder module to generate an eye medical image without eyelashes occlusion.

[0025] According to the above embodiment, preferably, the calculation process of the CNN Encoder module specifically includes: obtaining training samples or feature maps of training samples; calculating sampling positions: for each sampling point in the output position and the convolution kernel, according to the fixed convolution kernel position and the offset corresponding to the fixed convolution kernel position, calculate the sampling position after offset adjustment; bilinear interpolation: estimate the feature value of the sampling point from the nearest pixel value through bilinear interpolation or other interpolation methods; deformation convolution calculation: use the sampling point after offset adjustment to perform convolution calculation to generate pixel values ​​on the output feature map, and the calculation result of each output position is the weighted sum of the convolution kernel on the input feature map area after offset adjustment; convolution calculation: perform convolution calculation on the result of the deformation convolution calculation; repeat the above calculation process several times to complete the feature extraction process; output continuous vector: obtain a continuous vector through a fully connected layer. The above steps of calculating sampling position, bilinear interpolation, deformation convolution calculation and convolution calculation are repeated 3-7 times.

[0026] According to the above embodiment, preferably, before step S102, the method further includes: checking the information entropy, contrast, clarity, Laplacian gradient, and grayscale difference function SMD of the training sample images to exclude images that do not meet the training requirements.

[0027] According to the above embodiment, preferably, it also includes: after obtaining the eye medical image without eyelash occlusion, performing image quality assessment; if the assessment result does not meet expectations, modifying the model parameters and retraining. The evaluation indicators of image quality assessment include: structural similarity index, learning perception image block similarity index and image generation evaluation index.

[0028] According to another embodiment of the present invention, a specific implementation process of the above-mentioned eye medical image de-occlusion method is also disclosed: step S1, obtaining an original eye image without eyelash occlusion; step S2, inputting the image obtained in the above S1 as a training sample into the improved VQGAN network for training, the improved VQGAN network learns how to repair the image, and obtains a trained eyelash occlusion repair model, and the output of the eyelash occlusion repair model is a predicted eye image without eyelash occlusion; step S3, inputting the newly collected eye image with eyelash occlusion into the eyelash occlusion repair model generated in step S2, and obtaining a predicted eye image without eyelash occlusion.

[0029] like Figure 2 As shown, in this embodiment, step S2 specifically includes: Step S21, using the original eye image without eyelashes as a training sample, and training the CNN Encoder part, CodeBook part and CNN Decoder part of the improved VQGAN network by self-supervised learning; the image processing process of step S21 includes firstly obtaining a continuous vector through the input image through the CNN Encoder, and then introducing the CodeBook (using the CodeBook to further discretize the above continuous vector to obtain a discrete vector, and finally decoding the above discrete vector into an image through the CNN Decoder, so far, the first step of the training process is completed. Among them, the way to introduce the CodeBook is to use the nn.Embedding function; the discrete vector is a code group, which consists of multiple codes.

[0030] Step S22, based on the trained CNN Encoder and CodeBook, the generalization ability of the Transformer is improved by randomly replacing the code with strong noise, and then using the Transformer model to reconstruct its code group. Finally, a trained Transformer model is obtained. The closer the discrete vector output by the Transformer model is to the discrete vector (code group) without adding random noise, the better.

[0031] Specifically, the image processing process of step S22 is to randomly mask the discrete vectors described in step S21 to obtain masked discrete vectors, and finally reconstruct the masked discrete vectors using the Transformer model to obtain uncovered discrete vectors. At this point, the second step of the training process is completed.

[0032] Through the above two-step training process, after the training is completed, a trained eyelash occlusion repair model is obtained, and the trained eyelash occlusion repair model includes a trained CNN Encoder part, a trained CodeBook part, a trained Transformer model part and a trained CNN Decoder part connected in sequence according to the data processing order.

[0033] According to the above implementation, the step of obtaining a continuous vector from the input image through the CNN Encoder in step S21 specifically includes: Step S211: Input the original eye image without eyelashes (the feature map of the original image is input in the second and subsequent cycles) into the CNN Encoder (size H×W×C in , where H and W are the height and width respectively, C in is the number of input channels), calculate the sampling position: For the output position (h', w') and each sampling point in the convolution kernel, the actual sampling position on the input feature map is calculated based on the fixed convolution kernel position and the offset corresponding to that position.

[0034] Step S212, bilinear interpolation: Since the actual sampling position may not be on the integer coordinates of the input feature map, it is necessary to estimate the feature value of the sampling point from the nearest pixel value through bilinear interpolation or other interpolation methods.

[0035] Step S213, convolution calculation: Use these offset-adjusted sampling points to perform convolution calculation to generate pixel values ​​on the output feature map. The calculation result of each output position is the weighted sum of the convolution kernel on the offset-adjusted input feature map area.

[0036] Step S214: After the above steps, the output feature map size is H'×W'×C out , where H' and W' are determined by factors such as the input feature map size and the convolution step size, C out Is the number of output channels. Convolution is performed on this feature map.

[0037] Step S215, repeat S211-S214 several times (usually 3-7 times).

[0038] Step S216, obtaining a continuous vector through a fully connected layer.

[0039] like Figure 3 As shown, step S3 of this embodiment specifically includes: inputting the captured image with eyelash occlusion into the trained eyelash occlusion restoration model, and sequentially passing through the trained CNN Encoder part, the trained CodeBook part, the trained Transformer model part and the trained CNN Decoder part to obtain a generated image without eyelash occlusion.

[0040] like Figure 4 As shown, according to another embodiment of the present invention, an ocular medical image de-occlusion device 400 is also disclosed, including: a memory 401, used to store program instructions; a processor 402, used to call the program instructions stored in the memory to implement the ocular medical image de-occlusion method as in the above embodiment.

[0041] According to yet another embodiment of the present invention, a computer-readable storage medium is disclosed. The computer-readable storage medium stores program codes, and the program codes are used to implement the ocular medical image de-occlusion method as described in the above embodiment.

[0042] According to the above embodiments, compared with CNN, which is good at capturing local features, the advantage of Transformer is that it can better integrate the global information of the image. However, the self-attention operation of Transformer is too expensive and can only generate some images with lower resolution. Therefore, the advantages of CNN and Transformer can be combined. First, the image can be compressed into a small image with smaller size and richer information using CNN-based VQGAN, and then the small image can be generated using Transformer. It can enhance the ability to repair eyelash occlusion in ocular medical images.

[0043] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.

[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for deocclusion of an eye medical image, characterized in that: include: Construct a training set: obtain eye images without eyelash occlusion as training samples; Constructing an eyelash occlusion repair model based on VQGAN: the eyelash occlusion repair model includes a CNN Encoder module, a CodeBook module, a Transformer module and a CNN Decoder module; wherein the CNN Encoder module performs feature extraction based on deformable convolution; Model training: encoding the training sample into a continuous vector based on the CNN Encoder module; encoding the continuous vector into a discrete vector based on the CodeBook module; adding random noise to the discrete vector to generate a covered discrete vector; reconstructing the covered discrete vector using the Transformer module to generate an uncovered discrete vector; re-decoding the uncovered discrete vector into an image based on the CNN Decoder module; thereby completing the training of the eyelash occlusion repair model; Image de-occlusion: receiving an eye medical image with eyelash occlusion, inputting the trained eyelash occlusion repair model, and obtaining an eye medical image without eyelash occlusion.

2. The method for de-occlusion of an eye medical image according to claim 1, characterized in that: The step of receiving the eye medical image with eyelash occlusion, inputting the trained eyelash occlusion repair model, and obtaining the eye medical image without eyelash occlusion specifically includes: The trained CNN Encoder module of the eye medical image occluded by eyelashes generates a continuous vector of the image with eyelashes; The continuous vector of the image with eyelashes enters the trained CodeBook module to generate a discrete vector of the image with eyelashes; The discrete vector of the image with eyelashes enters the trained Transformer module to generate a discrete vector of the image without eyelashes; The discrete vector of the eyelash-free image enters the trained CNN Decoder module to generate an eye medical image without eyelash occlusion.

3. The method for de-occlusion of an eye medical image according to claim 1, characterized in that: The calculation process of the CNN Encoder module specifically includes: Obtaining training samples or feature maps of training samples; Calculate the sampling position: For each sampling point in the output position and the convolution kernel, calculate the offset-adjusted sampling position based on the fixed convolution kernel position and the offset corresponding to the fixed convolution kernel position; Bilinear interpolation: Estimate the feature value of the sampling point from the nearest pixel value through bilinear interpolation or other interpolation methods; Deformed convolution calculation: Use the offset-adjusted sampling points for convolution calculation to generate pixel values ​​on the output feature map. The calculation result of each output position is the weighted sum of the convolution kernel on the offset-adjusted input feature map area. Convolution calculation: performing convolution calculation on the result of the deformation convolution calculation; Repeat the above calculation process several times to complete the feature extraction process; Output continuous vector: Get a continuous vector through a fully connected layer.

4. The method for de-occlusion of an eye medical image according to claim 3, characterized in that: The steps of calculating the sampling position, the bilinear interpolation, the deformation convolution calculation and the convolution calculation are repeated 3-7 times.

5. The method for de-occlusion of an eye medical image according to claim 1, characterized in that: Before inputting the training samples into the eyelash occlusion repair model for model training, the method further includes: Check the information entropy, contrast, clarity, Laplacian gradient, and grayscale difference function SMD of the training sample images to exclude images that do not meet the training requirements.

6. The method for de-occlusion of an eye medical image according to claim 1, characterized in that: Also includes: After obtaining the eye medical image without eyelash occlusion, image quality assessment is performed.

7. The method for de-occlusion of an eye medical image according to claim 6, characterized in that: The evaluation indicators of the image quality evaluation include: a structural similarity indicator, a learning perception image block similarity indicator and an image generation evaluation indicator.

8. An eye medical image de-occlusion device, characterized in that: include: A memory for storing program instructions; A processor, configured to call the program instructions stored in the memory to implement the ocular medical image de-occlusion method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code, and the program code is used to implement the ocular medical image de-occlusion method according to any one of claims 1 to 7.

Images