Cataract grading method and device, electronic device, storage medium

Abstract

The present disclosure relates to the fields of computer vision and ophthalmic medicine, and provides a method and device for cataract grading, an electronic device, and a storage medium. The method includes: obtaining an AS-OCT image of the eye of the subject to be measured; extracting image features of the AS-OCT image of the eye using a cataract feature extractor; respectively calculating the similarity between the image features and the reference features of the AS-OCT images of cataracts at each grade, and taking the cataract severity grade corresponding to the AS-OCT image reference feature with the highest similarity as the cataract diagnosis grade of the AS-OCT image of the eye; the reference features of the AS-OCT images of cataracts are obtained based on the standard slit lamp images of cataracts at each grade during the training stage. By introducing the standard slit lamp image as reference information in the cataract grading of AS-OCT images, the present disclosure improves the interpretability of cataract diagnosis and grading based on AS-OCT images, making the cataract diagnosis and grading more objective.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of computer vision and ophthalmology, and particularly relates to a method and device for grading cataracts, an electronic device, and a storage medium. Background Art

[0002] Nuclear Cataract (NC) is a common type of cataract. In patients with nuclear cataract, turbidity appears in the core area of the eye lens, which causes blurred vision and visual impairment in patients. Generally speaking, as people age, proteins in the lens gradually aggregate and oxidize. Therefore, most patients with nuclear cataract are the elderly. With the intensification of social aging, it can be predicted that the number of patients with nuclear cataract will continue to rise in the future. Since cataract may lead to blindness in the late stage, the diagnosis and treatment in the early stage of cataract are crucial for patients.

[0003] In recent years, the research on deep learning has flourished, and many deep learning methods in the field of computer vision have inspired the research on deep learning in the field of medical images. When applied to cataract diagnosis, deep learning can learn the characteristics of cataracts and achieve automatic grading diagnosis of cataracts based on these characteristics, greatly shortening the diagnosis time. Therefore, by applying deep learning to cataract diagnosis, more early-stage cataract patients can be helped to be diagnosed as early as possible and receive better treatment.

[0004] Currently, in clinical practice, ophthalmologists regard slit lamp examination as the gold standard for nuclear cataract diagnosis. Specifically, when an ophthalmologist performs a slit lamp examination on a patient, a slit lamp image is taken of the patient, and the taken image is compared with the standard slit lamp images of each grade of cataract, and the cataract grade of the patient is judged based on experience. This makes the accuracy of the slit lamp examination results highly dependent on the experience level of ophthalmologists.

[0005] With the development of medical technology, new types of ophthalmic images are also used in cataract diagnosis, such as anterior segment optical coherence tomography (AS-OCT) images. AS-OCT can obtain high-resolution images of the anterior segment in a non-contact manner, which can reflect the anterior segment structure more clearly than slit lamp images and is also convenient for deep learning models to better learn the ocular features of cataract patients. Therefore, it is feasible to use deep learning models based on AS-OCT images to assist doctors in diagnosing cataracts. However, when most existing technologies use deep learning models for cataract detection and grading tasks, only AS-OCT images and fundus images are used. Although this can achieve a certain degree of accuracy, since there are no clear medical guidelines for the abnormalities in these two types of images and the cataract grading situation, the interpretability of the diagnostic results obtained by the corresponding deep learning models still needs to be enhanced.

[0006] In the prior art, Gao et al. selected the region of interest (ROI) from slit lamp images as the input of the convolutional neural network. Yang et al. used ensemble learning for automatic cataract detection and grading, and separately extracted wavelet-based, sketch-based, and texture-based features from each fundus image to form three independent feature sets, and constructed support vector machines and backpropagation neural networks for each feature set. Zhang et al. used a deep convolutional neural network to achieve cataract detection and grading, and used multiple population-based clinical retinal fundus images from hospitals, a total of 5,620 images, for training and testing the deep convolutional neural network. Zhou et al. used a deep neural network with discrete state transitions for automatic cataract classification based on fundus images and found that combinations of different types of features had better effects than single features. Xu et al. proposed a model composed of two-level subnets. The global attention subnet focuses on the global structural information of the fundus image, and the local attention subnet focuses on the local discriminant features of specific regions. These two types of subnets extract retinal features at different attention levels and then combine them for the final cataract classification.

[0007] In addition, Jacob et al. used the VGG-16 network to propose a new method for early cataract recognition based on the region of interest of a custom user, allowing professional medical staff to manually select the area in the fundus image that is most likely to have cataracts and input this area into the VGG-16 for classification. Zhang et al. based on AS-OCT images, with the deep residual network ResNet as the main framework of the model, proposed a region weight adjustment module based on the attention mechanism (RIR Block) to introduce clinical prior information into cataract grading prediction and grading tasks. Also based on AS-OCT images, Zhang et al. extracted features using a convolutional neural network (such as the VGG16 network), proposed an Adaptive Feature Squeeze Network (AFS Net) to dynamically compress local features and automatically adjust the weights of the features, and finally used the softmax function of the normalized exponential function to achieve the grading of nuclear cataracts.

[0008] However, although the above-mentioned existing technologies can achieve a certain degree of accuracy in the diagnosis and grading of cataracts, they usually only use a single-modal image as the discrimination basis, and the interpretability of the obtained diagnosis and grading results still needs to be enhanced. Summary of the Invention

[0009] The present disclosure aims to at least solve one of the problems existing in the prior art, and provides a cataract grading method, device, electronic device, and storage medium.

[0010] In one aspect of the present disclosure, a cataract grading method is provided, and the cataract grading method includes:

[0011] Obtain the AS-OCT image of the eyeball of the person to be measured;

[0012] Extract the image features corresponding to the AS-OCT image of the eyeball by using a cataract feature extractor;

[0013] Calculate the similarity between the image features and the reference features of the AS-OCT images of cataracts at each level respectively, and use the cataract severity level corresponding to the reference feature of the AS-OCT image of cataracts with the highest similarity to the image features as the cataract diagnosis level of the AS-OCT image of the eyeball; wherein, the reference features of the AS-OCT images of cataracts are obtained after the training is completed, and the obtaining method is: generating corresponding standard virtual AS-OCT images for the standard slit lamp images of cataracts at each level through a trained AS-OCT image generator respectively, and then using the trained cataract feature extractor to extract the features of the standard virtual AS-OCT images at each level, and using the features extracted from the standard virtual AS-OCT images as the reference features of the AS-OCT images of cataracts at the corresponding levels.

[0014] Optionally, the cataract feature extractor is trained according to the following training steps:

[0015] Obtain a slit lamp image of the eyeball and its corresponding real AS-OCT image;

[0016] Input the slit lamp image of the eyeball into the AS-OCT image generator to obtain a corresponding virtual AS-OCT image;

[0017] Input the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain corresponding real AS-OCT image features and virtual AS-OCT image features;

[0018] Combine the virtual AS-OCT image features with the corresponding virtual AS-OCT image to obtain a corresponding virtual AS-OCT feature map, and combine the real AS-OCT image features with the corresponding real AS-OCT image to obtain a corresponding real AS-OCT feature map;

[0019] Input the virtual AS-OCT feature map and the real AS-OCT feature map into the AS-OCT image discriminator to obtain corresponding discrimination results;

[0020] According to the discrimination results, adjust the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain the trained AS-OCT image generator and the trained cataract feature extractor.

[0021] Optionally, the training steps further include:

[0022] Based on the structural differences between the real AS-OCT image and the virtual AS-OCT image with the same cataract severity level, construct a pixel loss function, and use the pixel loss function to train the AS-OCT image generator.

[0023] Optionally, the training steps further include: constructing a discrimination loss function based on the AS-OCT image discriminator, and using the discrimination loss function to train the AS-OCT image generator and the cataract feature extractor.

[0024] Optionally, the training steps further include:

[0025] Use the real AS-OCT image features and the virtual AS-OCT image features as the input of the classifier, use the number of levels of the cataract severity level as the output of the classifier, train the classifier, and backpropagate the gradient to the cataract feature extractor and the AS-OCT image generator simultaneously.

[0026] Optionally, combining the virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combining the real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps includes:

[0027] Performing feature alignment on the virtual AS-OCT image features and the real AS-OCT image features;

[0028] Combining the feature-aligned virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combining the feature-aligned real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps.

[0029] Optionally, before calculating the similarity between the image features and the cataract AS-OCT image reference features of each level, the cataract grading method further includes:

[0030] Obtaining the cataract standard slit lamp images corresponding to each cataract severity level;

[0031] Inputting each of the cataract standard slit lamp images into the trained AS-OCT image generator to obtain corresponding standard virtual AS-OCT images;

[0032] Inputting each of the standard virtual AS-OCT images into the trained cataract feature extractor to obtain the cataract AS-OCT image reference features corresponding to each cataract severity level.

[0033] In another aspect of the present disclosure, a cataract grading device is provided, and the cataract grading device includes:

[0034] An acquisition module, configured to acquire the eye AS-OCT image of the subject;

[0035] An extraction module, configured to extract the image features corresponding to the eye AS-OCT image by using a cataract feature extractor;

[0036] A calculation module, configured to calculate the similarity between the image features and the benchmark features of cataract AS-OCT images at each level respectively, and use the cataract severity level corresponding to the cataract AS-OCT image benchmark feature with the highest similarity to the image features as the cataract diagnosis level of the eye AS-OCT image; wherein, the cataract AS-OCT image benchmark features are obtained after the training is completed, and the obtaining method is: respectively generating corresponding standard virtual AS-OCT images of cataract standard slit lamp images at each level through a trained AS-OCT image generator, and then using the trained cataract feature extractor to extract the features of the standard virtual AS-OCT images at each level, and taking the features extracted from the standard virtual AS-OCT images as the cataract AS-OCT image benchmark features corresponding to the corresponding levels.

[0037] Optionally, the cataract grading device further includes:

[0038] A training module, configured to train the trained cataract feature extractor according to the following training steps:

[0039] Obtain an eye slit lamp image and its corresponding real AS-OCT image;

[0040] Input the eye slit lamp image into the AS-OCT image generator to obtain a corresponding virtual AS-OCT image;

[0041] Input the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain corresponding real AS-OCT image features and virtual AS-OCT image features;

[0042] Combine the virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combine the real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps;

[0043] Input the virtual AS-OCT feature map and the real AS-OCT feature map into the AS-OCT image discriminator to obtain corresponding discrimination results;

[0044] According to the discrimination results, adjust the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain the trained AS-OCT image generator and the trained cataract feature extractor.

[0045] Optionally, the training module is further configured to:

[0046] Based on the structural differences between the real AS-OCT image and the virtual AS-OCT image corresponding to the same cataract severity level, a pixel loss function is constructed, and the AS-OCT image generator is trained using the pixel loss function.

[0047] Optionally, the training module is further configured to: construct a discriminant loss function based on the AS-OCT image discriminator, and use the discriminant loss function to train the AS-OCT image generator and the cataract feature extractor.

[0048] Optionally, the training module is further configured to:

[0049] Use the real AS-OCT image features and the virtual AS-OCT image features as the input of a classifier, use the number of levels of the cataract severity level as the output of the classifier, train the classifier, and backpropagate the gradients to the cataract feature extractor and the AS-OCT image generator simultaneously.

[0050] Optionally, the training module is configured to combine the virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combine the real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps, including:

[0051] The training module is configured to:

[0052] Perform feature alignment on the virtual AS-OCT image features and the real AS-OCT image features;

[0053] Combine the feature-aligned virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combine the feature-aligned real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps.

[0054] Optionally, the cataract grading device further includes:

[0055] A reference feature extraction module, configured to:

[0056] Obtain the cataract standard slit lamp images corresponding to each cataract severity level;

[0057] Input each of the cataract standard slit lamp images into the trained AS-OCT image generator to obtain corresponding standard virtual AS-OCT images;

[0058] Input each of the standard virtual AS-OCT images into the trained cataract feature extractor to obtain the cataract AS-OCT image reference features corresponding to each cataract severity level.

[0059] Another aspect of the present disclosure provides an electronic device, including:

[0060] At least one processor; and,

[0061] A memory communicatively connected to the at least one processor; wherein,

[0062] The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the cataract grading method described above.

[0063] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the cataract grading method described above is implemented.

[0064] Compared with the prior art, the present disclosure introduces the information of the corresponding slit lamp image into the cataract AS-OCT image reference features at each level. By comparing the image features of the AS-OCT image of the subject's eyeball with the cataract AS-OCT image reference features corresponding to different cataract severity levels, calculating the similarity, and taking the cataract severity level corresponding to the cataract AS-OCT image reference feature with the highest similarity as the cataract diagnosis level of the AS-OCT image of the subject's eyeball, it not only improves the interpretability of cataract diagnosis and grading based on AS-OCT images, but also makes the cataract diagnosis and grading more objective, reducing the dependence on the experience level of ophthalmologists. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplary illustrations do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, and the drawings in the figures do not constitute a scale limitation.

[0066] Figure 1 It is a flowchart of a cataract grading method provided by an embodiment of the present disclosure;

[0067] Figure 2 It is a flowchart of a cataract grading method provided by another embodiment of the present disclosure;

[0068] Figure 3 It is a flowchart of the training steps of a cataract feature extractor provided by another embodiment of the present disclosure;

[0069] Figure 4 The deep learning framework diagram for cataract grading provided by another embodiment of the present disclosure;

[0070] Figure 5 The structural schematic diagram of a cataract grading device provided by another embodiment of the present disclosure;

[0071] Figure 6 The structural schematic diagram of an electronic device provided by another embodiment of the present disclosure. Specific embodiments

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will elaborate on each embodiment of the present disclosure in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that in each embodiment of the present disclosure, many technical details are provided to help readers better understand the present disclosure. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present disclosure can still be implemented. The following division of each embodiment is for convenience of description and should not constitute any limitation to the specific implementation of the present disclosure. Each embodiment can be combined and cross-referenced with each other on the premise of not being contradictory.

[0073] One embodiment of the present disclosure relates to a cataract grading method, and its process is as Figure 1 shown, including:

[0074] Step S110, obtaining the AS-OCT image of the eye of the person to be measured.

[0075] Specifically, the AS-OCT image of the eye refers to the AS-OCT image taken of the eye of the person to be measured. In combination with Figure 2 , this AS-OCT image of the eye can be used as the unknown AS-OCT image for determining the severity level of cataract.

[0076] Step S120, using a cataract feature extractor to extract the image features corresponding to the AS-OCT image of the eye.

[0077] Specifically, in combination with Figure 2 , step S120 can input the AS-OCT image of the eye as the unknown AS-OCT image into the trained cataract feature extractor for feature extraction to obtain the corresponding image features, and use the extracted image features as the unknown AS-OCT image features.

[0078] The cataract feature extractor can be constructed using any network structure that can extract features from an image. For example, the cataract feature extractor can be constructed based on a convolutional neural network, a deep residual network, a deep residual shrinkage network, etc. Of course, the cataract feature extractor can also be constructed based on other network structures, and this embodiment does not limit this.

[0079] Step S130, respectively calculate the similarity between the image features and the cataract AS-OCT image benchmark features of each level, and use the cataract severity level corresponding to the cataract AS-OCT image benchmark feature with the highest similarity to the image features as the cataract diagnosis level of the eyeball AS-OCT image. The cataract AS-OCT image benchmark features are obtained after the training is completed, and the acquisition method is: each level of cataract standard slit lamp image is generated by a trained AS-OCT image generator to generate a corresponding standard virtual AS-OCT image, and then the trained cataract feature extractor is used to extract the features of the standard virtual AS-OCT image of each level, and the features extracted from the standard virtual AS-OCT image are used as the cataract AS-OCT image benchmark features of the corresponding level.

[0080] Specific, combined Figure 2 In step S130, the similarities between the unknown AS-OCT image feature and the cataract AS-OCT image benchmark feature corresponding to each cataract severity level can be respectively calculated, and the cataract severity level corresponding to the cataract AS-OCT image benchmark feature with the highest similarity to the unknown AS-OCT image feature can be used as the cataract diagnosis level corresponding to the unknown AS-OCT image, thereby realizing the cataract severity level corresponding to the unknown AS-OCT image by judging the similarity.

[0081] The higher the similarity between the cataract AS-OCT image benchmark features corresponding to a certain cataract severity level and the image features of the subject's eye AS-OCT image, the closer the cataract AS-OCT image benchmark features are to the image features of the subject's eye AS-OCT image. Therefore, the cataract severity level corresponding to the cataract AS-OCT image benchmark features that have the highest similarity to the image features of the subject's eye AS-OCT image can be used as the cataract diagnosis level corresponding to the subject's eye AS-OCT image, thereby realizing cataract diagnosis grading based on AS-OCT images.

[0082] The reference features of AS-OCT images for cataracts of all grades can be obtained before step S130. The obtaining method is as follows: The standard slit lamp images of cataracts of all grades are respectively input into the trained AS-OCT image generator to generate corresponding standard virtual AS-OCT images, and then the trained cataract feature extractor is used to extract the features of the standard virtual AS-OCT images of all grades. The features extracted from the standard virtual AS-OCT images are used as the reference features of AS-OCT images for cataracts of corresponding grades.

[0083] Compared with the prior art, the embodiments of the present disclosure introduce the information of the corresponding slit lamp images into the reference features of AS-OCT images for cataracts of all grades. By comparing the image features of the AS-OCT images of the subject's eyeball with the reference features of AS-OCT images for cataracts corresponding to different severities of cataracts, calculating the similarity, and taking the severity grade of cataracts corresponding to the reference features of AS-OCT images with the highest similarity as the cataract diagnosis grade of the AS-OCT image of the subject's eyeball, it not only improves the interpretability of cataract diagnosis based on AS-OCT images, but also makes the diagnosis and grading of cataracts more objective and reduces the dependence on the experience level of ophthalmologists.

[0084] Exemplarily, referring to Figure 3 and Figure 4 , the trained cataract feature extractor is obtained according to the following training steps:

[0085] Step S210, obtain the slit lamp image of the eyeball and its corresponding real AS-OCT image.

[0086] Specifically, the real AS-OCT image here refers to the scanned image obtained by using an AS-OCT device to examine the eyeball. The real AS-OCT image corresponding to the slit lamp image of the eyeball refers to the AS-OCT image obtained by using an AS-OCT device to examine the eyeball corresponding to the slit lamp image of the eyeball.

[0087] Step S220, input the slit lamp image of the eyeball into the AS-OCT image generator to obtain the corresponding virtual AS-OCT image.

[0088] Specifically, the AS-OCT image generator can adopt the generator in the Generative Adversarial Nets (GAN) model. The AS-OCT image generator is used to generate the corresponding virtual AS-OCT image based on the input slit lamp image of the eyeball, so that the virtual AS-OCT image is as realistic as possible to deceive the AS-OCT image discriminator.

[0089] Step S230: Input the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain the corresponding real AS-OCT image features and virtual AS-OCT image features.

[0090] Specifically, in step S230, the same cataract feature extractor is used to extract the image features of the real AS-OCT image and the virtual AS-OCT image respectively, so as to obtain the corresponding real AS-OCT image features and virtual AS-OCT image features.

[0091] Step S240: Combine the virtual AS-OCT image features with the corresponding virtual AS-OCT image to obtain the corresponding virtual AS-OCT feature map, and combine the real AS-OCT image features with the corresponding real AS-OCT image to obtain the corresponding real AS-OCT feature map.

[0092] Specifically, the present embodiment does not limit the specific manner of combining the image features with the corresponding images. As long as the manner of combining the virtual AS-OCT image features with the corresponding virtual AS-OCT image is the same as the manner of combining the real AS-OCT image features with the corresponding real AS-OCT image, it is ensured that the virtual AS-OCT feature map and the real AS-OCT feature map are obtained through the same combination manner, thereby reducing the error in the training process.

[0093] Step S250: Input the virtual AS-OCT feature map and the real AS-OCT feature map into the AS-OCT image discriminator to obtain the corresponding discrimination results.

[0094] Specifically, the AS-OCT image discriminator can adopt the discriminator in the GAN model. The AS-OCT image discriminator is used to distinguish the virtual AS-OCT feature map and the real AS-OCT feature map, and output the discrimination result as the corresponding discrimination result, so that the subsequent steps can improve the performance of the AS-OCT image generator according to the discrimination result.

[0095] Step S260: According to the discrimination results, adjust the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain the trained AS-OCT image generator and the trained cataract feature extractor.

[0096] Specifically, the present embodiment can also perform iterative training according to the above training steps, adjust the parameters of the AS-OCT image generator and the cataract feature extractor multiple times, so that the AS-OCT image generator and the cataract feature extractor can continuously learn and optimize.

[0097] In this embodiment, a deep learning framework for cataract grading is constructed using an AS-OCT image generator, a cataract feature extractor, and an AS-OCT image discriminator. During the training process of this deep learning framework, slit lamp images of the eyeball are introduced, and images of these two modalities, namely slit lamp images of the eyeball and AS-OCT images, are jointly used in the training process, enabling the trained deep learning framework to generate virtual AS-OCT images based on slit lamp images, eliminating the modality differences between slit lamp images and AS-OCT images. While retaining the cataract features in the original slit lamp images, it realizes the modality transfer from slit lamp images to AS-OCT images, effectively solving the problem of differences between different modalities and further improving the interpretability of cataract diagnosis and grading based on AS-OCT images.

[0098] It should be noted that in order to ensure that two types of features, namely virtual AS-OCT image features and real AS-OCT image features, can be mapped to the same space while retaining the original information, corresponding constraint conditions need to be introduced, and these constraint conditions can be reflected through loss functions.

[0099] Exemplarily, the training steps further include: constructing a pixel loss function based on the structural differences between real AS-OCT images and virtual AS-OCT images with the same cataract severity level, and using the pixel loss function to train the AS-OCT image generator.

[0100] Specifically, the pixel loss function is used to ensure that virtual AS-OCT images of cataracts corresponding to the same level are similar in structure to real AS-OCT images. Denote the real AS-OCT image as X ASOCT , and the virtual AS-OCT image as X′ ASOCT . Use the superscript i to represent the corresponding cataract severity level. Then the real AS-OCT image corresponding to cataract severity level i can be denoted as , and the virtual AS-OCT image corresponding to cataract severity level i can be denoted as , then is the virtual AS-OCT image generated based on the slit lamp image of the eyeball corresponding to cataract severity level i. On this basis, the pixel loss function L pix can be expressed as

[0101] Exemplarily, the training steps further include: constructing a discriminative loss function based on the AS-OCT image discriminator, and using the discriminative loss function to train the AS-OCT image generator and the cataract feature extractor.

[0102] Specifically, by optimizing the discriminative loss function, it can also ensure that virtual AS-OCT images of cataracts corresponding to the same level are similar in structure to real AS-OCT images.

[0103] The AS-OCT image generator and the AS-OCT image discriminator constitute a generative adversarial network structure. The AS-OCT image generator can be represented by G, and the AS-OCT image discriminator can be represented by D. The output of the AS-OCT image discriminator D is a probability value between 0 and 1. The AS-OCT image discriminator D believes that the more likely the input feature map it receives is a real AS-OCT feature map, the greater the probability value it outputs. On the contrary, the AS-OCT image discriminator D believes that the more likely the input feature map it receives is a virtual AS-OCT feature map, the smaller the probability value it outputs. Denote the discriminant loss function as L G_GAN Denote the real AS-OCT image feature as M ASOCT Denote the virtual AS-OCT image feature as M' ASOCT Then there is

[0104] Exemplarily, the training step further includes: using the real AS-OCT image feature and the virtual AS-OCT image feature as the input of the classifier, using the number of levels of the cataract severity level as the output of the classifier, training the classifier, and simultaneously backpropagating the gradient to the cataract feature extractor and the AS-OCT image generator.

[0105] Specifically, in order to ensure that both the real AS-OCT image feature and the virtual AS-OCT image feature can contain the cataract severity level information, in this embodiment, a classifier is set up. The real AS-OCT image feature M ASOCT and the virtual AS-OCT image feature M' ASOCT are used as the input of the classifier, and the specific number of levels of the cataract severity level is used as the output of the classifier. When training the classifier, the gradient is simultaneously backpropagated to the cataract feature extractor and the AS-OCT image generator, so that the features extracted by the cataract feature extractor can retain the cataract severity level information as much as possible.

[0106] Denote the sample input to the classifier, such as the real AS-OCT image feature or the virtual AS-OCT image feature, as x, and denote the probability value that the classifier predicts the correct cataract severity level corresponding to the sample x as p(x). Denote the loss function of the classifier as L cls When the number of samples in a batch is n, use cross-entropy to construct the loss function of the classifier, then there is

[0107] Exemplarily, step S240 includes: performing feature alignment on the virtual AS-OCT image features and the real AS-OCT image features; combining the virtual AS-OCT image features after feature alignment with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combining the real AS-OCT image features after feature alignment with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps.

[0108] Specifically, in this embodiment, a feature alignment technique can be adopted to perform feature alignment on the virtual AS-OCT image features and the real AS-OCT image features, so that the virtual AS-OCT image features and the real AS-OCT image features corresponding to the same level of cataract are as similar as possible, that is, the similarity is greater, and at the same time, the difference between the virtual AS-OCT image features and the real AS-OCT image features corresponding to different levels of cataract is as large as possible, that is, the similarity is smaller, thereby eliminating the intra-modal gap between the virtual AS-OCT image features and the real AS-OCT image features corresponding to the same level of cataract, and further reducing the error in the training process.

[0109] To ensure that the virtual AS-OCT image features and the real AS-OCT image features corresponding to the same level of cataract are as similar as possible, that is, the similarity is as large as possible, and the virtual AS-OCT image features and the real AS-OCT image features corresponding to different levels of cataract are as different as possible, that is, the similarity is as small as possible. During the feature alignment process, cosine similarity can be used as a standard to measure similarity. The more similar the image features are, the larger the corresponding cosine similarity value is. For a batch of n samples, the triplet loss can be used to calculate the difference between the similarity of the AS-OCT image features corresponding to the same level of cataract in this batch and the similarity of the AS-OCT image features corresponding to different levels of cataract. Accordingly, the triplet loss function L tri can be expressed as wherein, represents the real AS-OCT image features corresponding to the cataract severity level i, represents the virtual AS-OCT image features corresponding to the cataract severity level i, represents the virtual AS-OCT image features corresponding to the cataract severity level j, represents the similarity between and represents the similarity between and tri That is to say, the role of the triplet loss function L is to increase the similarity between and Similarity with between.

[0110] Exemplarily, before calculating the similarity between the image features and the reference features of the cataract AS-OCT images of each grade in step S130, the cataract grading method further includes:

[0111] Obtain the standard slit lamp images of cataracts corresponding to each grade of cataracts respectively; input each standard slit lamp image of cataracts into the trained AS-OCT image generator respectively to obtain the corresponding standard virtual AS-OCT images; input each standard virtual AS-OCT image into the trained cataract feature extractor respectively to obtain the reference features of the cataract AS-OCT images corresponding to each severity grade of cataracts, that is, the reference features of the cataract AS-OCT images of each grade.

[0112] Specifically, the reference features of the cataract AS-OCT images of each grade can be obtained based on the standard slit lamp images of cataracts of each grade by using the trained AS-OCT image generator and the trained cataract feature extractor, so as to introduce the information of the slit lamp images, which are the gold standard for cataract diagnosis, into the cataract diagnosis and grading based on AS-OCT images, further improving the interpretability of the cataract diagnosis and grading based on AS-OCT images, reducing the dependence on the experience level of ophthalmologists, and making the cataract diagnosis and grading results obtained based on AS-OCT images more objective.

[0113] Another embodiment of the present disclosure relates to a cataract grading device, as Figure 5 shown, including:

[0114] An acquisition module 510, configured to acquire the eye AS-OCT image of the subject;

[0115] An extraction module 520, configured to extract the image features corresponding to the eye AS-OCT image by using a cataract feature extractor;

[0116] A calculation module 530 is configured to calculate the similarity between the image features and the reference features of the cataract AS-OCT images at each level respectively, and use the cataract severity level corresponding to the cataract AS-OCT image reference feature with the highest similarity to the image features as the cataract diagnosis level of the eye AS-OCT image. Among them, the cataract AS-OCT image reference features are obtained after the training is completed. The obtaining method is as follows: The standard slit lamp images of cataracts at each level are respectively input into the trained AS-OCT image generator to generate corresponding standard virtual AS-OCT images, and then the trained cataract feature extractor is used to extract the features of the standard virtual AS-OCT images at each level, and the features extracted from the standard virtual AS-OCT images are used as the cataract AS-OCT image reference features corresponding to the respective levels.

[0117] Compared with the prior art, the embodiments of the present disclosure introduce the information of the corresponding slit lamp images into the reference features of the cataract AS-OCT images at each level. By comparing the image features of the eye AS-OCT image of the subject with the reference features of the cataract AS-OCT images corresponding to different cataract severity levels, calculating the similarity, and using the cataract severity level corresponding to the cataract AS-OCT image reference feature with the highest similarity as the cataract diagnosis level of the eye AS-OCT image of the subject, it not only improves the interpretability of cataract diagnosis and grading based on AS-OCT images, but also makes the cataract diagnosis and grading more objective, reducing the dependence on the experience level of ophthalmologists.

[0118] Exemplarily, the cataract grading device further includes:

[0119] A training module for training a trained cataract feature extractor according to the following training steps:

[0120] Obtain the eye slit lamp image and its corresponding real AS-OCT image; input the eye slit lamp image into the AS-OCT image generator to obtain the corresponding virtual AS-OCT image; input the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain the corresponding real AS-OCT image features and virtual AS-OCT image features; combine the virtual AS-OCT image features with the corresponding virtual AS-OCT image to obtain the corresponding virtual AS-OCT feature map, and combine the real AS-OCT image features with the corresponding real AS-OCT image to obtain the corresponding real AS-OCT feature map; input the virtual AS-OCT feature map and the real AS-OCT feature map into the AS-OCT image discriminator to obtain the corresponding discrimination result; according to the discrimination result, adjust the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain a trained AS-OCT image generator and a trained cataract feature extractor.

[0121] Exemplarily, the training module is further configured to: construct a pixel loss function based on the structural differences between the real AS-OCT image and the virtual AS-OCT image with the same cataract severity level as the real AS-OCT image, and use the pixel loss function to train the AS-OCT image generator.

[0122] Exemplarily, the training module is further configured to: construct a discriminant loss function based on the AS-OCT image discriminator, and use the discriminant loss function to train the AS-OCT image generator and the cataract feature extractor.

[0123] Exemplarily, the training module is further configured to: use the real AS-OCT image features and the virtual AS-OCT image features as the inputs of the classifier, use the number of levels of the cataract severity level as the output of the classifier, train the classifier, and simultaneously backpropagate the gradients to the cataract feature extractor and the AS-OCT image generator.

[0124] Exemplarily, the training module is configured to combine the virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combine the real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps, including:

[0125] The training module is configured to: perform feature alignment on the virtual AS-OCT image features and the real AS-OCT image features; combine the feature-aligned virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combine the feature-aligned real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps.

[0126] Exemplarily, the cataract grading device further includes:

[0127] A reference feature extraction module, configured to: obtain the cataract standard slit lamp images corresponding to each cataract severity level respectively; input each of the cataract standard slit lamp images into the trained AS-OCT image generator to obtain the corresponding standard virtual AS-OCT images respectively; input each of the standard virtual AS-OCT images into the trained cataract feature extractor to obtain the cataract AS-OCT image reference features corresponding to each cataract severity level respectively.

[0128] For the specific implementation method of the cataract grading device provided by the embodiments of the present disclosure, reference may be made to the cataract grading method provided by the embodiments of the present disclosure, which will not be elaborated herein.

[0129] Another embodiment of the present disclosure relates to an electronic device, such as Figure 6 as shown, comprising:

[0130] at least one processor 601; and,

[0131] a memory 602 communicatively connected to the at least one processor 601; wherein,

[0132] the memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 to enable the at least one processor 601 to execute the cataract grading method described in the above embodiment.

[0133] Wherein, the memory and the processor are connected by a bus. The bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art, and thus will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or multiple elements, such as multiple receivers and transmitters, and provides a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0134] The processor is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory may be used to store data used by the processor when performing operations.

[0135] Another embodiment of the present disclosure relates to a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the cataract grading method described in the above embodiment is implemented.

[0136] That is, those skilled in the art can understand that all or part of the steps in implementing the method described in the above embodiment can be completed by a program instructing relevant hardware. The program is stored in a storage medium, including several instructions for causing a device (which may be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in various embodiments of the present disclosure. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0137] Those of ordinary skill in the art will understand that the above-described embodiments are specific implementations of the present disclosure, and in actual applications, various changes may be made to them in form and detail without departing from the spirit and scope of the present disclosure.

Claims

1. A method for grading cataracts, characterized in that, The cataract grading method includes: Obtaining the AS-OCT image of the eyeball of the person to be measured; Extracting the image features corresponding to the AS-OCT image of the eyeball by using a cataract feature extractor; Calculating the similarity between the image features and the reference features of the AS-OCT images of cataracts at each level respectively, and taking the cataract severity level corresponding to the reference feature of the AS-OCT image of the cataract with the highest similarity to the image features as the cataract diagnosis level of the AS-OCT image of the eyeball; wherein, the reference features of the AS-OCT images of cataracts are obtained after the training is completed, and the obtaining method is: generating the corresponding standard virtual AS-OCT images from the standard slit lamp images of cataracts at each level through the trained AS-OCT image generator respectively, and then using the trained cataract feature extractor to extract the features of the standard virtual AS-OCT images at each level, and taking the features extracted from the standard virtual AS-OCT images as the reference features of the AS-OCT images of cataracts at the corresponding levels; The cataract feature extractor is trained according to the following training steps: Obtaining the slit lamp image of the eyeball and its corresponding real AS-OCT image; Inputting the slit lamp image of the eyeball into the AS-OCT image generator to obtain the corresponding virtual AS-OCT image; Inputting the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain the corresponding real AS-OCT image features and virtual AS-OCT image features; Combining the virtual AS-OCT image features with the corresponding virtual AS-OCT image to obtain the corresponding virtual AS-OCT feature map, and combining the real AS-OCT image features with the corresponding real AS-OCT image to obtain the corresponding real AS-OCT feature map; Inputting the virtual AS-OCT feature map and the real AS-OCT feature map into the AS-OCT image discriminator to obtain the corresponding discrimination results; According to the discrimination results, adjusting the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain the trained AS-OCT image generator and the trained cataract feature extractor.

2. The cataract grading method according to claim 1, wherein The training steps further include: Based on the structural differences between the real AS-OCT image and the virtual AS-OCT image with the same cataract severity level corresponding thereto, constructing a pixel loss function, and using the pixel loss function to train the AS-OCT image generator.

3. The cataract grading method according to claim 1, wherein, The training steps further include: constructing a discrimination loss function based on the AS-OCT image discriminator, and using the discrimination loss function to train the AS-OCT image generator and the cataract feature extractor.

4. The cataract grading method according to claim 1, wherein, The training steps further include: Using the real AS-OCT image features and the virtual AS-OCT image features as the input of a classifier, and using the number of levels of the cataract severity level as the output of the classifier, train the classifier, and backpropagate the gradients to the cataract feature extractor and the AS-OCT image generator simultaneously.

5. The cataract grading method according to claim 1, characterized in that, The combining of the virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and the combining of the real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps includes: Performing feature alignment on the virtual AS-OCT image features and the real AS-OCT image features; Combining the feature-aligned virtual AS-OCT image features with the corresponding virtual AS-OCT images to obtain corresponding virtual AS-OCT feature maps, and combining the feature-aligned real AS-OCT image features with the corresponding real AS-OCT images to obtain corresponding real AS-OCT feature maps.

6. The cataract grading method according to any one of claims 1 to 5, characterized in that, Before calculating the similarity between the image features and the reference features of cataract AS-OCT images of each level, the cataract grading method further includes: Obtaining the corresponding cataract standard slit lamp images for each cataract severity level; Inputting each of the cataract standard slit lamp images into the trained AS-OCT image generator to obtain corresponding standard virtual AS-OCT images; Inputting each of the standard virtual AS-OCT images into the trained cataract feature extractor to obtain the reference features of cataract AS-OCT images corresponding to each cataract severity level.

7. A cataract grading device, characterized in that, The cataract grading device includes: An acquisition module for acquiring the eye AS-OCT image of the subject; An extraction module for extracting the image features corresponding to the eye AS-OCT image by using a cataract feature extractor; A calculation module for calculating the similarity between the image features and the reference features of cataract AS-OCT images of each level respectively, and taking the cataract severity level corresponding to the reference feature of cataract AS-OCT image with the highest similarity to the image features as the cataract diagnosis level of the eye AS-OCT image; wherein, the reference features of cataract AS-OCT images are obtained after the training is completed, and the obtaining method is: generating corresponding standard virtual AS-OCT images for each level of cataract standard slit lamp images through the trained AS-OCT image generator, and then using the trained cataract feature extractor to extract the features of each level of standard virtual AS-OCT images, and taking the features extracted from the standard virtual AS-OCT images as the reference features of cataract AS-OCT images corresponding to the corresponding levels; The cataract feature extractor is trained according to the following training steps: Obtain a slit lamp image of the eyeball and its corresponding real AS-OCT image; Input the slit lamp image of the eyeball into an AS-OCT image generator to obtain a corresponding virtual AS-OCT image; Input the real AS-OCT image and the virtual AS-OCT image into the cataract feature extractor respectively to obtain corresponding real AS-OCT image features and virtual AS-OCT image features; Combine the virtual AS-OCT image features with the corresponding virtual AS-OCT image to obtain a corresponding virtual AS-OCT feature map, and combine the real AS-OCT image features with the corresponding real AS-OCT image to obtain a corresponding real AS-OCT feature map; Input the virtual AS-OCT feature map and the real AS-OCT feature map into an AS-OCT image discriminator to obtain corresponding discrimination results; According to the discrimination results, adjust the parameters of the AS-OCT image generator and the cataract feature extractor respectively to obtain the trained AS-OCT image generator and the trained cataract feature extractor.

8. An electronic device, characterized in that, Comprising: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the cataract grading method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the cataract grading method according to any one of claims 1 to 6.

Images