Interpretable age-related macular degeneration classification method, device and storage medium

Abstract

The present invention discloses an interpretable age-related macular degeneration classification method, device and storage medium, and belongs to the field of image processing. The method includes: classifying fundus photos for AMD to obtain non-AMD fundus images and AMD fundus images; identifying pathological feature areas of AMD fundus images to generate disease feature descriptions; constructing a pathological feature labeling matrix for AMD fundus images, and obtaining feature segmentation images based on semi-supervised image segmentation; labeling disease and predisposing factors for feature segmentation images; establishing a first multi-classification model based on the labeling results and feature segmentation images; judging the true pathological features of feature segmentation images; performing data enhancement based on the Gan network; optimizing the image classification model based on a positive data set; obtaining a second multi-classification model based on the true pathological feature images; and generating an interpretable disease description based on an interpretable algorithm according to the classification results of the second multi-classification model. The present invention can help ophthalmologists better understand the principle and basis of the prediction model.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an interpretable age-related macular degeneration classification method, device and storage medium. Background Art

[0002] In recent years, with the continuous improvement of medical imaging acquisition equipment and the continuous development of image processing, pattern recognition, machine learning and other disciplines, the multidisciplinary field of medical image processing and analysis has achieved fruitful results. These results are of great significance in assisting doctors to make rapid and accurate diagnoses.

[0003] Age-related macular degeneration (AMD) is an eye fundus lesion that seriously threatens the visual function of the elderly. In Western countries, AMD is the main cause of irreversible severe visual impairment in patients over 50 years old. Clinically, AMD is divided into two types: exudative (wet) and condensed (dry). Condensed AMD accounts for about 90% of all AMD, and exudative AMD accounts for about 10% of all AMD. According to relevant research and analysis, 60% to 90% of blind eyes due to AMD are caused by exudative lesions. In view of the increasing severity and incidence of this lesion, more and more newly developed inspection equipment has been used in the inspection of AMD.

[0004] Although machine learning models can assist doctors in classifying and diagnosing age-related macular degeneration, there are still many problems. For example, the amount of data used for learning is small. The public data sets for medical imaging images of common diseases are mostly around 100 cases, and there are fewer for rare fundus diseases, while real-world data is difficult to obtain. In addition, it is difficult to accurately label a large amount of data. Labeling in the medical field, especially pixel-based labeling, is difficult and requires high standards for labeling personnel. The image size is large. For UWF, a single image is large in size, high in resolution, and has too many noise systems, which places certain requirements on the processing speed of the model. It is difficult to interpret and quantify AMD symptoms based on fundus images. The quantification of AMD fundus symptoms based on OCT, RCF, and UWF is still a difficult point in the fields of medicine and imaging, and has high requirements for researchers in cross-fields. The model requirements are high. The boundary of the target segmentation of fundus images is fuzzy and the gradient is complex, which requires high accuracy of the algorithm segmentation. Multimodal. Fundus data includes multimodal data such as OCT, OCTA, RCFP, and UWF. The generalization and robustness of the model are required to be high. The interpretability of artificial intelligence technology in ophthalmology is difficult, mainly reflected in: a) There is a big difference between the data calculation mode and the result diagnosis mode: artificial intelligence technology is mainly based on digital-driven judgment, while ophthalmology focuses on clinical manifestations; in addition, the logic of AI diagnosis is to have a diagnosis result first, and then explain the cause, which is lagging and prone to misidentification and missed lesions, while clinical diagnosis is based on pathological characteristics and emphasizes a priori. b) Due to the black box and robustness problems of AI methods, they cannot be fully trusted in the medical field. c) AI methods are highly dependent on data quality, and there are large test differences under the premise of displaying a small amount of clinical data during the experiment. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an interpretable age-related macular degeneration classification method, device and storage medium, which can help ophthalmologists better understand the principle and basis of the prediction model.

[0006] On the one hand, an embodiment of the present invention provides an interpretable age-related macular degeneration classification method, comprising the following steps: performing AMD classification on fundus photographs based on an image classification model to obtain non-AMD fundus images and AMD fundus images; performing pathological feature area identification on the AMD fundus images based on an interpretable algorithm to generate a disease feature description; constructing a pathological feature labeling matrix for the AMD fundus images, and obtaining a feature segmentation picture based on semi-supervised image segmentation; labeling the disease and the cause on the feature segmentation picture to obtain a labeling result; establishing a first multi-classification model based on the labeling result and the feature segmentation picture; performing true pathological feature judgment on the feature segmentation picture to obtain a true pathological feature picture and a non-true pathological feature picture; based on a Gan network, synthesizing the true pathological feature picture into a positive data set of a training data set, and synthesizing the non-true pathological feature picture into a negative data set of a training data set; optimizing the image classification model based on the positive data set; obtaining a second multi-classification model based on the true pathological feature picture; and generating an interpretable disease description based on an interpretable algorithm according to the classification result of the second multi-classification model.

[0007] According to some embodiments of the present invention, the method further comprises: clustering the feature segmentation pictures to obtain clustering results of the feature segmentation pictures.

[0008] According to some embodiments of the present invention, marking symptoms and causes of the feature segmented image to obtain marking results includes: marking symptoms of the feature segmented image at the pixel level by using marking software to obtain marking results.

[0009] According to some embodiments of the present invention, obtaining the feature segmentation picture based on semi-supervised image segmentation includes: performing image segmentation based on the MsTGANet+Gan semi-supervised image model to obtain the feature segmentation picture; wherein the MsTGANet+Gan semi-supervised image model includes: CGAN, used to generate a verification data set, complete pre-training and training; MsTGANet, including encoding, MsTNL, MsGCS and decoder.

[0010] According to some embodiments of the present invention, based on the Gan network, synthesizing the real pathological feature image into the positive data set of the training data set, and synthesizing the non-real pathological feature image into the negative data set of the training data set includes: performing data enhancement based on the improved data enhancement model of CGAN to obtain generated positive images and generated negative images, and synthesizing the generated positive images into the positive data set, and synthesizing the generated negative images into the negative data set; the improved data enhancement model includes: a generator, which is a U-shaped network structure; a discriminator, including a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, and the structure of the discriminator is Conv×2+BN+ ReLU layer + Conv + activation function; a coarse generator, whose structure is mapping + padding, three convolutions + several regularizations + linear rectification functions and four ResBlks, is used to generate the main features of the fundus, which include the macula, optic disc and color structure; the first discriminator, whose input is a low-pixel image and an image generated by the coarse generator, and whose output is a half-resolution image; a fine generator, whose structure is three ResBlks, one Deconv, one Conv and one output activation layer, is used to generate local information, which includes retinal venules, arterioles, hemorrhage, exudation and microaneurysm; the second The discriminator, whose input is an image which is a high-pixel picture and an image generated by a fine generator, and whose output is a full-resolution image; the coarse generator and the fine generator constitute a coarse+fine generator, and the structure of the coarse+fine generator is obtained by adding the last ResBlk of the coarse generator to the first Conv layer of the fine generator, and is used to generate the main features of the fundus and the local information; the third discriminator, whose input is an image which is a low-pixel picture and an image generated by the coarse+fine generator, and whose output is a full-resolution image; the fourth discriminator, whose input is a high-pixel picture and an image generated by the coarse+fine generator, and whose output is a full-resolution image.

[0011] According to some embodiments of the present invention, optimizing the image classification model based on the positive data set includes: marking the symptoms and causes of the positive data set, and marking the area , ;like If it is no, reduce the weight of the pixel nodes in the region in the last layer of the image classification model; if If it is no, increase the weight of the pixel node of the region in the last layer of the image classification model.

[0012] According to some embodiments of the present invention, the fundus photographs include OCT fundus photographs, RCEP fundus photographs and UWF fundus photographs.

[0013] According to some embodiments of the present invention, the method further comprises: classifying the classification results into early, middle and late stages to obtain early AMD, middle AMD or late AMD.

[0014] On the other hand, an embodiment of the present invention provides an interpretable age-related macular degeneration classification device, comprising: a first module, for classifying fundus photos for AMD based on an image classification model to obtain non-AMD fundus images and AMD fundus images; a second module, for identifying pathological feature areas of the AMD fundus images based on an interpretable algorithm to generate a disease feature description; a third module, for constructing a pathological feature labeling matrix for the AMD fundus images, and obtaining a feature segmentation image based on semi-supervised image segmentation; a fourth module, for labeling the disease and the cause of the disease on the feature segmentation image to obtain a labeling result; a fifth module, for A first multi-classification model is established; a sixth module is used to judge the true pathological features of the feature segmentation image to obtain a true pathological feature image and a non-true pathological feature image; a seventh module is used to synthesize the true pathological feature image into the positive data set of the training data set based on the Gan network, and synthesize the non-true pathological feature image into the negative data set of the training data set; an eighth module is used to optimize the image classification model based on the positive data set; a ninth module is used to obtain a second multi-classification model based on the true pathological feature image; and a tenth module is used to generate an interpretable disease description based on an interpretable algorithm according to the classification result of the second multi-classification model.

[0015] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0016] The embodiments of the present invention achieve at least the following beneficial effects: the method of the embodiments of the present invention generates a disease feature heat map through a picture-interpretable algorithm to provide ophthalmologists with a real-world research causal basis, helping ophthalmologists to better understand the principles and basis of the prediction model, that is, the characteristics of disease identification; through deep learning, accurate classification of age-related macular degeneration types in fundus photos is achieved; the Gan network is used to solve the problems of less pathological feature data, different standardization of the same pathological features, and large differences between gold standards and actual images; and the positive data set is used to optimize the image classification model to improve the accuracy of prediction classification.

[0017] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0019] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the structure of the MsTGANet+Gan semi-supervised learning mechanism of an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of the MsTNL model used in the embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of the MsGCS model used in the embodiment of the present invention;

[0023] Figure 5 A schematic diagram of the MsTGANet semi-supervised learning mechanism applied in an embodiment of the present invention;

[0024] Figure 6 Schematic diagram of a model structure of data enhancement based on CGAN according to an embodiment of the present invention;

[0025] Figure 7 A schematic flow chart of a method according to another embodiment of the present invention;

[0026] Figure 8 The present invention is a schematic block diagram of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0028] In the description of the present invention, "several" means one or more, "more" means two or more, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0029] Glossary:

[0030] OCT: OCT angiography is a non-contact, high-resolution fundus imaging system. Its advantage is that it uses the split-spectrum amplitude-decorrelation angiography (SSADA) algorithm, which can quickly perform fundus vascular imaging with high resolution without the use of contrast agents, and has strong repeatability. In addition, it uses the traditional cross-sectional scan (B scan) combined with the coronal scan (C scan) En-face mode to obtain three-dimensional data images, which can directly observe the range and depth of eye lesions and the changes in the length, caliber, and area of ​​new blood vessels, which has important clinical guidance significance.

[0031] UWF: Ultra-wide-angle fundus imaging. The advent of ultra-wide-angle fundus imaging (UWF) has made it possible to see almost the entire fundus through a small pupil. UWF is easy to use and can see 82.5% of the retina without mydriasis. The ultra-wide-angle retinal imaging reaches 200°, which can obtain a nearly panoramic fundus image. UWF can not only collect ultra-wide-angle pseudo-color fundus images, but also take ultra-wide-angle fundus autofluorescence (FAF) images, fluorescein fundus angiography (FFA) and choroidal vascular angiography (ICGA) images. The application of new ultra-wide-angle fundus examination and diagnosis technologies has given us some new insights into the screening, classification and treatment of some common clinical diseases.

[0032] Explainable algorithms: Currently, many machine learning models can make very good predictions, but they cannot explain how they make predictions. Many data scientists find it difficult to understand why the algorithm gets such predictions. For the classification diagnosis of multiple fundus diseases, the existing methods generally output the classification results directly. Doctors cannot know the basis for the classification results, while explainable algorithms can provide a basis for judgment.

[0033] GAN network: Generative Adversarial Network. GAN contains two models, one is the generative model and the other is the discriminative model. The task of the generative model is to generate instances that look natural and real and similar to the original data. The task of the discriminative model is to determine whether a given instance looks natural and real or artificially forged (the real instance comes from the data set, and the forged instance comes from the generative model).

[0034] Conditional Generative Adversarial Nets (CGAN): Compared with ordinary GAN networks, CGAN generators and discriminators introduce some additional information As a condition, can be any type of auxiliary information, such as class labels or data from other modalities. This is fed into both the discriminator and the generator as an additional input layer to perform conditioning. In the generator, the previous input noise and are combined in a joint hidden representation, and the adversarial training framework allows considerable flexibility in how to compose this hidden representation. In the discriminator, and As input, it is provided to the discriminant function. The objective function of the minimax game is as follows. Using CGAN, angiography images are generated based on ordinary fundus photographs.

[0035]

[0036] Symptom feature heat map: According to the weight value of the last layer of the neural network, pixels with large weights will be displayed, thus forming feature visualization. This step only tells medical staff the principle of the model's prediction, that is, the prediction results can only be obtained because these displayed pixels are more important. These pixels with large weights are very likely to be the reason for judging the patient as positive, that is, the location of his symptom feature value. This knowledge assists medical staff in understanding and assisting in diagnosis through visualization.

[0037] Pixel marking: Use marking software to mark points and draw lines along the image segmentation area. The marked area will mark the pixels as a special value, usually fluorescent color or white.

[0038] Reference Figure 1 The method of the embodiment of the present invention comprises the following steps:

[0039] S1. Classify the fundus photographs for AMD based on the image classification model to obtain non-AMD fundus images and AMD fundus images; the image classification model of this embodiment adopts a pixel-level image binary classification algorithm based on deep learning.

[0040] It is understandable that the fundus photographs of this embodiment include but are not limited to: OCT, RCFP and UWF three types of fundus images. This embodiment uses multiple data sources to improve the fault tolerance of the model, because clinical data may be one of these categories of data. If it is only set to OCT or fundus photographs, then in actual clinical practice, once the data source is different, the model cannot be used, just like fundus photographs are also different, with 45° angles, 98° angles, and 200° angles. Different angles may result in different diagnostic results.

[0041] S2. Identify the pathological feature areas of the AMD fundus image based on an interpretable algorithm and generate a description of the disease features. It is understandable that the interpretable algorithms of this embodiment include but are not limited to CAM, GAP, Grad-CAM and other image interpretable algorithms. Although the fundus photos are classified, it is not clear which pixels in the fundus photos are used to determine the classification of the fundus photos. This step implements the feature area marking and generates a description of the disease features through an image interpretability algorithm, thereby obtaining pixel-level clues based on the original fundus photos.

[0042] S3. Construct a pathological feature labeling matrix for AMD fundus images and obtain feature segmentation images based on semi-supervised image segmentation.

[0043] S4. Label the symptoms and causes of the feature segmented images to obtain the labeling results. It is not difficult to understand that this step adopts a mode of labeling by one researcher and correction and review by two doctors to achieve the labeling of symptoms for AMD positive data at the pixel level of OCT, RCFP and UWF, and label the training data by tracing points. The labeling software can use Visneylabel.

[0044] S5. Establish a first multi-classification model based on the marking results and the feature segmentation pictures. In this step, a deep learning model is trained based on the marking results and the feature segmentation pictures to obtain a first multi-classification model.

[0045] S6. Perform true pathological feature judgment on the feature segmentation image to obtain a true pathological feature image and a non-true pathological feature image.

[0046] S7. Based on the Gan network, the real pathological feature images are synthesized into the positive data set of the training data set, and the non-real pathological feature images are synthesized into the negative data set of the training data set. In order to solve the problems of less pathological feature data, different standardization of the same pathological features, and large difference between the gold standard and the actual image, this embodiment realizes AMD positive disease data enhancement of three images of OCT, RCFP and UWF based on CGAN.

[0047] S8. Optimize the image classification model based on the positive dataset.

[0048] S9. Based on the real pathological feature images, a second multi-classification model is obtained.

[0049] S10. Generate an interpretable symptom description based on the interpretable algorithm according to the classification result of the second multi-classification model. The interpretable symptom description of this embodiment is generated by a symptom feature descriptor (PATHOLOGICAL DESCRIPTOR, PD). In this embodiment, for AMD, since it is only in the macular area, the symptom feature descriptor of this embodiment can directly process the gray value of the segmented image pixel area of ​​the macular area of ​​positive AMD, add it to the frame fundus image, or use other image fusion algorithms to fuse it into the frame image at a specific position.

[0050] Preferably, this embodiment can also be done through a rough AMD classification neural network, setting the pixel activation mechanism (the weight of the pixel point is greater than a certain value, or the activation function for the neural network layer), retaining the upsampling and downsampling mechanism of the node (pixel point), and finally extracting it into an automated PD.

[0051] The interpretable symptom description of this embodiment includes a symptom feature heat map. The feature heat map obtained in this step of this embodiment is obtained based on the weight value of the last layer of the neural network. Pixels with large weights will be displayed to form feature visualization. This step can tell medical staff the principle of the model's prediction, that is, the prediction results can only be obtained because these displayed pixels are more important. These pixels with large weights are very likely to be the reason for judging the patient as positive, that is, the location of his symptom feature value.

[0052] It should be understood that the AMD classification of the embodiment of the present invention includes the following classifications (the macular area is defined as within 5 mm of the fovea, and the perimacula is defined as within 10 mm of the fovea): 1. There are unclear small hard / soft / calcified crystalline drusen in the macular area; 2. There are soft drusen with blurred edges in the macular area; 3. There is focal hyperpigmentation in the macular area; 4. The basal drusen (and / or) detachment of retinal pigment epithelium (RPE) tissue that has been fused can be seen in the macular area, and the area occupies less than 1 / 10 of the macular area; 5. There are clear individual large / large area hard / soft / calcified crystalline drusen in the macular area; 6. The basal drusen (and / or) detachment of retinal pigment epithelium (RPE) tissue that has been fused can be seen in the macular area, and the area occupies greater than or equal to 1 / 10 of the macular area; 7. There is a complete ring-shaped dark spot (Ring) around the small central visual field residue Scotoma); 8. Massive serous detachment of the retinal pigment epithelium (RPE) and bleeding of the subretinal neovascular membrane; 9. Hemorrhage under the fovea and detachment of the retinal pigment epithelium around the fovea; 10. Large choroidal vessels in the macular area and glial scar tissue in the perimacula; 11. Subretinal hemorrhage with residue in the perimacula; 12. Glial scar in the macular area; 13. Temporal hemorrhage residue in the macular area.

[0053] In some embodiments, the method of the embodiment of the present invention further includes: clustering the feature segmentation pictures to obtain clustering results of the feature segmentation pictures.

[0054] In some embodiments, obtaining a feature segmentation picture based on semi-supervised image segmentation includes: performing image segmentation based on the MsTGANet+Gan semi-supervised image model to obtain a feature segmentation picture.

[0055] The case segmentation task based on fundus images has the following challenges: 1) How to improve the network's ability to capture multi-dimensional non-local features in order to handle complex vitreous pathological manifestations in fundus images, especially in terms of size and shape; 2) How to improve the network's ability to learn global features while suppressing noise interference, and solve the noise interference problems caused by low contrast of fundus images and inherent technology; 3) There is less accurate labeled data for segmentation tasks (based on pixel-level labels), in this case how to achieve model optimization and recognition tasks; 4) In addition, although the commonly used Unet, transfer learning and attention mechanisms have good performance on a small number of data sets, they still cannot achieve a good solution to the above problems. Therefore, for RCFP and UWF, the embodiment of the present invention proposes to introduce the MsTGANet+Gan semi-supervised learning mechanism into the disease multi-segmentation task to solve the shortcoming that the Unet network cannot take into account global features, enrich the labeled data, and improve the accuracy.

[0056] MsTGANet is based on the Unet model and adds a multi-scale transformer non-local module (Multi-Scale Transformer Non-Local Module, MsTNL) and a multi-semantic global channel and spatial joint attention module (Multi-Semantic Global Channel And Spatial Joint Attention Module, MsGCS). The MsTGANet model combines the two newly proposed models, MsTNL and MsGCS, to capture multi-dimensional non-local features with long-term dependency information from different layers of the encoder. In addition, the MsGCS model between the encoder and decoder can guide the model to extract multi-semantic global features, improve the model's ability to learn significant features, and suppress the interference of irrelevant local features. Figure 2 As shown in Figure 2, the MsTGANet+Gan semi-supervised learning model is divided into two parts: (1) a semi-supervised mechanism based on CGAN to generate a validation dataset to complete pre-training and training; (2) MsTGANet uses the Unet structure as the basic framework, which is mainly composed of the encoder channel, the MsTNL model (such as Figure 3 ), MsGCS model (such as Figure 4 ) and decoding channels. In the basic framework, the encoder is the same as Unet. The five parts of the MsTGANet encoder, except for the first part which has only two convolutional layers, each of the other blocks consists of MaxPool operation and two convolutional layers. The MaxPool operation is used to downsample the feature map and expand the receptive field, and the convolutional layer is used for feature extraction at different stages. MsTNL is inserted at the end of the encoder path, and feature matrices with different dimensional information from different encoder layers are used as the input of MsTNL to capture multi-dimensional non-local features with remote correlations. The MsGCS model is used to replace the jump connection between the encoder and decoder, aiming to guide the model to fuse multi-semantic global features, thereby improving the model's ability to learn global features while suppressing the interference of irrelevant local features. The pictures generated by the other part of the Gan network are used for the pre-training part in semi-supervised learning (such as Figure 5 ), which mainly includes pixel-based classification and labeling, CGAN-based image generation, and verification. The pixel-based classification and labeling rules are as follows: 1) Binarize the original labeled image and the training segmentation result image to obtain a binary image and 2) Label The calculation formula is as follows, and are the corresponding pixel points of A and B, The interpretability of this algorithm model is mainly reflected in the use of 1) XAI through feature importance; 2) XAI based on attention mechanism (Attention); 3) XAI through knowledge refinement and rule extraction; 3) XAI based on Gan.

[0057]

[0058] Reference Figure 6 In some embodiments, based on the Gan network, controlling the appropriate position and size, synthesizing the real pathological feature image into the positive data set of the training data set, and synthesizing the non-real pathological feature image into the negative data set of the training data set includes: performing data enhancement based on the improved data enhancement model of CGAN to obtain generated positive images and generated negative images, and synthesizing the generated positive images into the positive data set, and synthesizing the generated negative images into the negative data set. This embodiment is based on the classification of disease labeling data at the pixel level to achieve data enhancement.

[0059] Reference Figure 6 , this embodiment adopts two generator models and four discriminator models ( Figure 6 A) is used to take fundus photographs to generate diverse positive AMD images. The generation block consists of two generators, a fine generator and a coarse generator, and adopts a U-shaped network structure design. The coarse generator consists of mapping + padding, three convolutions (Convolution, Conv) + batch normalization (Batch Normalization, BN) + linear rectification function (Rectified Linear Unit, RELU) and four new residual blocks (ResidualBlocks, ResBlk), followed by two transposed convolutions (Deconv), a Conv+BN+ReLU, a Conv and an output activation layer ( Figure 6 B-left), and is responsible for generating the coarse and global structures of the fundus image, such as the structure of the macula, optic disc, and color. The fine generator consists of a reflection + filling, a Conv+BN+ReLU, and a Conv layer, followed by three ResBlk, a Deconv, a Conv, and an output activation layer ( Figure 6B-right), and produces local information including retinal venules, arterioles, hemorrhages, exudates, and microaneurysms. The last ResBlk of the coarse generator is added to the first Conv layer of the fine generator to combine the global features from the coarse generator with the local information in the fine generator. The discriminator of the proposed network is the last layer encoder tied to the fully connected classification, taking a pair of real and generated images and judging which one is real. The fine discriminator takes the full-resolution real image and the generated image pair, while the coarse discriminator takes the half-resolution image ( Figure 6 A). Each discriminator consists of Conv×2+BN+ ReLU layer+Conv+ activation function ( Figure 6 B - bottom).

[0060] Reference Figure 7 In some embodiments, optimizing the image classification model based on the positive data set includes: marking the symptoms and causes of the positive data set, and marking the area , ;like If it is no, reduce the weight of the pixel nodes in the region in the last layer of the image classification model; if If false, increase the weight of the region's pixel nodes in the last layer of the image classification model.

[0061] It should be understood that the method of this embodiment further includes: classifying the classification results into early, middle and late stages to obtain early AMD, middle AMD or late AMD. Referring to the above AMD classification, the classification results are further classified, and the specific classification is as follows:

[0062] 1. Early AMD (any of 1 or 2 or 3 or 4 or 5);

[0063] 2. Mid-term AMD (any two or more of 1, 2, 3, 4, 5);

[0064] 3. Late AMD (any or above of 6, 7, 8, 9, 10, 11, 12, 13).

[0065] Reference Figure 8The present embodiment proposes an interpretable age-related macular degeneration classification device, comprising: a first module, for classifying fundus photos for AMD based on an image classification model to obtain non-AMD fundus images and AMD fundus images; a second module, for identifying pathological feature areas of AMD fundus images based on an interpretable algorithm to generate a disease feature description; a third module, for constructing a pathological feature labeling matrix for AMD fundus images, and obtaining a feature segmentation image based on semi-supervised image segmentation; a fourth module, for labeling the disease and the cause of the disease on the feature segmentation image to obtain a labeling result; a fifth module, for constructing a pathological feature labeling matrix based on the labeling result and the feature segmentation image A first multi-classification model is established; a sixth module is used to judge the true pathological features of the feature segmentation pictures to obtain true pathological feature pictures and non-true pathological feature pictures; a seventh module is used to synthesize the true pathological feature pictures into the positive data set of the training data set based on the Gan network, and synthesize the non-true pathological feature pictures into the negative data set of the training data set; an eighth module is used to optimize the image classification model based on the positive data set; a ninth module is used to obtain a second multi-classification model based on the true pathological feature pictures; and a tenth module is used to generate an explainable disease description based on an explainable algorithm according to the classification results of the second multi-classification model.

[0066] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above method when executed by a processor.

[0067] Although specific embodiments are described herein, those of ordinary skill in the art will recognize that many other modifications or alternative embodiments are also within the scope of the present disclosure. For example, any of the functions and / or processing capabilities described in conjunction with a particular device or component may be performed by any other device or component. In addition, although various exemplary implementations and architectures have been described according to embodiments of the present disclosure, those of ordinary skill in the art will recognize that many other modifications to the exemplary implementations and architectures described herein are also within the scope of the present disclosure.

[0068] Some aspects of the present disclosure are described above with reference to the block diagram and flow chart of the system, method, system and / or computer program product according to the exemplary embodiments.It should be understood that the combination of one or more blocks in the block diagram and the flow chart and the blocks in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively.Equally, according to some embodiments, some blocks in the block diagram and the flow chart may not need to be executed in the order shown, or may not need to be executed in full.In addition, additional components and / or operations beyond those components and / or operations shown in the blocks in the block diagram and the flow chart may be present in certain embodiments.

[0069] Therefore, the blocks in the block diagrams and flow charts support the combination of devices for performing the specified functions, the combination of elements or steps for performing the specified functions, and the program instruction devices for performing the specified functions. It should also be understood that each block in the block diagrams and flow charts and the combination of blocks in the block diagrams and flow charts can be implemented by a special-purpose hardware computer system or a combination of special-purpose hardware and computer instructions that performs a specific function, element or step.

[0070] The program modules, applications, etc. described herein may include one or more software components, including, for example, software objects, methods, data structures, etc. Each such software component may include computer-executable instructions that, in response to execution, cause at least a portion of the functionality described herein (e.g., one or more operations of the illustrative methods described herein) to be performed.

[0071] Software components can be encoded with any of various programming languages. An exemplary programming language can be a low-level programming language, such as an assembly language associated with a specific hardware architecture and / or an operating system platform. The software component comprising assembly language instructions may need to be converted to executable machine code by an assembler before being executed by hardware architecture and / or platform. Another exemplary programming language can be a higher-level programming language, which can be transplanted across multiple architectures. The software component comprising a higher-level programming language may need to be converted to an intermediate representation by an interpreter or a compiler before execution. Other examples of programming languages ​​include but are not limited to macro languages, shells or command languages, job control languages, scripting languages, database queries or search languages ​​or report writing languages. In one or more exemplary embodiments, the software component comprising the instruction of one of the above-mentioned programming language examples can be directly executed by an operating system or other software components, without first being converted into another form.

[0072] Software components may be stored as files or other data storage structures. Software components of similar type or related functions may be stored together, such as in a specific directory, folder, or library. Software components may be static (e.g., preset or fixed) or dynamic (e.g., created or modified at execution time).

[0073] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An interpretable age-related macular degeneration classification method, characterized in that: The following steps are involved: Based on the image classification model, the fundus photos are classified into AMD categories to obtain non-AMD fundus images and AMD fundus images; Based on an interpretable algorithm, pathological feature regions are identified in the AMD fundus image to generate a description of the disease features; Constructing a pathological feature labeling matrix for the AMD fundus image, and obtaining a feature segmentation picture based on semi-supervised image segmentation; Marking symptoms and causes on the feature segmentation images to obtain marking results; Establishing a first multi-classification model based on the labeling result and the feature segmentation picture; Performing true pathological feature judgment on the feature segmentation image to obtain a true pathological feature image and a non-true pathological feature image; Based on the Gan network, the real pathological feature images are synthesized into the positive data set of the training data set, and the non-real pathological feature images are synthesized into the negative data set of the training data set; Optimizing the image classification model based on the positive dataset; Based on the real pathological feature image, a second multi-classification model is obtained; generating an explainable disease description based on an explainable algorithm according to the classification result of the second multi-classification model; Wherein, based on the Gan network, synthesizing the real pathological feature image into the positive data set of the training data set, and synthesizing the non-real pathological feature image into the negative data set of the training data set includes: Performing data enhancement based on the improved data enhancement model of CGAN to obtain generated positive pictures and generated negative pictures, and synthesizing the generated positive pictures into the positive data set, and synthesizing the generated negative pictures into the negative data set; The improved data enhancement model includes: Generator, a U-shaped network structure; The discriminator includes a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, wherein the structure of the discriminator is Conv×2+BN+ ReLU layer+Conv+activation function; A coarse generator, whose structure is mapping+filling, three convolutions+several regularizations+linear rectification functions and four ResBlks, is used to generate the main features of the fundus, which include the macula, optic disc and color structure; The first discriminator, input is a low-pixel image and a picture generated by a coarse generator, and output is a half-resolution image; A fine generator, having a structure of three ResBlk, a Deconv, a Conv, and an output start layer, for generating local information, including retinal venules, arterioles, hemorrhages, exudates, and microaneurysms; The second discriminator, the input is a high-pixel image and a picture generated by the fine generator, and the output is a full-resolution image; The coarse generator and the fine generator constitute a coarse+fine generator, and the structure of the coarse+fine generator is obtained by adding the last ResBlk of the coarse generator to the first Conv layer of the fine generator, which is used to generate the main features of the fundus and the local information; The third discriminator, whose input is a low-pixel image and a generated image of the coarse+fine generator, outputs a full-resolution image; The fourth discriminator takes as input the high-pixel image and the image generated by the coarse+fine generator, and outputs a full-resolution image.

2. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: The method further comprises: Clustering is performed on the feature segmentation pictures to obtain clustering results of the feature segmentation pictures.

3. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: The marking of symptoms and causes on the feature segmentation image to obtain the marking result includes: The feature segmentation image is marked at the pixel level by a marking software to obtain a marking result.

4. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: The method of obtaining a feature segmentation picture based on semi-supervised image segmentation includes: Perform image segmentation based on MsTGANet+Gan semi-supervised image model to obtain the feature segmentation picture; The MsTGANet+Gan semi-supervised image model includes: CGAN, used to generate validation datasets and complete pre-training and training; MsTGANet, including encoder, MsTNL, MsGCS and decoder.

5. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: Optimizing the image classification model based on the positive data set includes: The positive data set is marked with symptoms and causes, and the marked area is , ; like If no, reduce the weight of the pixel nodes in the region in the last layer of the image classification model; like If it is no, increase the weight of the pixel node of the region in the last layer of the image classification model.

6. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: The fundus photographs include OCT fundus photographs, RCEP fundus photographs and UWF fundus photographs.

7. The interpretable age-related macular degeneration classification method according to claim 1, characterized in that: The method further comprises: The classification results are classified into early, middle and late stages to obtain early AMD, middle AMD or late AMD.

8. An interpretable age-related macular degeneration classification device, characterized in that: include: The first module is used to classify the fundus photos for AMD based on the image classification model to obtain non-AMD fundus images and AMD fundus images; The second module is used to identify the pathological characteristic area of ​​the AMD fundus image based on an interpretable algorithm and generate a description of the disease characteristics; The third module is used to construct a pathological feature labeling matrix for the AMD fundus image and obtain a feature segmentation picture based on semi-supervised image segmentation; The fourth module is used to mark the symptoms and causes of the feature segmentation images to obtain the marking results; A fifth module is used to establish a first multi-classification model based on the labeling result and the feature segmentation picture; The sixth module is used to judge the true pathological features of the feature segmentation pictures to obtain true pathological feature pictures and non-true pathological feature pictures; The seventh module is used to synthesize the real pathological feature image into the positive data set of the training data set and synthesize the non-real pathological feature image into the negative data set of the training data set based on the Gan network; An eighth module, used for optimizing the image classification model based on the positive data set; A ninth module, used for obtaining a second multi-classification model based on the real pathological feature image; a tenth module, configured to generate an explainable disease description based on an explainable algorithm according to the classification result of the second multi-classification model; Wherein, based on the Gan network, synthesizing the real pathological feature image into the positive data set of the training data set, and synthesizing the non-real pathological feature image into the negative data set of the training data set includes: Performing data enhancement based on the improved data enhancement model of CGAN to obtain generated positive pictures and generated negative pictures, and synthesizing the generated positive pictures into the positive data set, and synthesizing the generated negative pictures into the negative data set; The improved data enhancement model includes: Generator, a U-shaped network structure; The discriminator includes a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, wherein the structure of the discriminator is Conv×2+BN+ ReLU layer+Conv+activation function; A coarse generator, whose structure is mapping+filling, three convolutions+several regularizations+linear rectification functions and four ResBlks, is used to generate the main features of the fundus, which include the macula, optic disc and color structure; The first discriminator, input is a low-pixel image and a picture generated by a coarse generator, and output is a half-resolution image; A fine generator, having a structure of three ResBlk, a Deconv, a Conv, and an output start layer, for generating local information, including retinal venules, arterioles, hemorrhages, exudates, and microaneurysms; The second discriminator, the input is a high-pixel image and a picture generated by the fine generator, and the output is a full-resolution image; The coarse generator and the fine generator constitute a coarse+fine generator, and the structure of the coarse+fine generator is obtained by adding the last ResBlk of the coarse generator to the first Conv layer of the fine generator, which is used to generate the main features of the fundus and the local information; The third discriminator, whose input is a low-pixel image and a generated image of the coarse+fine generator, outputs a full-resolution image; The fourth discriminator takes as input the high-pixel image and the image generated by the coarse+fine generator, and outputs a full-resolution image.

9. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

Images