Cataract classification method, lesion visualization method and related device

Through the training of cataract classification network and the use of multi-level convolutional network modules, the problem of time-consuming and high misdiagnosis rate of cataract classification in the existing technology is solved, and the rapid and accurate cataract classification and lesion visualization are achieved, which improves the Patients’ treatment efficiency and diagnostic accuracy.

CN114972844BActive Publication Date: 2024-11-01TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210451295.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-27
Publication Date
2024-11-01
Estimated Expiration
2042-04-27

AI Technical Summary

Technical Problem

The existing cataract classification methods rely on manual judgment of ophthalmologists, which is time-consuming and labor-intensive, and are prone to misdiagnosis, resulting in some patients not being able to receive timely and effective treatment.

Method used

By obtaining the fundus image to be tested and inputting it to the trained cataract classification network, obtaining the output feature vector for classification, reducing the manual judgment process, providing cataract classification methods and lesion visualization methods, using multi-level convolutional network modules and pooling Feature extraction and classification are performed by technologies such as layer, fully connected layer, activation function, etc.

Benefits of technology

It realizes rapid classification of cataracts, reduces treatment time, improves user experience, and facilitates visualization of lesions, improving the accuracy and efficiency of diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114972844B_ABST
    Figure CN114972844B_ABST
Patent Text Reader

Abstract

The present invention discloses a cataract classification method, a lesion visualization method and related devices, which are applied to the field of image processing, including obtaining a fundus image to be tested; inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector; and obtaining type information of the fundus image to be tested according to the output feature vector. The fundus image to be tested is obtained, the fundus image to be tested is input into a trained cataract classification network to obtain a feature vector, and the fundus image to be tested is classified according to the output feature vector to obtain type information of the fundus image to be tested. The cataract classification method can obtain the cataract classification result corresponding to the fundus image to be tested, reduce the process of manual judgment, make the treatment of patients more immediate, and improve the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to a cataract classification method, a lesion visualization method, and related devices. Background Art

[0002] Cataracts are eye diseases caused by lens metabolism disorders due to aging, heredity, local nutritional disorders, immune and metabolic abnormalities, trauma, poisoning, radiation, etc., which lead to lens protein denaturation and turbidity. The turbidity of the lens of cataract patients prevents light from being projected onto the retina, causing blurred vision. At present, cataracts are clinically divided into cortical cataracts, nuclear cataracts, and posterior subcapsular cataracts according to the location of the lesions.

[0003] In the related art, the LOCS III classification method is used to observe the slit lens and back reflection images to evaluate the patient's cataract type. However, this method relies on the ophthalmologist's manual judgment of the cataract type and requires the use of multiple instruments and equipment, which takes a long time. Due to the limited resources of ophthalmologists and equipment, the judgment of cataract type is prone to misdiagnosis, resulting in some cataract patients being unable to receive timely and effective treatment. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiments of the present application provide a cataract classification method, a lesion visualization method and related devices, which can reduce the time for cataract classification, make the patient's treatment more immediate, and improve the user experience.

[0006] In a first aspect, the present invention provides a method for classifying cataracts, comprising:

[0007] Acquire the fundus image to be tested;

[0008] Inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector;

[0009] The type information of the fundus image to be measured is obtained according to the output feature vector.

[0010] The cataract classification method provided according to the first aspect of the present invention has at least the following beneficial effects: obtaining a fundus image to be tested, inputting the fundus image to be tested into a trained cataract classification network to obtain a feature vector, and classifying the fundus image to be tested according to the output feature vector to obtain type information of the fundus image to be tested. The cataract classification method can obtain a cataract classification result corresponding to the fundus image to be tested, reduces the process of manual judgment, makes the patient's treatment more immediate, and improves the user experience.

[0011] According to some embodiments of the first aspect of the present invention, the type information of the fundus image to be measured includes at least one of the following fundus image types: cortical cataract, nuclear cataract, posterior subcapsular cataract, and no cataract.

[0012] According to some embodiments of the first aspect of the present invention, the training step of the cataract classification network includes:

[0013] Acquire multiple sample fundus images and cataract categories corresponding to the sample fundus images;

[0014] Inputting the sample fundus image into the cataract classification network to be trained to obtain a sample feature vector corresponding to the sample fundus image;

[0015] The sample feature vector is classified, and the obtained sample feature vector corresponds to the cataract category corresponding to the sample fundus image.

[0016] According to some embodiments of the first aspect of the present invention, before inputting the fundus image to be tested into a trained cataract classification network, the method includes:

[0017] Homogenizing the size of the fundus image to be measured;

[0018] Normalizing the RGB values ​​of the fundus image to be tested;

[0019] The result after the size and RGB value are normalized is updated as the fundus image to be tested.

[0020] According to some embodiments of the first aspect of the present invention, the step of normalizing the size of the fundus image to be measured includes:

[0021] The size of the fundus image to be tested is unified as N 0 *N 0 , where N 0 is a positive integer greater than or equal to 1;

[0022] The step of normalizing the RGB values ​​of the fundus image to be tested includes:

[0023] According to the preset RGB mean and RGB variance, the RGB value of the fundus image to be tested is calculated, wherein the RGB value of the fundus image to be tested can be expressed as:

[0024] x = (x - mean(x)) / std(x)

[0025] Wherein, mean is the preset RGB mean, std is the preset RGB variance, and the fundus image to be tested after the size and RGB value are normalized is a 3-channel N0 *N 0 structure.

[0026] According to some embodiments of the first aspect of the present invention, inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector includes:

[0027] The fundus image to be tested is expanded through a multi-level convolutional network module to obtain a first feature atlas, wherein the first feature atlas is N of M channels. t *N t Structure, M is a positive integer greater than or equal to 1, N t is a positive integer greater than or equal to 1;

[0028] The first feature atlas is sequentially passed through a pooling layer, a fully connected layer, and an activation function to obtain an output feature vector.

[0029] According to some embodiments of the first aspect of the present invention, the number of channels of the convolutional network module increases step by step, and / or the size of the feature map generated by the convolutional network module decreases step by step, so that the size of the feature map in the first feature map set satisfies: N t =N 0 / 2 s , s is the number of levels of the convolutional network module.

[0030] According to some embodiments of the first aspect of the present invention, the fundus image to be measured is a 3-channel N*N structure;

[0031] The method of expanding the fundus image to be tested through a multi-level convolutional network module to obtain a first feature atlas includes:

[0032] The fundus image to be tested is passed through the first-level convolutional network module to obtain a second feature atlas, wherein the second feature atlas is M / 2 5 Channel N / 2 2 *N / 2 2 structure;

[0033] The second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas, wherein the third feature atlas is M / 2 3 Channel N / 2 2 *N / 2 2 structure;

[0034] The third feature atlas is passed through the third-level convolutional network module to obtain a fourth feature atlas, wherein the fourth feature atlas is M / 2 2 Channel N / 2 3 *N / 2 3 structure;

[0035] The fourth feature atlas is passed through the fourth-level convolutional network module to obtain a fifth feature atlas, wherein the fifth feature atlas is N / 2 of the M / 2 channels. 4 *N / 2 4 structure;

[0036] The fifth feature atlas is passed through the fifth-level convolutional network module to obtain the first feature atlas, where N t =N 0 / 2 5

[0037] According to some embodiments of the first aspect of the present invention,

[0038] The step of passing the fundus image to be tested through a first-level convolutional network module to obtain a second feature atlas includes:

[0039] Passing the fundus image to be tested through a convolution kernel;

[0040] The fundus image to be tested that has passed the convolution kernel is sequentially passed through a batch normalization layer, an activation function, and a maximum pooling layer to obtain a second feature atlas.

[0041] According to some embodiments of the first aspect of the present invention, the second-level convolutional network module, the third-level convolutional network module, the fourth-level convolutional network module and the fifth-level convolutional network module respectively include multiple residual modules.

[0042] In a second aspect, the present invention provides a method for visualizing a lesion, comprising:

[0043] Acquire the fundus image to be tested;

[0044] Passing the fundus image to be tested through a multi-level convolutional network module in a trained cataract classification network to obtain a first feature atlas;

[0045] According to the first feature atlas, calculating a class activation map corresponding to the first feature atlas;

[0046] Superimposing the class activation map with the fundus image to be tested to obtain a target image for visualizing the lesion point;

[0047] The target image is output.

[0048] The lesion point visualization method provided in the second aspect of the present invention has at least the following beneficial effects: the fundus image to be tested is input into the multi-level classification network module in the trained cataract classification network, a first feature atlas is calculated, and based on the first feature atlas, its corresponding class activation map is calculated, and the class activation map is superimposed on the fundus image to be tested, so as to obtain a target image for visualizing the lesion points. The visualization of the lesion points in the fundus image to be tested facilitates the user to obtain the location of the lesion points, facilitates later treatment, and improves the user experience.

[0049] According to some embodiments of the second aspect of the present invention, calculating, according to the first feature atlas, a class activation map corresponding to the first feature atlas includes:

[0050] According to the first feature atlas, calculating a gradient atlas corresponding to the first feature atlas;

[0051] Averaging the gradient maps of the gradient atlas to obtain weights corresponding to the feature maps in the first feature atlas;

[0052] The feature maps in the first feature map set are weighted summed with the corresponding weights and passed through an activation function to obtain a class activation map corresponding to the first feature map set.

[0053] In a third aspect, the present invention provides a cataract classification device, comprising:

[0054] An image acquisition module, wherein the image acquisition module is used to acquire a fundus image to be tested;

[0055] A classification network module, wherein the classification network module is used to input the fundus image to be tested into a trained cataract classification network to obtain an output feature vector;

[0056] A classification and discrimination module is used to obtain type information of the fundus image to be tested according to the output feature vector.

[0057] Since the cataract classification device of the third aspect can execute any cataract classification method of the first aspect and / or any lesion point visualization method of the second aspect, it has all the beneficial effects of the first aspect of the present invention.

[0058] In a fourth aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to the cataract classification method described in any one of the first aspect and / or the lesion visualization method described in any one of the second aspect.

[0059] Since the processor of the fourth aspect executes the computer program as any cataract classification method of the first aspect and / or any lesion visualization method of the second aspect, it has all the beneficial effects of the first aspect of the present invention.

[0060] In a fifth aspect, the present invention provides a computer storage medium comprising computer executable instructions stored therein, wherein the computer executable instructions are used to execute the cataract classification method as described in any one of the first aspect and / or the lesion visualization method as described in any one of the second aspect.

[0061] Since the computer storage medium of the fifth aspect can execute any cataract classification method of the first aspect and / or any lesion visualization method of the second aspect, it has all the beneficial effects of the first aspect of the present invention.

[0062] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0064] Figure 1 is a structural block diagram of a cataract classification device provided by an embodiment of the present application;

[0065] Figure 2 is a diagram of the main steps of a cataract classification method provided by an embodiment of the present application;

[0066] Figure 3 is a diagram of the training steps of a cataract classification network provided in one embodiment of the present application;

[0067] Figure 4 is a method step diagram of preprocessing of a cataract classification method provided by an embodiment of the present application;

[0068] Figure 5 is another method step diagram of preprocessing of a cataract classification method provided by an embodiment of the present application;

[0069] Figure 6 It is a method step diagram of cataract classification network processing of a cataract classification method provided by an embodiment of the present application;

[0070] Figure 7is another method step diagram of cataract classification network processing of a cataract classification method provided by an embodiment of the present application;

[0071] Figure 8 It is a method step diagram of a convolutional network module processing of a cataract classification method provided by an embodiment of the present application;

[0072] Fig. 9 This is a diagram showing the main steps of a lesion visualization method provided by an embodiment of the present application;

[0073] Fig.10 This is a method step diagram for calculating a class activation map of a lesion point visualization method provided by an embodiment of the present application;

[0074] Fig.11 is a structural block diagram of a cataract classification network provided by an embodiment of the present application;

[0075] Fig.12 is a schematic diagram of a feature map generated by a cataract classification network provided by an embodiment of the present application;

[0076] Fig.13 is a schematic diagram of an output feature vector provided by an embodiment of the present application;

[0077] Fig.14 This is a comparison diagram of the visualization of cataract lesions provided in one embodiment of the present application. DETAILED DESCRIPTION

[0078] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the embodiments of the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the embodiments of the present application.

[0079] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0080] It should also be understood that the references to "one embodiment" or "some embodiments" described in the specification of the embodiments of the present application mean that one or more embodiments of the embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0081] Cataracts are eye diseases caused by lens metabolism disorders due to aging, heredity, local nutritional disorders, immune and metabolic abnormalities, trauma, poisoning, radiation, etc., which lead to lens protein denaturation and turbidity. The turbidity of the lens of cataract patients prevents light from being projected onto the retina, causing blurred vision. Cataracts are mainly divided into three types according to the location of the lens turbidity: cortical, nuclear, and posterior subcapsular.

[0082] At present, the LOCSIII classification method is used in clinical practice to observe the slit lens and back reflection images, and evaluate the three types separately. The LOCSIII evaluation method is completely based on the doctor's subjective observation and clinical experience, which is time-consuming and labor-intensive, and many grassroots and community clinics do not have such diagnostic conditions. The three different types of cataracts differ in surgical timing, surgical methods, and patient course characteristics. The judgment of the type of cataract may take up a lot of time, resulting in cataract patients not being able to receive timely treatment.

[0083] Based on this, the embodiments of the present application provide a cataract classification method, a lesion visualization method and related devices. The embodiments of the present application establish a cataract classification network model through known fundus images and their corresponding categories, and realize the classification of different types of cataracts based on fundus images.

[0084] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0085] like Figure 1 As shown, Figure 1 : is a structural block diagram of a cataract classification device provided by an embodiment of the present application. Figure 1 In the example, the cataract classification device includes an image acquisition module 100, a classification network module 200 and a classification discrimination module 300.

[0086] The image acquisition module 100 is in communication connection with the classification network module 200 , and the image acquisition module 100 is used to acquire the fundus image to be tested.

[0087] The classification network module 200 is respectively connected to the image acquisition module 100 and the classification and discrimination module 300 for communication. The classification network module 200 is used to input the fundus image to be tested into the trained cataract classification network to obtain an output feature vector.

[0088] The classification and discrimination module 300 is in communication connection with the classification network module 200 , and the classification and discrimination module 300 is used to obtain type information of the fundus image to be tested according to the output feature vector.

[0089] In this embodiment, the image acquisition module 100 sends the acquired fundus image to be tested to the classification network module 200; the classification network module 200 inputs the acquired fundus image to be tested into the trained cataract classification network to obtain an output feature vector, and sends the output feature vector to the classification discrimination module 300; the classification discrimination module 300 obtains the type information of the fundus image to be tested based on the obtained output feature vector.

[0090] The devices and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0091] It can be understood by those skilled in the art that Figure 1 The device structure shown in the figure does not constitute a limitation on the embodiments of the present application, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0092] exist Figure 1 In the device structure shown, each module can respectively call its stored cataract classification program to execute the cataract-like classification method.

[0093] Based on the above-mentioned device, various embodiments of the cataract classification method of the embodiment of the present application are proposed.

[0094] like Figure 2 As shown, Figure 2 1 is a diagram of the main steps of a cataract classification method provided in an embodiment of the present application. The cataract classification method includes but is not limited to the following steps:

[0095] Step S100: Acquire a fundus image to be tested.

[0096] Step S200: input the fundus image to be tested into the trained cataract classification network to obtain an output feature vector.

[0097] It can be understood that the cataract classification network is generated by training multiple groups of fundus images with known cataract types. As the amount of fundus image data increases, the performance of the cataract classification network improves, and its accuracy in identifying the fundus images to be tested increases.

[0098] Step S300: Obtain type information of the fundus image to be measured according to the output feature vector.

[0099] It can be understood that the cataract classification method can classify multiple types of cataracts. Preferably, the cataract classification method provided in the embodiment of the present application can perform three-classification or four-classification on the output feature vector. For the four-classification, the cataract classification method judges the fundus image to be tested as one of "cortical cataract, nuclear cataract, posterior subcapsular cataract". For the three-classification, the cataract classification method judges the fundus image to be tested as one of "cortical cataract, nuclear cataract, posterior subcapsular cataract".

[0100] It is understandable that, referring to Fig.13 , according to the direction of the line graph corresponding to the output feature vector, the fundus image to be tested is judged.

[0101] It can be understood that the cataract classification method provided in the embodiment of the present application obtains the fundus image to be tested, inputs the fundus image to be tested into a trained cataract classification network, obtains a feature vector, and classifies the fundus image to be tested according to the output feature vector to obtain the type information of the fundus image to be tested. The cataract classification method can obtain the cataract classification result corresponding to the fundus image to be tested, reduces the process of manual judgment, makes the patient's treatment more immediate, and improves the user experience.

[0102] It should be noted that, refer to Figure 3 , Figure 3 : is a step diagram of a training method for a cataract classification network provided in an embodiment of the present application, and the training method for a cataract classification network includes but is not limited to the following steps:

[0103] Step S310: Acquire a plurality of sample fundus images and cataract categories corresponding to the sample fundus images.

[0104] Step S320: input the sample fundus image into the cataract classification network to be trained to obtain a sample feature vector corresponding to the sample fundus image.

[0105] Step S330: classify the sample feature vector, and the obtained sample feature vector corresponds to the cataract category corresponding to the sample fundus image.

[0106] It can be understood that the correspondence between the sample fundus images and the cataract categories needs to be maintained extremely high. The higher the accuracy, the better the performance of the cataract classification network and the stronger the discrimination ability.

[0107] It can be understood that a higher number of sample fundus images will improve the performance of the cataract classification network and enhance the discrimination ability of the cataract classification network.

[0108] It should be noted that, refer to Figure 4 , Figure 4 1 is a method step diagram of preprocessing of a cataract classification method provided by an embodiment of the present application. The preprocessing of the cataract classification method is located between steps S100 and S200, and the method includes but is not limited to the following steps:

[0109] Step S410: normalize the size of the fundus image to be measured.

[0110] Step S420: normalize the RGB values ​​of the fundus image to be measured.

[0111] Step S430: Update the result after the size and RGB value are normalized as the fundus image to be tested.

[0112] It can be understood that the homogenization of the size and RGB values ​​keeps the fundus image to be tested uniform, so that the fundus image to be tested can match the three-color mean of the cataract classification network, which is convenient for post-processing.

[0113] It should be noted that, refer to Figure 5 , Figure 5 FIG. 4 is another method step diagram of preprocessing of a cataract classification method provided by an embodiment of the present application, wherein step S410 includes but is not limited to the following steps:

[0114] Step S411: unify the size of the fundus image to be tested to N 0 *N 0 , where N 0 is a positive integer greater than or equal to 1.

[0115] It is understandable that the uniformity of the fundus image to be tested is achieved by cutting the upper and lower edges of the fundus image to be tested. In addition, the image can be cut by any combination of the four edges of the fundus image to be tested, for example, the left and right edges of the fundus image to be tested are cut, or the four edges of the fundus image to be tested are cut at the same time, and the fundus image to be tested can also be proportionally reduced or enlarged to achieve uniformity of the size of the fundus image to be tested. For the fundus image to be tested, any combination of uniformization methods can also achieve uniformity of size.

[0116] In one embodiment, the size of the fundus image to be measured is unified as 224*224.

[0117] Step S410 includes but is not limited to the following steps:

[0118] Step S421: Calculate the RGB value of the fundus image to be tested according to the preset RGB mean and RGB variance.

[0119] The RGB value of the fundus image to be tested can be expressed as:

[0120] x = (x - mean(x)) / std(x)

[0121] Among them, mean is the preset RGB mean, std is the preset RGB variance, and the fundus image to be tested after the size and RGB value are normalized is a 3-channel N 0 *N 0 structure.

[0122] It can be understood that the preset RGB mean and RGB variance are obtained by processing the sample fundus images during the cataract classification network training process, the RGB mean is the mean of all sample fundus images, and the RGB variance is the variance of all sample fundus images.

[0123] It should be noted that, refer to Figure 6 and Fig.11 , Figure 6 1 is a method step diagram of cataract classification network processing of a cataract classification method provided by an embodiment of the present application, wherein step S200 includes but is not limited to the following steps:

[0124] Step S210: Expand the fundus image to be tested through a multi-level convolutional network module to obtain a first feature atlas, wherein the first feature atlas is N of M channels. t *N t Structure, M is a positive integer greater than or equal to 1, N t is a positive integer greater than or equal to 1.

[0125] It is understandable that by expanding the fundus image to be tested through a multi-level convolutional network module, the structure of the obtained first feature atlas can meet the requirements and facilitate the calculation of the output feature vector.

[0126] Step S220: Pass the first feature atlas through a pooling layer, a fully connected layer, and an activation function in sequence to obtain an output feature vector.

[0127] It can be understood that the pooling layer passed by the first feature atlas is an average pooling layer, which can reduce the dimension of the output vector and extract better features with stronger semantic information. Average pooling can retain background information, which is convenient for later identification of the type of cataract.

[0128] Understandably,

[0129] It should be noted that the number of channels of the convolutional network module increases step by step, and / or the size of the feature map generated by the convolutional network module decreases step by step, so that the size of the feature map in the first feature map set satisfies: N t =N 0 / 2 s , s is the number of convolutional network modules, and the size of the fundus image to be tested is N 0 *N 0 .

[0130] It can be understood that in the process of expanding the fundus image to be tested through a multi-level convolutional network module to obtain a first feature map set, when the size of the fundus image to be tested meets the requirements, the number of channels of the convolutional network module increases step by step, or, when the number of channels of the fundus image to be tested meets the requirements, the size of the feature map generated by the convolutional network module decreases step by step, or, when the size and number of channels of the fundus image to be tested do not meet the requirements, the number of channels of the convolutional network module increases step by step, and the size of the feature map generated by the convolutional network module decreases step by step.

[0131] It should be noted that, refer to Figure 7 and Fig.11 , Figure 7 This is another method step diagram of cataract classification network processing of a cataract classification method provided by an embodiment of the present application, and the fundus image to be tested is a 3-channel N*N structure, and step S210 includes but is not limited to the following steps:

[0132] Step S211: Pass the fundus image to be tested through the first-level convolutional network module to obtain a second feature atlas, wherein the second feature atlas is M / 2 5 Channel N / 2 2 *N / 2 2 structure.

[0133] Step S212: The second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas, wherein the third feature atlas is M / 2 3 Channel N / 2 2 *N / 2 2 structure.

[0134] Step S213: Pass the third feature atlas through the third-level convolutional network module to obtain a fourth feature atlas, wherein the fourth feature atlas is M / 2 2 Channel N / 2 3 *N / 2 3 structure.

[0135] Step S214: Pass the four feature atlases through the fourth-level convolutional network module to obtain a fifth feature atlas, wherein the fifth feature atlas is N / 2 of the M / 2 channels. 4 *N / 2 4 structure.

[0136] Step S215: Pass the fifth feature atlas through the fifth-level convolutional network module to obtain the first feature atlas, where N t =N 0 / 2 5

[0137] It is understandable that, referring to Fig.12 For the fundus image to be tested with a structure of 224*224 and 3 channels, the fundus image to be tested is passed through the first-level convolutional network module to obtain a second feature atlas with a structure of 64 channels and 56*56; the second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas with a structure of 256 channels and 56*56; the third feature atlas is passed through the third-level convolutional network module to obtain a fourth feature atlas with a structure of 28*28 and 512 channels; the fourth feature atlas is passed through the fourth-level convolutional network module to obtain a fifth feature atlas with a structure of 14*14 and 1024 channels; the fifth feature atlas is passed through the fifth-level convolutional network module to obtain a first feature atlas with a structure of 7*7 and 2048 channels.

[0138] It should be noted that, refer to Figure 8 , Figure 8 This is a method step diagram of a convolutional network module processing of a cataract classification method provided by an embodiment of the present application, wherein step S211 includes but is not limited to the following steps:

[0139] Step S216: passing the fundus image to be measured through the convolution kernel.

[0140] Step S217, the fundus image to be tested that has passed the convolution kernel passes through a batch normalization layer, an activation function, and a maximum pooling layer in sequence to obtain a second feature atlas.

[0141] It can be understood that the first-level convolutional network module includes a convolution kernel, a batch normalization layer, an activation function and a maximum pooling layer, and the first-level convolutional network is used to expand the fundus image to be measured through steps S217 to S217.

[0142] In one embodiment, the fundus image to be tested with a structure of 64 channels 112*112 is input into a 64-channel convolution kernel of 7*7 with a step size of 2, and then the fundus image to be tested after the convolution kernel is successively passed through a batch normalization layer, a RELU activation function and a maximum pooling layer with a step size of 2 to obtain a second feature atlas with a structure of 64 channels 56*56, wherein the maximum pooling layer is implemented by a 3*3 matrix.

[0143] It is understandable that, referring to Fig.12 ,The first four images in the first column are the images of the fundus image to be tested passing through the convolution kernel, batch normalization layer, activation function and maximum pooling layer.

[0144] It should be noted that the second-level convolutional network module, the third-level convolutional network module, the fourth-level convolutional network module and the fifth-level convolutional network module respectively include multiple residual modules.

[0145] It can be understood that the embodiment of the present application does not limit the number of residual modules in the second-level convolutional network module, the third-level convolutional network module, the fourth-level convolutional network module, and the fifth-level convolutional network module, and the number of residual modules can be changed according to actual conditions.

[0146] In one embodiment, the second-level convolutional network module includes 3 residual modules, the third-level convolutional network module includes 4 residual modules, the fourth-level convolutional network module includes 6 residual modules, and the fifth-level convolutional network module includes 3 residual modules.

[0147] It is understandable that, referring to Fig.12 , the first row from left to right is the image of the fundus image to be tested passing through the convolution kernel, batch normalization layer, activation function, and maximum pooling second-level convolution network. The second row from left to right is the image passing through the third-level convolution network, the fourth-level convolution network, the fifth-level convolution network, the average pooling layer, and the fully connected layer.

[0148] It is understandable that, referring to Figure 12 to Figure 13 , Fig.13 The images in the figure are the output feature vectors corresponding to cortical cataract, nuclear cataract, and posterior subcapsular cataract from top to bottom. In the horizontal axis, "1" represents cortical cataract, "2" represents nuclear cataract, and "3" represents posterior subcapsular cataract. The specific type of cataract can be determined based on the output feature vector.

[0149] Reference Figures 11 to 13 The fundus image to be tested is preprocessed to achieve uniform size and RGB value, and then the fundus image to be tested is input into the trained cataract classification network to obtain an output feature vector. The cataract classification network includes a multi-level convolutional network module and a pooling layer, a fully connected layer, and an activation function. The multi-level convolutional network module is used to expand the image. According to the output feature vector, the type information of the fundus image to be tested can be obtained. The cataract classification method reduces the process of manual judgment, makes the patient's treatment more immediate, and improves the user experience.

[0150] In addition, if Fig. 9 As shown, the present application embodiment provides a lesion point visualization method, including but not limited to the following steps:

[0151] Step S510: Acquire the fundus image to be tested.

[0152] Step S520: Pass the fundus image to be tested through the multi-level convolutional network module in the trained cataract classification network to obtain a first feature atlas.

[0153] It can be understood that the multi-level convolutional network module in the cataract classification network is the multi-level convolutional network module of steps S310 to S330 and steps S210 to S220.

[0154] Step S530: Calculate a class activation map corresponding to the first feature atlas according to the first feature atlas.

[0155] Step S540: superimpose the class activation map with the fundus image to be tested to obtain a target image for visualizing the lesion point.

[0156] Step S550: output the target image.

[0157] It is understandable that, referring to Fig.14 The first row is the fundus image to be tested corresponding to nuclear cataract and the fourth feature atlas, the fifth feature atlas, and the target image corresponding to the first feature atlas. The first row is the fundus image to be tested corresponding to cortical cataract and the fourth feature atlas, the fifth feature atlas, and the target image corresponding to the first feature atlas. The first row is the fundus image to be tested corresponding to posterior subcapsular cataract and the fourth feature atlas, the fifth feature atlas, and the target image corresponding to the first feature atlas. The visualization effect of the first feature atlas is the best, and the location of the cataract lesion can be obtained according to the target image.

[0158] The lesion point visualization method provided in the second aspect of the present invention has at least the following beneficial effects: the fundus image to be tested is input into the multi-level classification network module in the trained cataract classification network, a first feature atlas is calculated, and based on the first feature atlas, its corresponding class activation map is calculated, and the class activation map is superimposed on the fundus image to be tested, so as to obtain a target image for visualizing the lesion points. The visualization of the lesion points in the fundus image to be tested facilitates the user to obtain the location of the lesion points, facilitates later treatment, and improves the user experience.

[0159] It should be noted that, refer to Fig.10 , Fig.10 This is a method step diagram for calculating a class activation map of a lesion visualization method provided by an embodiment of the present application. Step S530 includes but is not limited to the following steps:

[0160] Step S531, calculating a gradient atlas corresponding to the first feature atlas according to the first feature atlas;

[0161] Step S532, averaging the gradient maps in the gradient atlas to obtain weights corresponding to the feature maps in the first feature atlas;

[0162] Step S533: perform weighted summation on the feature maps in the first feature map set and the corresponding weights and pass them through an activation function to obtain a class activation map corresponding to the first feature map set.

[0163] It can be understood that by calculating the class activation map of the first feature atlas and superimposing it with the fundus image to be tested, a target image for visualizing the lesion points can be obtained. Visualization of the lesion points in the fundus image to be tested facilitates the user to obtain the location of the lesion points, facilitates subsequent treatment, and improves the user experience.

[0164] In addition, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a cataract classification method such as steps S100 to S300 and / or a lesion visualization method such as steps S510 to S550 is performed.

[0165] The processor and the memory may be connected via a bus or other means.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The non-transient software program and instructions required to implement the cataract classification method and / or lesion visualization method of the above-mentioned embodiment are stored in the memory. When executed by the processor, the cataract classification method of the above-mentioned embodiment is executed, for example, the above-described Figure 2 Alternatively, the method of visualizing the lesion point in the above embodiment is performed, for example, the method of visualizing the lesion point in the above embodiment is performed. Fig. 9 The method comprises steps S510 to S550.

[0168] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, which enables the processor to execute the cataract classification method in the above embodiment, for example, to execute the above described Figure 2 The method steps S100 to S300, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S411 and S421 of the method, Figure 6 Steps S210 to S220 of the method, Figure 7 Steps S211 to S215 of the method, Figure 8 In another example, the method steps S216 to S217 in the embodiment are executed by a processor in the above embodiment, so that the processor can execute the fundus image enhancement method in the above embodiment, for example, executing the above described Fig. 9 Steps S510 to S550 of the method, Fig.10 Method steps S531 to S533 in.

[0170] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0171] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for classifying cataracts, It is characterized in that include: Acquire the fundus image to be tested; Inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector; Obtaining type information of the fundus image to be measured according to the output feature vector; The step of inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector includes: The fundus image to be tested is expanded through a multi-level convolutional network module to obtain a first feature atlas, wherein the first feature atlas is N of M channels. t *N t Structure, M is a positive integer greater than or equal to 1, N t is a positive integer greater than or equal to 1; Passing the first feature atlas through a pooling layer, a fully connected layer, and an activation function in sequence, obtaining an output feature vector; The number of channels of the convolutional network module increases step by step, and / or the size of the feature map generated by the convolutional network module decreases step by step, so that the size of the feature map in the first feature map set satisfies: N t =N 0 / 2 s , s is the number of levels of the convolutional network module; The fundus image to be tested is a 3-channel N*N structure; the fundus image to be tested is expanded through a multi-level convolutional network module to obtain a first feature atlas, including: The fundus image to be tested is passed through the first-level convolutional network module to obtain a second feature atlas, wherein the second feature atlas is M / 2 5 Channel N / 2 2 *N / 2 2 structure; The second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas, wherein the third feature atlas is M / 2 3 Channel N / 2 2 *N / 2 2 structure; The third feature atlas is passed through the third-level convolutional network module to obtain a fourth feature atlas, wherein the fourth feature atlas is M / 2 2 Channel N / 2 3 *N / 2 3 structure; The fourth feature atlas is passed through the fourth-level convolutional network module to obtain a fifth feature atlas, wherein the fifth feature atlas is N / 2 of the M / 2 channels. 4 *N / 2 4 structure; The fifth feature atlas is passed through the fifth-level convolutional network module to obtain the first feature atlas, where N t =N 0 / 2 5 .

2. The cataract classification method according to claim 1, It is characterized in that The type information of the fundus image to be measured includes at least one of the following fundus image types: cortical cataract, nuclear cataract, posterior subcapsular cataract, and no cataract.

3. The cataract classification method according to claim 1, It is characterized in that The training steps of the cataract classification network include: Acquire multiple sample fundus images and cataract categories corresponding to the sample fundus images; Inputting the sample fundus image into the cataract classification network to be trained to obtain a sample feature vector corresponding to the sample fundus image; The sample feature vector is classified, and the obtained sample feature vector corresponds to the cataract category corresponding to the sample fundus image.

4. The cataract classification method according to claim 1, It is characterized in that Before inputting the fundus image to be tested into the trained cataract classification network, the method includes: Homogenizing the size of the fundus image to be measured; Normalizing the RGB values ​​of the fundus image to be tested; The result after the size and RGB value are normalized is updated as the fundus image to be tested.

5. The cataract classification method according to claim 4, It is characterized in that The step of homogenizing the size of the fundus image to be measured comprises: The size of the fundus image to be tested is unified as N 0 *N 0 , where N 0 is a positive integer greater than or equal to 1; The step of normalizing the RGB values ​​of the fundus image to be tested includes: According to the preset RGB mean and RGB variance, the RGB value of the fundus image to be tested is calculated, wherein the RGB value of the fundus image to be tested can be expressed as: x = (x - mean(x)) / std(x) Wherein, mean is the preset RGB mean, std is the preset RGB variance, and the fundus image to be tested after the size and RGB value are normalized is a 3-channel N 0 *N 0 structure.

6. The cataract classification method according to claim 1, It is characterized in that The step of passing the fundus image to be tested through a first-level convolutional network module to obtain a second feature atlas includes: Passing the fundus image to be tested through a convolution kernel; The fundus image to be tested that has passed the convolution kernel is sequentially passed through a batch normalization layer, an activation function, and a maximum pooling layer to obtain a second feature atlas.

7. The cataract classification method according to claim 1, It is characterized in that The second-level convolutional network module, the third-level convolutional network module, the fourth-level convolutional network module and the fifth-level convolutional network module respectively include multiple residual modules.

8. A method for visualizing lesion points, It is characterized in that include: Acquire the fundus image to be tested; Passing the fundus image to be tested through a multi-level convolutional network module in a trained cataract classification network to obtain a first feature atlas; According to the first feature atlas, calculating a class activation map corresponding to the first feature atlas; Superimposing the class activation map with the fundus image to be tested to obtain a target image for visualizing the lesion point; Outputting the target image; Among them, the first feature atlas is N of M channels t *N t Structure, M is a positive integer greater than or equal to 1, N t is a positive integer greater than or equal to 1; the number of channels of the convolutional network module increases step by step, and / or the size of the feature map generated by the convolutional network module decreases step by step, so that the size of the feature map in the first feature map set satisfies: N t =N 0 / 2 s , s is the number of convolutional network modules; the fundus image to be tested is a 3-channel N*N structure; The method of passing the fundus image to be tested through a multi-level convolutional network module in a trained cataract classification network to obtain a first feature atlas includes: The fundus image to be tested is passed through the first-level convolutional network module to obtain a second feature atlas, wherein the second feature atlas is M / 2 5 Channel N / 2 2 *N / 2 2 structure; The second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas, wherein the third feature atlas is M / 2 3 Channel N / 2 2 *N / 2 2 structure; The third feature atlas is passed through the third-level convolutional network module to obtain a fourth feature atlas, wherein the fourth feature atlas is M / 2 2 Channel N / 2 3 *N / 2 3 structure; The fourth feature atlas is passed through the fourth-level convolutional network module to obtain a fifth feature atlas, wherein the fifth feature atlas is N / 2 of the M / 2 channels. 4 *N / 2 4 structure; The fifth feature atlas is passed through the fifth-level convolutional network module to obtain the first feature atlas, where N t =N 0 / 2 5 .

9. The method for visualizing lesions according to claim 8, It is characterized in that include: The calculating, according to the first feature atlas, a class activation map corresponding to the first feature atlas includes: According to the first feature atlas, calculating a gradient atlas corresponding to the first feature atlas; Averaging the gradient maps of the gradient atlas to obtain weights corresponding to the feature maps in the first feature atlas; The feature maps in the first feature map set are weighted summed with the corresponding weights and passed through an activation function to obtain a class activation map corresponding to the first feature map set.

10. A cataract classification device, It is characterized in that include: An image acquisition module, wherein the image acquisition module is used to acquire a fundus image to be tested; A classification network module, wherein the classification network module is used to input the fundus image to be tested into a trained cataract classification network to obtain an output feature vector; A classification and discrimination module, the classification and discrimination module is used to obtain type information of the fundus image to be tested according to the output feature vector; The step of inputting the fundus image to be tested into a trained cataract classification network to obtain an output feature vector includes: The fundus image to be tested is expanded through a multi-level convolutional network module to obtain a first feature atlas, wherein the first feature atlas is N of M channels. t *N t Structure, M is a positive integer greater than or equal to 1, N t is a positive integer greater than or equal to 1; Passing the first feature atlas through a pooling layer, a fully connected layer, and an activation function in sequence, obtaining an output feature vector; The number of channels of the convolutional network module increases step by step, and / or the size of the feature map generated by the convolutional network module decreases step by step, so that the size of the feature map in the first feature map set satisfies: N t =N 0 / 2 s , s is the number of levels of the convolutional network module; The fundus image to be tested is a 3-channel N*N structure; the fundus image to be tested is expanded through a multi-level convolutional network module to obtain a first feature atlas, including: The fundus image to be tested is passed through the first-level convolutional network module to obtain a second feature atlas, wherein the second feature atlas is M / 2 5 Channel N / 2 2 *N / 2 2 structure; The second feature atlas is passed through the second-level convolutional network module to obtain a third feature atlas, wherein the third feature atlas is M / 2 3 Channel N / 2 2 *N / 2 2 structure; The third feature atlas is passed through the third-level convolutional network module to obtain a fourth feature atlas, wherein the fourth feature atlas is M / 2 2 Channel N / 2 3 *N / 2 3 structure; The fourth feature atlas is passed through the fourth-level convolutional network module to obtain a fifth feature atlas, wherein the fifth feature atlas is N / 2 of the M / 2 channels. 4 *N / 2 4 structure; The fifth feature atlas is passed through the fifth-level convolutional network module to obtain the first feature atlas, where N t =N 0 / 2 5 .

11. An electronic device, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to the cataract classification method as described in claims 1 to 7 and / or the lesion visualization method as described in claims 8 to 9.

12. A computer storage medium, It is characterized in that The method comprises storing computer executable instructions, wherein the computer executable instructions are used to execute the cataract classification method as described in claims 1 to 7 and / or the lesion visualization method as described in claims 8 to 9.