Methods, systems, devices, and storage media for grading diabetic retinopathy

Abstract

The application discloses a diabetic retinopathy grading method, system, device and storage medium, wherein the method comprises the following steps: dividing a to-be-graded image into a plurality of patches with the same size, identifying the lesion probability corresponding to each patch according to the lesion condition of the patch; assigning a corresponding attention weight to the corresponding patch according to the lesion probability, and splicing the attention weights of all the patches to obtain a first attention weight map; predicting the lesion position and category of the to-be-graded image according to a preset lesion positioning network to obtain a prediction result, and generating a second attention weight map according to the prediction result; fusing the first attention weight map and the second attention weight map to obtain a third attention weight map, and inputting the third attention weight map and the to-be-graded image after weighting into a preset grading network to obtain the diabetic retinopathy grading result of the to-be-graded image; and the application can improve the accuracy of diabetic retinopathy grading.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image grading, in particular to a diabetic retinopathy grading method, system, device and storage medium. BACKGROUND

[0002] Diabetic retinopathy (DR) is a microvascular disease and a very common ophthalmic complication of diabetes. It is one of the main factors that endanger human vision worldwide and seriously affects people's production and life. According to the international clinical diabetic retinopathy protocol, the grading of diabetic retinopathy usually depends on the number and size of different related pathological manifestations and complications, and is divided into five grades: no retinopathy (0 grade), mild non-proliferative retinopathy (1 grade), moderate retinopathy (2 grade), severe non-proliferative retinopathy (3 grade) and proliferative retinopathy (4 grade). Non-proliferative is the early stage of DR, while proliferative retinopathy is the late stage of DR. Patients with proliferative diabetic retinopathy have a 60% chance of severe visual decline within five years. Regular fundus screening helps to detect the disease and reduce the risk of blindness. According to the first world vision report released by the World Health Organization, at least 2.2 billion people worldwide have visual impairment or blindness, of which at least 1 billion people have visual impairment that can be prevented or has not been addressed.

[0003] Traditional DR auxiliary detection methods mainly extract pathological features based on morphological methods and further design classifiers to classify and diagnose DR. At present, it is extremely difficult for traditional methods to process a large amount of multi-element data, and there is still a high misdiagnosis rate. SUMMARY

[0004] The present application aims to solve one of the technical problems in the prior art. To this end, the present application provides a diabetic retinopathy grading method, system, device and storage medium, which can further refine the attention of pathological changes when plaques exist multiple pathological changes, improve the utilization of pathological information through weighted optimization method, improve the accuracy of classification, and fuse the weighted optimization attention weight map and the attention weight map of pathological positioning into the grading network, so that the grading network obtains more complete retinal deep features, and improves the accuracy and robustness of diabetic retinopathy grading.

[0005] In a first aspect, embodiments of the present application provide a diabetic retinopathy grading method, the method comprising:

[0006] obtaining a diabetic retinopathy image to be graded;

[0007] dividing the image to be graded into a plurality of patches of the same size, identifying a lesion probability corresponding to each patch according to a lesion condition of the patch, assigning a corresponding attention weight to the corresponding patch according to the lesion probability, and splicing the attention weights of all the patches to obtain a first attention weight map;

[0008] predicting a lesion position and a lesion category of the image to be graded according to a preset lesion positioning network to obtain a prediction result, and generating a second attention weight map according to the prediction result;

[0009] fusing the first attention weight map and the second attention weight map to obtain a third attention weight map, and inputting the third attention weight map and the image to be graded after weighting into a preset grading network to obtain a diabetic retinopathy grading result of the image to be graded.

[0010] The diabetic retinopathy grading method according to the embodiments of the present application has at least the following beneficial effects:

[0011] By dividing the image to be graded into a plurality of patches of the same size, the image is unified in format and size, and a unified standard is established, which facilitates subsequent splicing to obtain a first attention weight map. By assigning an attention weight to a patch according to a lesion probability, a more refined unbalanced attention degree is given to different degrees of lesion areas in the image to be graded, which avoids the submersion of fine lesion information in the image to be graded. The second attention weight map obtained by the lesion positioning network is fused with the first attention weight map to obtain a third attention weight map, and the third attention weight map and the image to be graded are input into the grading network after weighting, so that the grading network obtains more complete retinal deep features, and the accuracy and robustness of diabetic retinopathy grading are improved.

[0012] According to some embodiments of the present application, the lesion probability includes a four-dimensional probability vector and a label value of the patch; and the four-dimensional probability vector includes probabilities of four dimensions of normal, microaneurysm, hemorrhage and exudation.

[0013] According to some embodiments of the present application, the lesion probability corresponding to each patch is identified according to a preset local network; the local network is composed of convolution layers, max-pooling layers and fully connected layers connected in sequence, the activation function between the convolution layers is a Mish function, and dropout is used after the fully connected layer to prevent overfitting.

[0014] According to some embodiments of the present application, assigning a corresponding attention weight to the corresponding patch according to the lesion probability includes:

[0015] Let the patches divided from the image to be graded be first-level patches;

[0016] If probabilities of at least two dimensions in the first-level four-dimensional lesion probability corresponding to the first-level plaque are greater than a preset probability threshold, the first-level plaque is enlarged to a second-level plaque, the second-level plaque is input into the local network, and a second-level four-dimensional lesion probability corresponding to the second-level plaque is obtained, if probabilities of at least two dimensions in the second-level four-dimensional lesion probability are greater than the preset probability threshold, the second-level plaque is enlarged to a third-level plaque, the third-level plaque is input into the local network, and so on, until a N-level four-dimensional lesion probability corresponding to a N-level plaque does not have probabilities of at least two dimensions greater than the preset probability threshold, wherein a size of the N-level plaque is m times of a size of an (N-1)-level plaque;

[0017] If the N-level four-dimensional lesion probability of the N-level plaque does not have probabilities of at least two dimensions greater than the probability threshold, a label value of the N-level plaque and a maximum probability in a four-dimensional probability vector corresponding to the N-level plaque are obtained, and an attention weight of the N-level plaque is obtained according to the label value and the maximum probability.

[0018] According to some embodiments of the present application, the probability threshold is obtained by:

[0019] It is determined whether there are probabilities of at least two dimensions in the four-dimensional probability vector that are similar, and the probabilities that are similar represent that probabilities of two dimensions are greater than 0.5 and differ by no more than 0.15;

[0020] If there are probabilities of at least two dimensions in the four-dimensional probability vector that are similar, a minimum probability value in the probabilities that are similar is obtained, and the minimum probability value is x;

[0021] It is determined whether the plaque label with the probabilities of at least two dimensions that are similar has multiple lesions, and if yes, y=1, and if not, y=0.5;

[0022] The x and the y are mapped, and the probability threshold is obtained by using a k-means algorithm.

[0023] According to some embodiments of the present application, the enlargement includes:

[0024] The size of the (N-1)-level plaque is enlarged m times by a cubic convolution interpolation algorithm;

[0025] The image of the (N-1)-level plaque after being enlarged m times is up-sampled to obtain m N-level plaques.

[0026] According to some embodiments of the present application, the attention weight is constructed by:

[0027] mapping the label value of the patch and the maximum probability to obtain an integrated value;

[0028] increasing each entry of the integrated value to obtain a weighted matrix of the same size of the patch, the weighted matrix being an attention weight of the patch.

[0029] In a second aspect, embodiments of the present application provide a diabetic retinopathy grading system, comprising:

[0030] an image acquisition module configured to acquire a diabetic retinopathy image to be graded;

[0031] an attention weight generation module configured to divide the image to be graded into a plurality of patches of the same size, identify a lesion probability corresponding to each patch according to a lesion condition of the patch, assign a corresponding attention weight to the corresponding patch according to the lesion probability, and splice the attention weights of all the patches to obtain a first attention weight map;

[0032] a lesion positioning module configured to predict a lesion position and a lesion category of the image to be graded according to a preset lesion positioning network, obtain a prediction result, and generate a second attention weight map according to the prediction result;

[0033] a grading module configured to fuse the first attention weight map and the second attention weight map to obtain a third attention weight map, input the third attention weight map and the image to be graded after weighting to a preset grading network, and obtain a diabetic retinopathy grading result of the image to be graded.

[0034] In a third aspect, embodiments of the present application provide an electronic device, comprising at least one control processor and a memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the method for grading diabetic retinopathy according to the first aspect.

[0035] In a fourth aspect, embodiments of the present application provide a computer storage medium, the computer readable storage medium storing computer executable instructions, and the computer executable instructions are used to enable a computer to perform the method for grading diabetic retinopathy according to the first aspect.

[0036] It should be noted that the beneficial effects of the second aspect to the fourth aspect of the present application and the prior art are the same as those of the method for grading diabetic retinopathy according to the first aspect, which will not be described here.

[0037] Other features and advantages of the present application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings.

[0039] Figure 1 is a flowchart of a method for grading diabetic retinopathy according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of assigning attention weights according to an embodiment of the present application;

[0041] Figure 3 is a flowchart of obtaining a probability threshold according to an embodiment of the present application;

[0042] Figure 4 is a flowchart of patch magnification according to an embodiment of the present application;

[0043] Figure 5 is a flowchart of constructing attention weights according to an embodiment of the present application;

[0044] Figure 6 is a block diagram of a grading system for diabetic retinopathy according to an embodiment of the present application;

[0045] Figure 7 is an electronic device according to an embodiment of the present application;

[0046] Figure 8 is a schematic diagram of a method for grading diabetic retinopathy according to an embodiment of the present application;

[0047] Figure 9 is a schematic diagram of constructing attention weights according to an embodiment of the present application;

[0048] Figure 10 is a schematic diagram of obtaining a probability threshold according to an embodiment of the present application;

[0049] Figure 11 is a schematic diagram of assigning attention weights according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] Embodiments of the present application are described in detail below with reference to the attached drawings. The embodiments of the present application are examples of the present application and are not intended to limit the present application.

[0051] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the sequence of indicated technical features.

[0052] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc. is based on the orientation or position relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0053] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0054] Reference Figure 1 In some embodiments of the present application, a diabetic retinopathy grading method is provided, comprising:

[0055] Step S100, obtaining a diabetic retinopathy image to be graded.

[0056] Step S200, dividing the image to be graded into a plurality of patches of the same size, identifying the lesion probability corresponding to each patch according to the lesion condition of the patch, assigning a corresponding attention weight to the corresponding patch according to the lesion probability, and splicing the attention weights of all patches to obtain a first attention weight map.

[0057] Step S300, predicting the lesion location and category of the image to be graded according to a preset lesion positioning network, obtaining a prediction result, and generating a second attention weight map according to the prediction result.

[0058] Step S400, fusing the first attention weight map and the second attention weight map to obtain a third attention weight map, and inputting the third attention weight map and the image to be graded after weighting into a preset grading network to obtain a diabetic retinopathy grading result of the image to be graded.

[0059] According to the diabetic retinopathy grading method provided by the embodiment of the present application, at least the following beneficial effects are achieved:

[0060] The image is uniformly formatted and sized by dividing the image to be graded into a plurality of patches of the same size, a uniform standard is established, and a first attention weight map is obtained by subsequent splicing; the lesion probability is used to assign attention weights to the patches, giving different degrees of lesion areas in the image to be graded more refined and unbalanced attention, avoiding the submersion of fine lesion information in the image to be graded; the second attention weight map obtained by the lesion positioning network is fused with the first attention weight map to obtain a third attention weight map, and the third attention weight map is weighted with the image to be graded and input into the grading network, so that the grading network obtains more complete retinal deep features, and the accuracy and robustness of diabetic retinopathy grading are improved.

[0061] In some embodiments of the application, the lesion probability includes a four-dimensional probability vector and a label value of the patch; the four-dimensional probability vector includes probabilities of normal, microaneurysm, hemorrhage and exudation.

[0062] The lesion probability includes the probabilities of four dimensions of the lesion, which can handle various lesion conditions, further refine the lesion attention when multiple lesions exist in the patch, improve the utilization of lesion information through the weighted optimization method, and improve the accuracy of classification and diagnosis.

[0063] In some embodiments of the application, the lesion probability corresponding to each patch is identified according to the lesion condition of the patch and is completed by a preset local network; the local network is composed of a convolution layer, a max-pooling layer and a fully connected layer connected in sequence, the activation function between the convolution layers is a Mish function, and dropout is used after the fully connected layer to prevent overfitting.

[0064] The local network is trained to predict the lesion probability of the patch and display it in the form of a four-dimensional probability vector, and the label value of the patch is obtained, the local network can improve the prediction efficiency of the image to be graded, and the prediction accuracy can be continuously improved as the training samples increase, and the structure of the local network is shown in Table 1:

[0065] Table 1

[0066]

[0067] Referring to Figure 2 In some embodiments of the application, the corresponding attention weight is assigned to the corresponding patch according to the lesion probability, including:

[0068] In step S210, the patch divided from the image to be graded is a first-level patch.

[0069] Step S220, if there are at least two dimensions of probabilities in the first-level four-dimensional lesion probability corresponding to the first-level plaque greater than the preset probability threshold, the first-level plaque is enlarged to a second-level plaque, the second-level plaque is input into the local network to obtain a second-level four-dimensional lesion probability corresponding to the second-level plaque, if there are at least two dimensions of probabilities in the second-level four-dimensional lesion probability greater than the preset probability threshold, the second-level plaque is enlarged to a third-level plaque, the third-level plaque is input into the local network, and so on, until there are at least two dimensions of probabilities in the Nth-level four-dimensional lesion probability corresponding to the Nth-level plaque not greater than the preset probability threshold, wherein the size of the Nth-level plaque is m times the size of the (N-1)th-level plaque.

[0070] Step S230, if there are at least two dimensions of probabilities in the Nth-level four-dimensional lesion probability of the Nth-level plaque not greater than the probability threshold, the label value of the Nth-level plaque and the maximum probability in the four-dimensional probability vector corresponding to the Nth-level plaque are obtained, and the attention weight of the Nth-level plaque is obtained according to the label value and the maximum probability.

[0071] When the image to be graded has multiple dimensions of lesions, the plaque image needs to be further refined to make the attention weight distribution more reasonable and accurate. By judging whether the plaque needs to be enlarged through the overall judgment of the four-dimensional lesion probability, the lesion information is further refined, and then the attention weight distribution processing is performed, so as to improve the accuracy of the attention weight distribution, thereby further improving the accuracy of the diabetic retinopathy grading.

[0072] Reference Figure 3 In some embodiments of the present application, the acquisition of the probability threshold comprises:

[0073] Step S211, judging whether there are at least two dimensions of probabilities close to each other in the four-dimensional probability vector, the probabilities close to each other indicating that the probabilities of the two dimensions are both greater than 0.5 and differ by no more than 0.15.

[0074] Step S212, if there are at least two dimensions of probabilities close to each other in the four-dimensional probability vector, the minimum probability value in the probabilities close to each other is obtained, and the minimum probability value is x.

[0075] Step S213, judging whether the plaque label with at least two dimensions of probabilities close to each other has multiple lesions, y=1 if there are multiple lesions, and y=0.5 if there are no multiple lesions.

[0076] Step S214, forming a mapping of x and y, and obtaining the probability threshold by using a k-means algorithm.

[0077] The x value and y value are obtained by judging whether there is multi-dimensional probability proximity and whether there are multiple lesions, and the probability threshold is obtained by mapping the x value and y value, which can consider multiple lesion conditions and multi-dimensional probability proximity conditions, and the probability threshold obtained by the method can accurately determine whether to enlarge the plaque image, and ensure the rationality and accuracy of attention weight distribution.

[0078] Referring to Figure 4 In some embodiments of the application, the enlargement algorithm comprises:

[0079] Step S221, the size of the N-1 level plaque is enlarged m times by a cubic convolution interpolation algorithm.

[0080] Step S222, the image of the N-1 level plaque enlarged m times is up-sampled to obtain m N level plaques.

[0081] The cubic convolution interpolation algorithm can ensure the resolution of the enlarged plaque size, and obtain a plaque with the same enlargement value, so that the enlarged plaque image also has all regions for attention weight distribution judgment, and ensures that the fine lesion information in the image to be graded is not missed.

[0082] Referring to Figure 5 In some embodiments of the application, the construction of attention weight comprises:

[0083] Step S231, mapping the label value of the plaque and the maximum probability to obtain an integrated value.

[0084] Step S232, each entry of the integrated value is increased to a matrix of the same size as the plaque to obtain a weighted matrix, and the weighted matrix is the attention weight of the plaque.

[0085] The construction of attention weight maps the label value of the plaque and the maximum probability to construct a weighted matrix, which ensures accurate distribution of attention weight and can construct an attention weight map that pays more attention to plaques with more serious lesions.

[0086] Referring to Figures 8 to 11 In order to facilitate the understanding of those skilled in the art, one specific embodiment of the application provides a diabetic retinopathy grading method, comprising:

[0087] First, divide the input image to be graded into h*h size plaques by a sliding window of h*h, called first level blocks. The local network is trained to predict the lesion probability of the plaque and display it in the form of a four-dimensional probability vector and the label value of the plaque, and the plaque is divided into four categories, namely 0 (normal), 1 (microaneurysm), 2 (hemorrhage), and 3 (exudation).

[0088] Second step, through the weight optimization module, first of all, whether there are at least two dimensions of the four-dimensional probability vector generated by the local network of the image block of the last stage is greater than the probability threshold value set by the lesion, then a series of calculations are carried out according to the situation to obtain the corresponding weight matrix. The selection method of the probability threshold value is: (1) whether there are at least two dimensions of the four-dimensional probability vector are similar, the similar probability means that the probability of two dimensions is greater than 0.5 and the difference is not greater than 0.15. (2) If there are at least two dimensions of the four-dimensional probability vector are similar, get the minimum probability value in the similar probability, let the minimum probability value be x. (3) whether the plaque label with at least two dimensions of similar probability exists multiple lesions, if there are multiple lesions, let y = 1, if there is no multiple lesions, let y = 0.5. (4) form a mapping with x and y, and get the probability threshold value by using k-means algorithm. When there is only one dimension probability less than the probability threshold value or four dimensions probability are less than the probability threshold value, the maximum output probability mP 1 i = max (P 1 i 1, P 1 i 2, P 1 i 3, P 1 i 4) of the block through the local network softmax layer is reserved, and the label value L 1 of the block is obtained. Based on the two mappings, the corresponding weight matrix (i.e. the first stage block attention weight) of the current first stage block is constructed: (1) the label value mapping of the first stage plaque and the maximum probability mapping in the four-dimensional probability vector of the first stage plaque are integrated as LP 1 = (L 1 + 1)☉mP 1 i . (2) each entry in LP 1 is added to an h*h matrix (corresponding to the size of the first stage block); wherein, L 1 is the label value of the first stage plaque, P 1 represents the four-dimensional probability corresponding to the first stage plaque, i is the plaque index, and mP 1 i represents the maximum probability in the four-dimensional probability corresponding to the first stage plaque.

[0089] When the probability of having at least two dimensions is greater than the probability threshold, the first-level block is input into the weight optimization module, and the image upsampling operation is performed by the magnification algorithm to make the first-level patch magnified four times to obtain the second-level patch. The magnification algorithm used is the cubic convolution interpolation algorithm. The second-level patch is input into the local network to obtain the corresponding second-level four-dimensional lesion probability vector and second-level patch label value. Similarly, after obtaining the four-dimensional probability vector of the second-level patch, it is first judged whether the probability of having at least two dimensions exceeds the probability threshold. If it exists, the attention weight of the third-level patch is further refined in the same way through the weight optimization module. If it does not exist, the corresponding weighted matrix (becomes the second-level block weight) is constructed for the current second-level patch: (1) The label value of the second-level patch is mapped and integrated with the maximum probability in the four-dimensional probability vector of the second-level patch as LP 2 = (L 2 + 1)☉mP 2 i (2) Each entry in LP 2 is increased to an h*h matrix (corresponding to the size of the second-level block).

[0090] When constructing the third-level patch attention weight, the four-dimensional probability vector generated by the third-level patch through the local network is not judged. Through preliminary experiments, it is found that the further refinement of the fourth-level patch attention weight under the current third-level patch attention weight does not obviously improve the model effect. The refinement of the fourth-level patch attention weight leads to the information of some larger lesions being submerged in further segmentation, and some redundant weight information is generated. Therefore, in actual operation, for the generation of the third-level patch attention weight, this paper directly takes the maximum output of the softmax layer and the label value of all third-level blocks to construct the third-level patch attention weight according to the case where only one dimension probability is less than the probability threshold or all four dimensions probability is less than the probability threshold in the first-level patch.

[0091] Finally, the total first attention weight map is obtained by splicing in order.

[0092] Third step, in order to indicate the lesion area which has important indicative significance for patients and doctors in the clinical diagnosis process, and to fuse the lesion positioning information into the attention weight to improve the higher attention of the lesion area in the classification task, a lesion positioning module based on YOLOv3 network is designed. YOLOv3 is an open target detector model that can predict target bounding boxes and their classes, generating a second attention weight map.

[0093] Fourthly, the first attention weight map and the second attention weight map are fused to obtain a third attention weight map, and the third attention weight map and the image to be graded are weighted and input into a preset grading network to obtain a diabetic retinopathy grading result of the image to be graded. The grading result is graded according to the severity of diabetic retinopathy in the international clinical diabetic retinopathy scale. The grading network is a global network. The depth of the global network is deeper than that of the local network. Similar to the local network, BN is used before each ReLU activation, and dropout is used after the fully connected layer. The network structure of the global network is shown in Table 2:

[0094] Table 2

[0095]

[0096] With reference to Figure 6 In one embodiment of the present application, a diabetic retinopathy grading system 1000 is also provided, comprising an image acquisition module 1001, an attention weight generation module 1002, a lesion positioning module 1003 and a grading module 1004, comprising:

[0097] The image acquisition module 1001 is configured to acquire an image to be graded of diabetic retinopathy.

[0098] The attention weight generation module 1002 is configured to divide the image to be graded into a plurality of patches of the same size, identify the lesion probability corresponding to each patch according to the lesion condition of the patch, assign the corresponding attention weight to the corresponding patch according to the lesion probability, and splice the attention weights of all patches to obtain a first attention weight map.

[0099] The lesion positioning module 1003 is configured to predict the lesion position and type of the image to be graded according to a preset lesion positioning network to obtain a prediction result, and generate a second attention weight map according to the prediction result.

[0100] The grading module 1004 is configured to fuse the first attention weight map and the second attention weight map to obtain a third attention weight map, and input the third attention weight map and the image to be graded into a preset grading network after weighting to obtain a diabetic retinopathy grading result of the image to be graded.

[0101] It should be noted that the diabetic retinopathy grading system in the present embodiment and the diabetic retinopathy grading method described above are based on the same inventive concept, and therefore the corresponding contents in the method embodiment are also applicable to the present device embodiment, which will not be described in detail here.

[0102] Reference Figure 7Another embodiment of the present application also provides an electronic device 6000, which can be any type of smart terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0103] Specifically, the electronic device 6000 includes one or more control processors 6001 and a memory 6002. Figure 7 For example, the control processor 6001 and the memory 6002 can be connected through a bus or other means. Figure 7 For example, the control processor 6001 and the memory 6002 can be connected through a bus.

[0104] The memory 6002, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the electronic device in the embodiments of the present application.

[0105] The control processor 6001 executes various functional applications and data processing of the method for grading diabetic retinopathy by running the non-transitory software programs, instructions and modules stored in the memory 6002, i.e. implements the method for grading diabetic retinopathy.

[0106] The memory 6002 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; the data storage area can store data created by using the method for grading diabetic retinopathy, etc. In addition, the memory 6002 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 6002 can optionally include a memory remotely arranged relative to the control processor 6001, and these remote memories can be connected to the electronic device 6000 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0107] When one or more modules are stored in the memory 6002 and executed by the one or more control processors 6001, the method for grading diabetic retinopathy in the above method embodiments is executed, such as the method steps of the above described Figures 1 to 5 .

[0108] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include remotely disposed memory relative to the processor, which can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0109] It should be noted that, since the electronic device in the embodiment and the above-mentioned diabetic retinopathy grading method are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the device embodiment, which will not be described in detail here.

[0110] One embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions for performing the diabetic retinopathy grading method of the above-mentioned embodiment.

[0111] It should be noted that, since the computer-readable storage medium in the embodiment and the above-mentioned diabetic retinopathy grading method are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the device embodiment, which will not be described in detail here.

[0112] As will be appreciated by one of ordinary skill in the art, all or some of the steps, systems, etc. in the above-disclosed methods can be embodied in software, firmware, hardware, and any suitable combination thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media), and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as carrier waves or other transport mechanisms, and includes any information delivery media.

[0113] In the description of the present application, the terms "one embodiment", "some embodiments", "an exemplary embodiment", "an example", "a specific example", or "some examples" and the like mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the present application. The appearances of the above terms in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0114] While embodiments of the application have been shown and described, it is to be understood that various modifications, substitutions, combinations, and variations can be made in the embodiments without departing from the spirit and scope of the application, which is defined by the claims and their equivalents.

Claims

1. A method of grading diabetic retinopathy, characterized by, The method comprises: obtaining an image to be graded of diabetic retinopathy; dividing the image to be graded into a plurality of patches of the same size, identifying a lesion probability corresponding to each patch according to the lesion condition of the patch, assigning a corresponding attention weight to the corresponding patch according to the lesion probability, and splicing the attention weights of all the patches to obtain a first attention weight map; the lesion probability comprises a four-dimensional probability vector and a label value of the patch; the four-dimensional probability vector comprises probabilities of four dimensions of normal, microaneurysm, hemorrhage and exudation; the assigning of the corresponding attention weight to the corresponding patch according to the lesion probability comprises: letting the patches divided from the image to be graded be first-level patches; if there are at least two dimensions of the first-level four-dimensional lesion probability corresponding to the first-level patch whose probabilities are greater than a preset probability threshold, then the first-level patch is enlarged to a second-level patch, the second-level patch is input into a local network to obtain a second-level four-dimensional lesion probability corresponding to the second-level patch, if there are at least two dimensions of the second-level four-dimensional lesion probability whose probabilities are greater than the preset probability threshold, then the second-level patch is enlarged to a third-level patch, the third-level patch is input into the local network, and so on, until there are at least two dimensions of the Nth-level four-dimensional lesion probability corresponding to the Nth-level patch whose probabilities are not greater than the preset probability threshold, wherein the size of the Nth-level patch is m times the size of the (N-1)th-level patch; if there are at least two dimensions of the Nth-level four-dimensional lesion probability of the Nth-level patch whose probabilities are not greater than the probability threshold, obtaining a label value of the Nth-level patch and a maximum probability in the four-dimensional probability vector corresponding to the Nth-level patch, and obtaining an attention weight of the Nth-level patch according to the label value and the maximum probability; performing prediction of lesion location and category on the image to be graded according to a preset lesion positioning network to obtain a prediction result, and generating a second attention weight map according to the prediction result; fusing the first attention weight map and the second attention weight map to obtain a third attention weight map, and inputting the third attention weight map and the image to be graded after weighting into a preset grading network to obtain a diabetic retinopathy grading result of the image to be graded.

2. The method of claim 1, wherein the method is for grading diabetic retinopathy. The identification of the lesion probability corresponding to each patch according to the lesion condition of the patch is completed by a preset local network; the local network is composed of convolution layers, max-pooling layers and fully connected layers connected in sequence, the activation function between the convolution layers is a Mish function, and dropout is used after the fully connected layer to prevent overfitting.

3. The method of claim 1, wherein the method is for grading diabetic retinopathy. The acquisition of the probability threshold comprises: judging whether there are at least two dimensions of probabilities in the four-dimensional probability vector that are similar, the similar probabilities indicating that the probabilities of the two dimensions are both greater than 0.5 and differ by no more than 0.15; if there are at least two dimensions of probabilities in the four-dimensional probability vector that are similar, obtaining a minimum probability value in the similar probabilities, and letting the minimum probability value be x. determining whether the label of the plaque with similar probabilities of at least two dimensions exists multiple lesions, the existence of multiple lesions is y=1, and the non-existence of multiple lesions is y=0.5; The method comprises the steps of: The method comprises the steps of: The method comprises the steps of: The probability threshold is obtained by a mean algorithm.

4. The method of claim 1, wherein the method is for grading diabetic retinopathy. The amplification comprises: amplifying the size of the N-1 level plaque by m times through a cubic convolution interpolation algorithm; the image of the N-1 level plaque amplified by m times is up-sampled to obtain m N level plaques.

5. The method of claim 1, wherein the method is for grading diabetic retinopathy. The construction of the attention weight comprises: mapping the label value of the plaque and the maximum probability to obtain an integrated value; each entry of the integrated value is increased to a matrix with the same size as the plaque to obtain a weighted matrix, which is the attention weight of the plaque.

6. A grading system for diabetic retinopathy, characterized in that, It comprises: an image acquisition module for acquiring a diabetic retinopathy image to be graded; an attention weight generation module for dividing the image to be graded into a plurality of plaques with the same size, identifying the corresponding lesion probability of each plaque according to the lesion condition of the plaque, assigning the corresponding attention weight to the corresponding plaque according to the lesion probability, and splicing the attention weights of all the plaques to obtain a first attention weight map; the lesion probability comprises a label value of the plaque and a four-dimensional probability vector; the four-dimensional probability vector comprises the probabilities of four dimensions of normal, microaneurysm, hemorrhage and exudation; the assignment of the corresponding attention weight to the corresponding plaque according to the lesion probability comprises: the plaque divided from the image to be graded is a first level plaque; if at least two dimensions of the first level four-dimensional lesion probability corresponding to the first level plaque have probabilities greater than a preset probability threshold, the first level plaque is amplified to a second level plaque, the second level plaque is input into a local network to obtain a second level four-dimensional lesion probability corresponding to the second level plaque, if at least two dimensions of the second level four-dimensional lesion probability have probabilities greater than the preset probability threshold, the second level plaque is amplified to a third level plaque, the third level plaque is input into a local network, and so on, until the N level four-dimensional lesion probability corresponding to the N level plaque does not have at least two dimensions with probabilities greater than the preset probability threshold, wherein the size of the N level plaque is m times the size of the N-1 level plaque; if the N level four-dimensional lesion probability of the N level plaque does not have at least two dimensions with probabilities greater than the probability threshold, the label value of the N level plaque and the maximum probability in the four-dimensional probability vector corresponding to the N level plaque are obtained, and the attention weight of the N level plaque is obtained according to the label value and the maximum probability; a lesion positioning module for predicting the lesion position and category of the image to be graded according to a preset lesion positioning network to obtain a prediction result, and generating a second attention weight map according to the prediction result; a grading module for fusing the first attention weight map and the second attention weight map to obtain a third attention weight map, and inputting the third attention weight map and the image to be graded into a preset grading network after weighting to obtain a diabetic retinopathy grading result of the image to be graded.

7. An electronic device, comprising: comprising at least one control processor and a memory communicatively connected to the at least one control processor; the memory storing instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a method of diabetic retinopathy grading according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to perform a method of diabetic retinopathy grading according to any one of claims 1 to 5.

Images