An AMD grading system based on macular attention mechanisms and uncertainty

By introducing macular attention mechanism and uncertainty analysis into macular disease detection system, combined with deep learning and Bayesian classification network, the problem of inaccurate detection results in the existing technology is solved, and a safer and more reliable macular disease is achieved diagnosis.

CN112446875BActive Publication Date: 2024-11-08ZHONGKE TAIMING (NANJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011453517.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-11
Publication Date
2024-11-08
Estimated Expiration
2040-12-11

AI Technical Summary

Technical Problem

The existing macular disease detection system based on deep learning algorithms cannot provide accurate detection results, which poses safety risks.

Method used

Adopting an AMD grading system based on macular attention mechanism and uncertainty, through the vascular disc segmentation module, macular area positioning module and classification network module, the attention network and Bayesian deep learning classification network model are used, combined with multiple dropouts of Monteca Luo method outputs the probability value and uncertainty of the lesion type to ensure the safety performance of the model.

Benefits of technology

The accuracy and safety of macular disease detection are improved, and whether manual rediagnosis is required is determined through uncertainty, which enhances the reliability of the model in clinical use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112446875B_ABST
    Figure CN112446875B_ABST
Patent Text Reader

Abstract

The present invention provides an AMD grading system based on macular attention mechanism and uncertainty, and relates to the technical field of macular disease detection. The present invention uses a segmentation network model to obtain an optic disc segmentation image and a blood vessel segmentation image, and then obtains a macular area image based on the obtained optic disc segmentation image and the blood vessel segmentation image, and finally uses an attention network submodule to obtain a multi-channel image, uses a Bayesian deep learning classification network model to extract features, and outputs four groups of probability values ​​and a group of noise corresponding to four lesion types through multiple dropout Monte Carlo; the classification network module gives accidental uncertainty and model uncertainty while finally outputting the model classification result. This ensures the safety performance of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of macular disease detection, and in particular to an AMD grading system based on macular attention mechanism and uncertainty. Background Art

[0002] The macula refers to the circular area with a diameter of about 5 to 6 mm between the posterior pole and the temporal vascular arch. Diseases caused by various reasons and occurring in this area are called macular diseases. Macular diseases, such as age-related macular degeneration (AMD), have seriously affected the quality of life and work of millions of people around the world. There are many types of diseases with complex characteristics, and it is difficult to diagnose them in the early stages. If not diagnosed in time and treated appropriately, these macular diseases will lead to irreversible blurred vision, distorted vision, visual field defects, and in the most serious cases, blindness.

[0003] Existing macular degeneration detection methods are all based on deep learning algorithms;

[0004] However, the existing macular disease detection system cannot give accurate test results and poses a safety hazard. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the shortcomings of the prior art, the present invention provides an AMD grading system based on macular attention mechanism and uncertainty, which solves the problem that the existing macular disease detection system based on deep learning algorithm cannot give accurate detection results and has safety hazards.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] An AMD grading system based on macular attention mechanisms and uncertainty, including:

[0010] The vascular optic disc segmentation module is used to segment the fundus color image to obtain a binary optic disc segmentation image and a vascular segmentation image;

[0011] A macular area positioning module, used for obtaining a macular area image based on the optic disc segmentation image and the blood vessel segmentation image;

[0012] The classification network module is used to take the fundus color image and the corresponding optic disc segmentation image, blood vessel segmentation image, and macular area image as the input of the trained deep learning classification network model based on the attention mechanism, and use the attention network to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image, and then use the main neural network to extract features of the multi-channel image, and through multiple dropout Monte Carlo, output four groups of probability values ​​and one group of noise corresponding to the four lesion types; and obtain the mean and variance of the four groups of probability values, take the lesion type with the largest probability mean as the final classification result, take the mean of the noise as the accidental uncertainty, and take the sum of the variances as the model uncertainty.

[0013] Furthermore, the classification network module includes:

[0014] The attention network submodule is used to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image;

[0015] The main neural network submodule is used to extract features from multi-channel images using the trained main neural network model, and output four sets of probability values ​​and one set of noise corresponding to four lesion types through multiple dropout Monte Carlo.

[0016] The classification result output submodule is used to obtain the mean and variance of the four groups of probability values, take the lesion type with the largest probability mean as the final classification result, take the mean of the noise as the accidental uncertainty, and take the sum of the variances as the model uncertainty.

[0017] Furthermore, the attention network submodule is a third-order attention module, and

[0018] The residual attention mechanism is:

[0019] H i,c (x) = (1 + M i,c (x))×F i,c (x)

[0020] H i,c (x) is the output of the attention network submodule, F i,c (x) is the image tensor feature of the previous layer, M i,c (x) is the attention parameter of the soft mask.

[0021] Furthermore, the trained main neural network model is a Bayesian deep learning classification network model, and 5 output nodes are set, including 4 classification nodes corresponding to lesion classification and one noise node corresponding to image noise;

[0022] The multi-channel image is a 512×512×5 multi-channel image fused from a 512×512 macular area image, a binary blood vessel segmentation image and a 512×512×3 fundus image;

[0023] And training the Bayesian deep learning classification network model includes:

[0024] A. Use Cross-Entropy Loss + aleatoric uncertainty + model uncertainty loss as the loss function;

[0025] B. Use the sub-sampling gradient descent algorithm as the learning algorithm for the convolutional neural network;

[0026] C. Each convolutional neural network is trained for 300 epochs, batch_size is set to 2, the initial learning rate is set to 0.001, the decay coefficient is 0.995, and the final learning rate drops to 0.0001 and stops decreasing. The training is stopped and the Adam optimizer is used for optimization training.

[0027] D. The diagnostic labels are 0, 1, 2 and 3, corresponding to healthy, mild, moderate and severe levels respectively; during the training process, data augmentation is performed on the training data;

[0028] E. During the test, multiple dropout Monte Carlo experiments were performed to obtain the predicted distribution, i.e., the probability values ​​of the four groups corresponding to the four lesion types.

[0029] Furthermore, the macular area positioning module includes:

[0030] An optic disc parameter calculation unit is used to perform ellipse fitting on the optic disc in the optic disc segmentation image to obtain the center and radius of the optic disc, and the major axis and minor axis of the ellipse;

[0031] A macular region of interest calculation unit, used to determine the macular region of interest on the blood vessel segmentation image based on the optic disc radius, and to dilate the optic disc in the optic disc segmentation image;

[0032] A quadrant division unit, used to divide the macular area of ​​interest into four quadrants based on the major axis and the minor axis of the ellipse;

[0033] The preferred main vascular arch calculation unit is used to perform Venn difference between the vascular segmentation image and the dilated optic disc segmentation image, and then perform 8-connected domain calculation, retaining the area with a connected domain pixel number of not less than 30, and obtain a connected domain calculation graph, from which the connected domain with the largest number of pixels is found as the preferred main vascular arch;

[0034] The blood vessel priority ranking unit is used to take the blood vessel widths at the optic disc radius positions of multiple preset multiples in the connected domain calculation map with the optic disc center as the center, and perform linear combination with the corresponding first preset proportional coefficient to obtain the average width of each blood vessel in the connected domain calculation map; and perform linear combination of the number of pixels in the connected domain and the average blood vessel width based on the second preset proportional coefficient, and rank the blood vessels according to the result.

[0035] The candidate main vascular arch calculation unit is used to determine the quadrant where the preferred main vascular arch is located; it is also used to determine, based on the order of the primary and secondary vessels, whether each vessel and the preferred main vascular arch are in the same quadrant on the left and right but in different quadrants on the upper and lower sides. If so, the candidate main vascular arch is obtained; otherwise, the next vessel is determined;

[0036] The macular area calculation unit is used to make a straight line where the long axis is located based on the vascular segmentation image after locating the two main vascular arches, and make a parallel line to the straight line where the long axis is located at a position of 5 times the optic disc radius on one side of the two main vascular arches, and form a closed area with the two main vascular arches; then make a circle fitting for the closed area, and the center of the circle is used as the center of the macular area, and the circle fitting result is used as the location of the macular area;

[0037] The closed area generating unit is used to draw a perpendicular line from the nearest point of the main vascular arch to the straight line where the long axis is located to the straight line where the long axis is located when the straight line where the long axis is located, the parallel line to the straight line where the long axis is located and the two main vascular arches cannot form a closed area; and then form a closed area by the straight line where the long axis is located, the parallel line to the straight line where the long axis is located, the perpendicular line and the two main vascular arches.

[0038] Furthermore, the expansion multiple of the optic disc is 1.2 times, and a square centered on the optic disc is made with a side length of 16 times the radius of the optic disc as the region of interest.

[0039] Furthermore, the blood vessel width is the distance from the outer edge of the blood vessel to the centerline of the blood vessel; the preset multiples are 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9 and 2.0, and the corresponding first preset proportional coefficients are 0.3, 0.3, 0.1, 0.06, 0.06, 0.06, 0.06, 0.06; the second preset proportional coefficients corresponding to the number of pixels in the connected domain and the average width of the blood vessel are 0.3 and 0.7.

[0040] Furthermore, the macular area position calculation module includes:

[0041] The macular positioning unit is used to make a straight line where the long axis is located based on the vascular image after the two main vascular arches are located, and to make a line parallel to the straight line where the long axis is located at a position 5 times the optic disc radius on one side of the two main vascular arches, and form a closed area with the two main vascular arches; then a circle is fitted to the closed area, and the center of the circle is used as the center of the macular area, and the circle fitting result is used as the positioning of the macular area.

[0042] Furthermore, the macular area positioning module also includes:

[0043] The closed area generating unit is used to draw a perpendicular line from the nearest point of the main vascular arch to the straight line where the long axis is located to the straight line where the long axis is located when the straight line where the long axis is located, the parallel line to the straight line where the long axis is located and the two main vascular arches cannot form a closed area; and then form a closed area by the straight line where the long axis is located, the parallel line to the straight line where the long axis is located, the perpendicular line and the two main vascular arches.

[0044] Furthermore, the system also includes:

[0045] The classification result credibility judgment module is used to judge that the prediction credibility of the fundus color image is low when the model uncertainty or the accidental uncertainty is higher than the average uncertainty of the training set by a threshold value, wherein the calculation method of the average uncertainty of the training set is to calculate the uncertainty for each fundus color image and then calculate the expectation.

[0046] (III) Beneficial effects

[0047] The present invention provides an AMD grading system based on macular attention mechanism and uncertainty. Compared with the prior art, it has the following beneficial effects:

[0048] The present invention uses a vascular optic disc segmentation module to obtain an optic disc segmentation image and a vascular segmentation image, calculates the macular area based on the obtained optic disc segmentation image and the vascular segmentation image, fuses the RGB original image, the vascular segmentation image and the corresponding pre-processed macular area into a multi-channel image using a classification network module, extracts features from the multi-channel image, and outputs four groups of probability values ​​and one group of noise corresponding to four lesion types through multiple dropout Monte Carlos; and obtains the mean and variance of the four groups of probability values, takes the lesion type with the largest probability mean as the final classification result, takes the mean of the noise as the accidental uncertainty, and takes the sum of the variances as the model uncertainty; the classification module gives the accidental uncertainty and the model uncertainty while finally outputting the model classification result. The safety performance of the model is guaranteed, and when encountering a diagnostic image that cannot give a very certain result, the two uncertainties can be used to determine whether a manual expert needs to re-diagnose, making the model safer and more reliable in clinical use. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is a system block diagram of an embodiment of the present invention;

[0051] Figure 2 is a training flow chart of the optic disc segmentation network and the blood vessel segmentation network according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of locating the macular area according to an embodiment of the present invention;

[0053] Figure 4 Schematic diagram of a classification network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The embodiments of the present application provide an AMD grading system based on macular attention mechanism and uncertainty, thereby solving the problem that the existing macular disease detection system based on deep learning algorithm cannot give accurate detection results and has potential safety hazards.

[0056] The technical solution in the embodiment of the present application is to solve the above technical problems. The overall idea is as follows: the optic disc segmentation image and the vascular segmentation image are obtained by using the vascular optic disc segmentation module, the macular area is calculated according to the obtained optic disc segmentation image and the vascular segmentation image, the RGB original image, the vascular segmentation image and the corresponding pre-processed macular area are fused into a multi-channel image by using the classification network module, and the multi-channel image is feature extracted, and four groups of probability values ​​and one group of noise corresponding to four lesion types are output through multiple dropout Monte Carlo; and the mean and variance of the four groups of probability values ​​are obtained, and the lesion type with the largest probability mean is taken as the final classification result, the mean of the noise is taken as the accidental uncertainty, and the sum of the variances is taken as the model uncertainty; the classification module gives the accidental uncertainty and the model uncertainty while finally outputting the model classification result. The safety performance of the model is guaranteed. When encountering a diagnostic image that cannot give a very certain result, the two uncertainties can be used to determine whether a manual expert needs to re-diagnose, making the model safer and more reliable in clinical use.

[0057] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] Embodiment 1:

[0059] like Figure 1 As shown, the present invention provides an AMD grading system based on macular attention mechanism and uncertainty, the system comprising:

[0060] The vascular optic disc segmentation module is used to segment the fundus color image to obtain a binary optic disc segmentation image and a vascular segmentation image;

[0061] A macular area positioning module, used for obtaining a macular area image based on the optic disc segmentation image and the blood vessel segmentation image;

[0062] The classification network module is used to take the fundus color image and the corresponding optic disc segmentation image, blood vessel segmentation image, and macular area image as the input of the trained deep learning classification network model based on the attention mechanism, and use the attention network to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image, and then use the main neural network to extract features of the multi-channel image, and through multiple dropout Monte Carlo, output four groups of probability values ​​and one group of noise corresponding to the four lesion types; and obtain the mean and variance of the four groups of probability values, take the lesion type with the largest probability mean as the final classification result, take the mean of the noise as the accidental uncertainty, and take the sum of the variances as the model uncertainty.

[0063] The beneficial effects of this embodiment are:

[0064] The embodiment of the present invention uses two trained segmentation network models to obtain the optic disc segmentation image and the blood vessel segmentation image, and then calculates the macular area based on the obtained optic disc segmentation image and the blood vessel segmentation image, and finally uses the attention network submodule to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image, and uses the trained Bayesian deep learning classification network model to extract features, and outputs four groups of probability values ​​and a group of noise corresponding to four lesion types through multiple dropout Monte Carlo; and obtains the mean and variance of the four groups of probability values, takes the lesion type with the largest probability mean as the final classification result, takes the mean of the noise as the accidental uncertainty, and takes the sum of the variances as the model uncertainty; the classification network module gives the accidental uncertainty and the model uncertainty while finally outputting the model classification result. The safety performance of the model is guaranteed. When encountering a diagnostic image that cannot give a very certain result, the two uncertainties can be used to determine whether a manual expert needs to re-diagnose, making the model safer and more reliable in clinical use.

[0065] The implementation process of the embodiment of the present invention is described in detail below:

[0066] (1) Vascular optic disc segmentation module:

[0067] The vascular optic disc segmentation module uses the architecture of two convolutional neural networks. The first convolutional neural network is a vascular segmentation network. After training, it can output a new image with pixel intensities ranging between 0 and 1 after inputting a retinal fundus image. Each pixel value represents the probability that it belongs to a retinal vessel. The vessel segmentation process can effectively eliminate changes in pigmentation, illumination, and non-vascular lesions. The second convolutional neural network is an optic disc segmentation network. After training, it can output a similar optic disc segmentation probability as the first network after inputting a fundus image.

[0068] The steps of constructing the optic disc segmentation network model and the blood vessel segmentation network model include:

[0069] 1-1) First, it is necessary to obtain labels containing pixel-level annotations of the optic disc and the blood vessels, which can be done by asking ophthalmologists to annotate the fundus images.

[0070] 1-2) The preprocessing module preprocesses the fundus images; specifically, the preprocessing includes uniformly scaling the size to 512×512 without deformation; and then normalizing all images at the pixel level by subtracting the mean and dividing the variance. Since the uneven illumination of the fundus image will affect the segmentation of the two lesions, adaptive histogram equalization and gamma correction are used to make it more robust.

[0071] 1-3) The optic disc segmentation network model and the blood vessel segmentation network model both adopt the blood vessel segmentation network model of the Refine-Unet structure. Since the proportion of the optic disc and blood vessels in each image is very small, but the detection of the optic disc and blood vessels is crucial to the classification of diabetic retinopathy, the Refine-Unet structure can firstly realize image-to-image segmentation. After the model is trained, a fundus image is input, and the corresponding optic disc or blood vessel can be automatically segmented, and the output size is exactly the same as the input size; secondly, it can realize the segmentation of tiny objects and maximize the segmentation accuracy of the optic disc or blood vessel. The output result of the model is a probability map, and each pixel on the map is between 0 and 1, indicating the probability that the pixel belongs to the optic disc or blood vessel.

[0072] Since the input is a preprocessed fundus color image, the input channel is 3 and the output size is 512×512. The downsampling and upsampling of the Refined-Unet network keep the same model times to ensure that the final output size is exactly the same as the input size. In the process of cascade downsampling, the feature map of the same size in the upsampling is convolved once and then cascaded with the downsampling map to ensure better performance of feature fusion. In addition, the model output is a multi-scale output. In the upsampling branch, each upsampling is output after two convolutions. The multi-scale loss is calculated with the corresponding scale label of the different scale outputs to strengthen the loss supervision effect and improve the model segmentation performance.

[0073] 1-4) The preprocessed fundus images containing pixel-level annotations of optic discs and fundus images containing pixel-level annotations of blood vessels are divided into a training set, a validation set, and a test set in a ratio of 70%, 15%, and 15%, respectively.

[0074] When training the optic disc segmentation network model and the blood vessel segmentation network model:

[0075] The preprocessed 512×512×3 fundus image is used as input to train the optic disc segmentation network model and the blood vessel segmentation network model respectively.

[0076] like Figure 2 As shown, the segmentation network training module trains the blood vessel and optic disc segmentation network on the training set separated from the blood vessel and optic disc segmentation datasets.

[0077] The specific details and parameters of the training include:

[0078] A. Use multi-scale cross entropy loss function, each scale uses Cross-Entropy Loss (SoftmaxLoss), and the total loss function is L = ΣL i (1≤i≤4), L i =Cross-Entropy Loss(Y i ,P i );

[0079] B. Due to limited GPU computing resources, sub-sampling is used to train the model;

[0080] C. The training epoch is set to 1000, batch_size is set to 2, the initial learning rate is set to 0.001, the decay coefficient is 0.995, and the final learning rate drops to 0.0001 and stops decreasing, so the training stops.

[0081] D. Use Adam optimizer for optimization training.

[0082] E. Add L2 Weight Decay regularization in each convolution operation to prevent overfitting problems caused by excessive weight parameters.

[0083] F. Since the amount of training data is not large, in order to improve performance, data augmentation is performed during training. Data augmentation includes random rotation, flipping, random Gaussian noise addition, and histogram equalization.

[0084] G. After training the two segmentation network models of optic disc and blood vessels, the weights remain fixed.

[0085] Finally, the structures of the trained optic disc segmentation network model and blood vessel segmentation network model are shown in Table 1:

[0086] Table 1

[0087]

[0088] (2) Macular area positioning module

[0089] Macular localization module, including:

[0090] An optic disc parameter calculation unit is used to perform ellipse fitting on the optic disc in the optic disc segmentation image to obtain the center and radius of the optic disc, and the major axis and minor axis of the ellipse;

[0091] A macular region of interest calculation unit, used to determine the macular region of interest on the blood vessel segmentation image based on the optic disc radius, and to dilate the optic disc in the optic disc segmentation image;

[0092] The quadrant division unit is used to divide the macular area of ​​interest into four quadrants based on the long axis and short axis of the ellipse, wherein the long axis is the nasal side and the temporal side on the left and the short axis is the two main vascular arches on the left.

[0093] Furthermore, the expansion multiple of the optic disc is 1.2 times, and a square centered on the optic disc is made with a side length of 16 times the radius of the optic disc as the region of interest.

[0094] It can be implemented to determine the macular region of interest based on the optic disc segmentation image.

[0095] Since the positions of blood vessels, optic disc, and fundus images are consistent, the determined area positions can be converted to each other on each image. For example, the macula region of interest obtained by the optic disc segmentation image can be located on the blood vessel segmentation image. The same is true for the fitted ellipse, major axis, minor axis, blood vessels, etc.

[0096] The preferred main vascular arch calculation unit is used to perform Venn difference between the vascular segmentation image and the expanded optic disc segmentation image, and then perform 8-connected domain calculation to retain the area with a connected domain pixel number of not less than 30 to obtain a connected domain calculation graph, from which the connected domain with the largest number of connected domain pixels is found as the preferred main vascular arch; the preferred main vascular arch occupies one quadrant.

[0097] The blood vessel priority ranking unit is used to take the blood vessel widths at the optic disc radius positions of multiple preset multiples in the connected domain calculation map with the optic disc center as the center, and perform linear combination with the corresponding first preset proportional coefficient to obtain the average width of each blood vessel in the connected domain calculation map; and perform linear combination of the number of pixels in the connected domain and the average blood vessel width based on the second preset proportional coefficient, and rank the blood vessels according to the result.

[0098] The alternative main vascular arch calculation unit is used to determine the quadrant where the preferred main vascular arch is located; it is also used to determine whether each blood vessel and the preferred main vascular arch are in the same quadrant on the left and right but in different quadrants on the upper and lower sides based on the primary and secondary order of the blood vessels. If so, the alternative main vascular arch is obtained; otherwise, the next blood vessel is judged.

[0099] Among them, the blood vessel width is the distance from the outer edge of the blood vessel to the centerline of the blood vessel; the preset multiples are 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9 and 2.0, and the corresponding first preset proportional coefficients are 0.3, 0.3, 0.1, 0.06, 0.06, 0.06, 0.06, 0.06; the second preset proportional coefficients corresponding to the number of pixels in the connected domain and the average width of the blood vessel are 0.3 and 0.7.

[0100] The preferred main vascular arch and the candidate main vascular arch, namely the superior and inferior temporal vascular arches, can be obtained.

[0101] like Figure 3 As shown, the macular area calculation unit is used to make a straight line where the long axis is located based on the vascular segmentation image after locating the two main vascular arches, and make a parallel line to the straight line where the long axis is located at a position of 5 times the optic disc radius on one side of the two main vascular arches, and form a closed area with the two main vascular arches; then a circle is fitted to the closed area, and the center of the circle is used as the center of the macular area, and the circle fitting result is used as the positioning of the macular area.

[0102] The closed area generating unit is used to draw a perpendicular line from the nearest point of the main vascular arch to the straight line where the long axis is located to the straight line where the long axis is located when the straight line where the long axis is located, the parallel line to the straight line where the long axis is located and the two main vascular arches cannot form a closed area; and then form a closed area by the straight line where the long axis is located, the parallel line to the straight line where the long axis is located, the perpendicular line and the two main vascular arches.

[0103] Finally, the position of the corresponding blood vessels on the segmented image is determined based on the positions of the two main vascular arches in the macular area of ​​interest, and the macular area is located in combination with the long axis and radius.

[0104] (3) Classification network module

[0105] like Figure 4 As shown, the classification network module includes:

[0106] The attention network submodule is used to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image;

[0107] The main neural network submodule is used to extract features from multi-channel images using the trained main neural network model, and output four sets of probability values ​​and one set of noise corresponding to four lesion types through multiple dropout Monte Carlo.

[0108] The classification result output submodule is used to obtain the mean and variance of the four groups of probability values, take the lesion type with the largest probability mean as the final classification result, take the mean of the noise as the accidental uncertainty, and take the sum of the variances as the model uncertainty.

[0109] Among them, the attention network submodule is a 3-stage attention module. Each attention module can be divided into two branches. The upper branch is called the trunk branch, which is the basic structure of the residual network (ResNet). The lower branch is the soft mask branch, and the main part of the soft mask branch is the residual attention learning mechanism. The attention mechanism is composed of down sampling, upsampling, and residual units.

[0110] The residual attention mechanism is:

[0111] H i,c (x) = (1 + M i,c (x))×F i,c (x)

[0112] H i,c (x) is the output of the attention network submodule, F i,c (x) is the image tensor feature of the previous layer, M i,c (x) is the soft mask attention parameter. This constitutes the residual attention module, which can input the image features and the features after enhanced attention into the next module. The F function can select different functions to get results in different attention domains.

[0113]

[0114]

[0115]

[0116] f1(x i,c ) is a direct sigmoid activation function for the image feature tensor, which is the attention of the mixed domain;

[0117] f2(x i,c ) is to directly perform global average pooling on the image feature tensor, so the attention in the channel domain is obtained (similar to SENet[5]);

[0118] f3(x i,c ) is an activation function that finds the average value of the image feature tensor in the channel domain, which is similar to ignoring the information in the channel domain, thereby obtaining attention in the spatial domain.

[0119] The main neural network model is a Bayesian deep learning classification network model, which changes the weights and biases of the ordinary deep convolutional neural network from fixed values ​​to distributions. For the training of the Bayesian deep learning network, given a training set D = {(X1, Y1),…, (X N ,Y N )}, to obtain the posterior distribution of the model weights p(W|X,Y), we can use the Bayesian formula:

[0120]

[0121] The prior distribution p(W) is set to the standard normal distribution. However, since the marginal distribution p(Y|X) needs to be integrated over the entire W, and the deep learning convolutional network has many layers and a large number of weight parameters, the space formed by these weights is very complex, and the integral result is difficult to obtain, so the true weight posterior probability distribution cannot be obtained.

[0122] The posterior distribution of the model is approximated by using the variational inference method, that is, using a simple point distribution q θ (W) approximates the distribution of the posterior probability p(W|X,Y). In order to make the approximate posterior distribution as close to the true posterior distribution as possible, the KL divergence between the two is calculated and optimized to make the KL divergence between the two as small as possible. Minimizing the KL divergence is equivalent to minimizing the negative ELBO (negative evidence lower bound), that is, minimizing -∫q θ (W)log p(Y|X,W)dW+KL[q θ (W)|p(W)], and finally get the best approximate posterior distribution

[0123] However, the variational method used to approximate the posterior in the Bayesian neural network will greatly increase the number of parameters in terms of computation. Optimizing the posterior distribution of the variational inference method is equivalent to optimizing the network model using dropout regularization. Therefore, dropout regularization is used as a Bayesian approximation method to obtain the posterior distribution of the model.

[0124] After obtaining the approximate posterior distribution of the model, when testing, due to the complexity of the approximate posterior distribution, the dropout Monte Carlo simulation method (for example, 50 forward propagations) is used to obtain the sampling The classification result distribution under . By obtaining the mean and variance of the classification results, the classification results and model uncertainty are obtained.

[0125] 3-1) Similarly, it is necessary to obtain fundus images labeled with the four types of lesions; the fundus images can be labeled with AMD grades by ophthalmologists.

[0126] 3-2) Before inputting into the network model, the same preprocessing module performs the same preprocessing on the fundus image for the same reason.

[0127] 3-3) The optic disc segmentation module and the blood vessel segmentation module respectively use the trained optic disc segmentation network model and the blood vessel segmentation network model to segment the preprocessed fundus image annotated with four types of lesions to obtain an optic disc segmentation image and a blood vessel segmentation image.

[0128] 3-4) The optic disc segmentation image and the blood vessel segmentation image obtained above are passed through the macular area calculation module to identify the macular area of ​​each image;

[0129] 3-5) The classification results and uncertainty are obtained by fusing the RGB and macular area detection results through information fusion and attention mechanism using the Bayesian deep learning classification network.

[0130] 3-6) When training the Bayesian deep learning classification network model:

[0131] The 512×512 macular area detection result image, the binary vascular image and the 512×512×3 fundus color photo are fused into a 512×512×5 multi-channel image as input.

[0132] The following method is used to train the Bayesian deep learning classification network model:

[0133] A. Use Cross-Entropy Loss + aleatoric uncertainty + model uncertainty loss (L = LCEL +L aleatoric +L modle ) as the loss function.

[0134] B. Use the sub-sampling gradient descent algorithm as the learning algorithm for the convolutional neural network;

[0135] C. Each convolutional neural network is trained for 300 epochs, and the training parameters and optimizer are the same as those of the segmentation model;

[0136] D. Diagnostic markers of diabetic retinal fundus data range from 0, 1, 2, and 3, corresponding to healthy, mild, moderate, and severe levels, respectively. During the training process, data augmentation is also performed on the training data to make the model performance more robust.

[0137] E. During the test, multiple dropout Monte Carlos are performed, as shown in Table 2, to obtain the predicted distribution, i.e., the probability values ​​of the four groups corresponding to the four lesion types;

[0138] Table 2

[0139]

[0140]

[0141] During the test, the lesion type with the highest probability mean is the final classification result, the variance is the model uncertainty, and the model noise node learning obtains the image noise average value as the accidental uncertainty of the image. The two uncertainties obtained from the model are comprehensively considered to compare the results obtained from the test set with the average uncertainty of the training set to obtain the credibility of the results, where the calculation method of the average uncertainty of the training set is to calculate the uncertainty for each image and then calculate the expectation.

[0142] The specific structure of each layer of the network structure of the finally trained Bayesian deep learning classification network model is shown in Table 3;

[0143] Table 3

[0144]

[0145]

[0146] (4) After the optic disc segmentation network model, blood vessel segmentation network model, and Bayesian deep learning classification network model are trained, the system can be used for classification. The process is as follows: Figure 4 As shown:

[0147] S1, obtaining a fundus image to be classified;

[0148] S2, preprocessing the fundus image;

[0149] S3, the vascular optic disc segmentation module uses the preprocessed fundus image to be classified as the input of the trained optic disc segmentation network model and the vascular segmentation network model, respectively, to obtain the optic disc segmentation image and the vascular segmentation image;

[0150] S4, a macular area positioning module performs macular area positioning based on the optic disc segmentation image and the blood vessel segmentation image to obtain a macular area image;

[0151] S5. The classification network module takes the fundus image to be classified, the corresponding optic disc segmentation image, the blood vessel segmentation image and the corresponding preprocessed macular area as input, and fuses them into a multi-channel image through the attention network sub-module. The trained Bayesian deep learning classification network model extracts features from the multi-channel image, and outputs four groups of probability values ​​and one group of noise corresponding to the four lesion types through multiple dropout Monte Carlo. The classification result output sub-module obtains the mean and variance of the four groups of probability values, takes the lesion type with the largest probability mean as the final classification result, takes the mean of the noise as the accidental uncertainty, and takes the sum of the variances as the model uncertainty.

[0152] S6. When the model uncertainty or the accidental uncertainty is 50% higher than the average uncertainty of the training set, the prediction credibility of the image is judged to be low, and the fundus image is sent to an ophthalmologist for re-diagnosis; wherein the average uncertainty of the training set is calculated by calculating the uncertainty for each image and then calculating the expectation.

[0153] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0154] ① The present invention uses two trained segmentation network models to obtain optic disc segmentation images and blood vessel segmentation images, calculates the macular area based on the obtained optic disc segmentation images and blood vessel segmentation images, uses the attention network submodule to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image, uses the trained Bayesian deep learning classification network model to extract features, and outputs four groups of probability values ​​and one group of noise corresponding to four lesion types through multiple dropout Monte Carlo; and obtains the mean and variance of the four groups of probability values, takes the lesion type with the largest probability mean as the final classification result, takes the mean of the noise as the accidental uncertainty, and takes the sum of the variances as the model uncertainty; the classification network module gives the accidental uncertainty and the model uncertainty while finally outputting the model classification result. The safety performance of the model is guaranteed. When encountering a diagnostic image that cannot give a very certain result, the two uncertainties can be used to determine whether a manual expert needs to re-diagnose, making the model safer and more reliable in clinical use.

[0155] ② The convolutional neural network is used to automatically segment the retinal blood vessels and optic disc in the fundus image; the parameters of the optic disc and blood vessels are calculated based on the segmentation results; based on the parameters of the blood vessels and optic disc, the main vascular arch structure of the retinal result is determined, and the position of the macular area is obtained based on its structure. Automatic macular area positioning based on the retinal structure is achieved, saving the time loss of manual annotation for each image.

[0156] It should be noted that, through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment. In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0157] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AMD grading system based on macular attention mechanisms and uncertainty, It is characterized in that The system comprises: The vascular optic disc segmentation module is used to segment the fundus color image to obtain a binary optic disc segmentation image and a vascular segmentation image; A macular area positioning module, used for obtaining a macular area image based on the optic disc segmentation image and the blood vessel segmentation image; The classification network module is used to use the fundus color image and the corresponding optic disc segmentation image, blood vessel segmentation image, and macular area image as the input of the trained deep learning classification network model based on the attention mechanism, and use the attention network to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image, and then use the main neural network to extract features of the multi-channel image, and output four groups of probability values ​​and one group of noise corresponding to four lesion types through multiple dropout Monte Carlo; and obtain the mean and variance of the four groups of probability values, and use the lesion type with the largest probability mean as the final classification result, the mean of the noise as the accidental uncertainty, and the sum of the variances as the model uncertainty; The macular area positioning module comprises: An optic disc parameter calculation unit is used to perform ellipse fitting on the optic disc in the optic disc segmentation image to obtain the center and radius of the optic disc, and the major axis and minor axis of the ellipse; A macular region of interest calculation unit, used to determine the macular region of interest on the blood vessel segmentation image based on the optic disc radius, and to dilate the optic disc in the optic disc segmentation image; A quadrant division unit, used to divide the macular area of ​​interest into four quadrants based on the major axis and the minor axis of the ellipse; The preferred main vascular arch calculation unit is used to perform Venn difference between the vascular segmentation image and the dilated optic disc segmentation image, and then perform 8-connected domain calculation, retaining the area with a connected domain pixel number of not less than 30, and obtain a connected domain calculation graph, from which the connected domain with the largest number of pixels is found as the preferred main vascular arch; a blood vessel primary and secondary sorting unit, for taking the optic disc center as the center, taking the blood vessel widths at the optic disc radius positions of multiple preset multiples in the connected domain calculation map, and performing linear combination with the corresponding first preset proportional coefficient to obtain the average width of each blood vessel in the connected domain calculation map; and performing linear combination of the number of pixels in the connected domain and the average blood vessel width based on the second preset proportional coefficient, and sorting the blood vessels according to the result; The candidate main vascular arch calculation unit is used to determine the quadrant where the preferred main vascular arch is located; it is also used to determine, based on the order of the primary and secondary vessels, whether each vessel and the preferred main vascular arch are in the same quadrant on the left and right but in different quadrants on the upper and lower sides. If so, the candidate main vascular arch is obtained; otherwise, the next vessel is determined; The macular area calculation unit is used to make a straight line where the long axis is located based on the vascular segmentation image after locating the two main vascular arches, and make a parallel line to the straight line where the long axis is located at a position of 5 times the optic disc radius on one side of the two main vascular arches, and form a closed area with the two main vascular arches; then make a circle fitting for the closed area, and the center of the circle is used as the center of the macular area, and the circle fitting result is used as the location of the macular area; The closed area generating unit is used to draw a perpendicular line from the nearest point of the main vascular arch to the straight line where the long axis is located to the straight line where the long axis is located when the straight line where the long axis is located, the parallel line to the straight line where the long axis is located and the two main vascular arches cannot form a closed area; and then form a closed area by the straight line where the long axis is located, the parallel line to the straight line where the long axis is located, the perpendicular line and the two main vascular arches.

2. An AMD grading system based on macular attention mechanism and uncertainty as claimed in claim 1, It is characterized in that The classification network module includes: The attention network submodule is used to fuse the fundus color image and the corresponding blood vessel segmentation image and macular area image to obtain a multi-channel image; The main neural network submodule is used to extract features from multi-channel images using the trained main neural network model, and output four sets of probability values ​​and one set of noise corresponding to four lesion types through multiple dropout Monte Carlo. The classification result output submodule is used to obtain the mean and variance of the four groups of probability values, take the lesion type with the largest probability mean as the final classification result, take the mean of the noise as the accidental uncertainty, and take the sum of the variances as the model uncertainty.

3. An AMD grading system based on macular attention mechanism and uncertainty as claimed in claim 2, It is characterized in that The attention network submodule is a third-order attention module, and The residual attention mechanism is: H i,c (x)=(1+M i,c (x))×F i,c (x) H i,c (x) is the output of the attention network submodule, F i,c (x) is the image tensor feature of the previous layer, M i,c (x) is the attention parameter of the soft mask.

4. An AMD grading system based on macular attention mechanism and uncertainty as claimed in claim 2, It is characterized in that The trained main neural network model is a Bayesian deep learning classification network model, and 5 output nodes are set, including 4 classification nodes corresponding to lesion classification and one noise node corresponding to image noise; The multi-channel image is a 512×512×5 multi-channel image fused from a 512×512 macular area image, a binary blood vessel segmentation image and a 512×512×3 fundus image; And training the Bayesian deep learning classification network model includes: A. Use Cross-Entropy Loss + accidental uncertainty + model uncertainty loss as the loss function; B. Use the sub-sampling gradient descent algorithm as the learning algorithm for the convolutional neural network; C. Each convolutional neural network is trained for 300 epochs, batch_size is set to 2, the initial learning rate is set to 0.001, the decay coefficient is 0.995, and the final learning rate drops to 0.0001 and stops decreasing. The training is stopped and the Adam optimizer is used for optimization training. D. The diagnostic labels are 0, 1, 2 and 3, corresponding to healthy, mild, moderate and severe levels respectively; during the training process, data augmentation is performed on the training data; E. During the test, multiple dropout Monte Carlo experiments were performed to obtain the predicted distribution, i.e., the probability values ​​of the four groups corresponding to the four lesion types.

5. The macular area positioning system based on retinal structure as claimed in claim 1, It is characterized in that The expansion multiple of the optic disc is 1.2 times, and a square centered on the optic disc is made with a side length of 16 times the radius of the optic disc as the region of interest.

6. A macular area positioning system based on retinal structure as claimed in claim 1, It is characterized in that The blood vessel width is the distance from the outer edge of the blood vessel to the centerline of the blood vessel; the preset multiples are 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9 and 2.0, and the corresponding first preset proportional coefficients are 0.3, 0.3, 0.1, 0.06, 0.06, 0.06, 0.06 and 0.06; the second preset proportional coefficients corresponding to the number of pixels in the connected domain and the average width of the blood vessel are 0.3 and 0.

7.

7. A macular area positioning system based on retinal structure as claimed in claim 3, It is characterized in that The macular area position calculation module comprises: The macular positioning unit is used to make a straight line where the long axis is located based on the vascular image after the two main vascular arches are located, and to make a line parallel to the straight line where the long axis is located at a position 5 times the optic disc radius on one side of the two main vascular arches, and form a closed area with the two main vascular arches; then a circle is fitted to the closed area, and the center of the circle is used as the center of the macular area, and the circle fitting result is used as the positioning of the macular area.

8. The macular area positioning system based on retinal structure as claimed in claim 1, It is characterized in that The macular area positioning module also includes: The closed area generating unit is used to draw a perpendicular line from the nearest point of the main vascular arch to the straight line where the long axis is located to the straight line where the long axis is located when the straight line where the long axis is located, the parallel line to the straight line where the long axis is located and the two main vascular arches cannot form a closed area; and then form a closed area by the straight line where the long axis is located, the parallel line to the straight line where the long axis is located, the perpendicular line and the two main vascular arches.

9. An AMD grading system based on macular attention mechanism and uncertainty as claimed in claim 1, It is characterized in that The system further comprises: The classification result credibility judgment module is used to judge that the prediction credibility of the fundus color image is low when the model uncertainty or the accidental uncertainty is higher than the average uncertainty of the training set by a threshold value, wherein the calculation method of the average uncertainty of the training set is to calculate the uncertainty for each fundus color image and then calculate the expectation.