A classification method, device and storage medium for retinal arteries and veins of the fundus
Through the fusion of feature-level and decision-making information of multi-scale deep-supervised full-convolution network and deep-supervised high-resolution network, the accuracy and efficiency of retinal arteriovenous classification in the existing technology are solved, and automated and efficient arteriovenous segmentation is achieved. and significantly improved classification, especially in complex and low contrast areas.
Patent Information
- Application Number
- CN202210223486.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-03-09
AI Technical Summary
The existing fundus retinal arteriovenous classification methods rely on artificial feature extraction and are prone to subjective errors, with low accuracy. Especially when the vascular structure complexity and contrast of retinal images are low, the arteriovenous classification performance is limited.
Using multi-scale deep-supervised full convolutional network and deep-supervised high-resolution network for feature-level and decision-making information fusion, through multi-scale feature fusion network and improved condition generation adversarial network, automatically segmentation and classification of retinal arteries and veins to reduce artificial Intervention.
It improves the accuracy and efficiency of retinal artery and vein classification, reduces labor intensity, and can more accurately segment and classify retinal artery and veins, especially in areas with low contrast and at small marginal blood vessels, significantly improving classification performance.
Smart Images

Figure CN115018756B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of fundus retinal artery and vein classification, and in particular relates to a fundus retinal artery and vein classification method, device and storage medium. Background Art
[0002] Eyes are one of the most important sensory organs for humans to exchange information with the outside world. However, due to factors such as age and eye habits, a variety of ophthalmic diseases such as age-related macular degeneration, cataracts, and glaucoma are induced. At the same time, the health of the eyes is also related to cardiovascular diseases such as diabetes, atherosclerosis, and hypertension. The retinal blood vessels in the fundus are the only deeper microvessels in the human body that can be observed non-invasively and directly. They have a certain correlation with the vascular network structure in other parts of the body in terms of physiological and anatomical characteristics. They have been highly valued by researchers in the field of ophthalmology for many years. The health of the human body can be reflected through blood vessels. Therefore, fundus image examination is an important means for certain systemic diseases such as cranial, cardiovascular, blood, and endocrine diseases. Its examination has a certain reference value for the diagnosis and prediction of certain systemic diseases.
[0003] During the acquisition of retinal images, there are non-uniform changes in color, contrast, and brightness within and between images, making it challenging to establish a complete vascular classification system. As the blood vessels radiate from the center of the optic disc to the edge, the diameter of the blood vessels becomes smaller and the color brightness becomes weaker. In the edge area far from the optic disc, the contrast differences in features such as the diameter and brightness of the arteries and veins are very low, which affects the classification results.
[0004] In the existing literature, the classification methods of retinal arteries and veins can be roughly divided into three categories: feature-based classification methods, graph-based classification methods, and deep learning-based classification methods. Both feature-based and graph-based arteriovenous classification methods are based on traditional machine learning methods, which require professional ophthalmologists to manually extract complex features. The whole process is very time-consuming and prone to subjective errors. The classification performance of arteries and veins depends on the results of vascular segmentation, so large-scale clinical applications are limited. In the past few years, deep learning has been successfully applied to fundus image analysis systems, such as exudate segmentation, arteriovenous segmentation, etc., with its powerful feature extraction capabilities. There are relatively few studies on deep learning in the classification of fundus images. Due to the high complexity of the vascular structure characteristics of retinal images, the classification of arteries and veins is often low in accuracy, and the error rate of pathological information segmentation is high, especially the capillaries with low contrast are easily lost. There is still a certain research gap in the automatic classification method of the retinal arteriovenous system. Summary of the invention
[0005] In order to fill the gap in the prior art, the present invention provides a classification method for fundus retinal arteries and veins, which does not require manual extraction of complex features, can improve classification accuracy, reduce labor intensity, and has high efficiency.
[0006] In order to solve the above technical problems and achieve the above technical effects, the present invention is implemented through the following technical solutions:
[0007] A method for classifying fundus retinal arteries and veins, comprising:
[0008] Acquire color fundus retinal images;
[0009] The fundus retinal images are input into the multi-scale deep supervised fully convolutional network (MDRDU-Net) and the deep supervised high-resolution network (DHRNet) respectively to obtain the processing results required for information fusion;
[0010] The two processing results obtained by the multi-scale deep supervision full convolutional network and the deep supervision high-resolution network are fused according to the feature level fusion or decision level fusion strategy and using the corresponding information fusion technology, thereby achieving the segmentation and classification of the overall structure of the retinal artery and vein and the small blood vessels at the same time.
[0011] Further, classification methods based on feature-level fusion strategies include:
[0012] Combining the multi-scale deep supervised fully convolutional network and the deep supervised high-resolution network to construct a multi-scale feature fusion network (FFNet);
[0013] The multi-scale deep supervision full convolutional network and the deep supervision high-resolution network learn features of different resolutions from different aspects and perform multi-scale fusion, wherein:
[0014] The multi-scale deep supervised fully convolutional network recovers the high-resolution representation from the low-resolution representation generated by the high-to-low resolution network, and connects the intermediate resolution representation through the residual path; the deep supervised high-resolution network maintains the high-resolution representation throughout the process, and gradually increases the high-low resolution switching subnetwork and the multi-resolution subnetwork parallel connection;
[0015] The multi-scale feature fusion network fuses the feature representations of different resolutions learned by the two neural networks to form complementary information, so the two are fused to improve the generalization ability and algorithm performance of the entire fusion network;
[0016] The multi-scale feature fusion network combines the multi-scale deep supervision full convolutional network and part of the framework structure of the deep supervision high-resolution network, and serially fuses the feature map focusing on edge information extracted by the multi-scale deep supervision full convolutional network and the feature map focusing on global information extracted by the deep supervision high-resolution network to obtain fusion features, and then passes through a 1×1 convolutional layer to obtain the final retinal artery and vein classification result;
[0017] At the same time, the output features of the multi-scale deep supervision full convolutional network and the deep supervision high-resolution network are subjected to an additional 1×1 convolutional layer to obtain the retinal artery and vein classification results of the two multi-scale deep supervision full convolutional networks and the deep supervision high-resolution network, and their loss function is calculated. The learning process is constrained together with the loss of the fusion feature output, and the optimized parameters are back-propagated to the entire multi-scale feature fusion network.
[0018] Further, the classification methods based on decision-level fusion strategy include:
[0019] Firstly, for the multi-scale deep supervised fully convolutional network and the deep supervised high-resolution network, two network model parameters are trained using color fundus image datasets respectively, so that the two neural networks are optimized in a statistical sense;
[0020] Then, weights are adaptively assigned to the two neural networks according to the specific classification conditions of different pixels in the test image, and the decision probability outputs of the two neural networks are fused through adaptive weighting to obtain the final retinal artery and vein classification result, thereby obtaining a decision performance that is better than that of a single multi-scale deep supervision full convolutional network or a single deep supervision high-resolution network.
[0021] Furthermore, the method for constructing the multi-scale deep supervision full convolutional network is:
[0022] Some original convolutional layers of the basic network U-Net are replaced with deformable convolutional layers (Defconv), and retinal blood vessels of different shapes and diameters are modeled by adaptively learning local dense receptive fields.
[0023] The residual path (Respath) connection is used to replace the simple skip connection in the original base network U-Net, which consists of a series of convolutional layers with residual connections to eliminate the semantic gap between the encoder and decoder.
[0024] A multi-scale fusion deep supervision module (Dsfusion) is added to constrain the feature expression of each layer, and the features of different levels of the decoder are cascaded to form the final feature map, so as to fuse the feature information at different scales;
[0025] The main network of the multi-scale deep supervised fully convolutional network contains a total of 10 convolutional layers, four of which are deformable convolutional layers. Each convolutional layer includes two 3×3 convolutions, batch normalization (Batch Normalization, BN) and a ReLU. Different numbers of residual modules are used to connect features between the encoder and decoder, and additional upsampling layers and 1×1 convolutional layers are added to each layer of the decoder.
[0026] Furthermore, the overall process of the fundus retinal artery and vein classification method using the multi-scale deep supervision full convolutional network is as follows:
[0027] In the training stage, the training set of the color fundus image dataset is first preprocessed and scaled up to obtain training samples, which are then sent to a multi-scale deep supervised fully convolutional network for training, the network parameters are adjusted, and the best model is selected from the validation set. The multi-scale deep supervised fully convolutional network uses a multi-class cross entropy loss function;
[0028] In the testing phase, the test set of the color fundus image dataset is first preprocessed and expanded in the same manner to obtain a test sample, which is then sent to the trained optimal model to obtain the final fundus retinal artery and vein classification result.
[0029] Furthermore, the specific operations for preprocessing and scaling the training set and test set of the color fundus image dataset are as follows:
[0030] The dataset is augmented by random cropping after preprocessing including grayscale, contrast-constrained adaptive histogram equalization, normalization, and gamma correction.
[0031] Furthermore, the method of improving the multi-scale deep supervised fully convolutional network into an improved conditional generative adversarial network is:
[0032] Using the multi-scale deep supervised fully convolutional network as a generating network;
[0033] Add binary blood vessel segmentation labels to the generative network as conditional constraints for the discriminator, so that the generative network can extract more feature information about the blood vessels themselves while learning the difference between arteries and veins;
[0034] The multi-class cross entropy loss function and Dice loss function are added to the loss function of the generative network to solve the problem of sample category imbalance.
[0035] Furthermore, the overall process of the fundus retinal artery and vein classification method using the improved conditional generative adversarial network is as follows:
[0036] In the training phase, the collected retinal images are preprocessed including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction and data amplification, and then input into the generator network of the conditional generative adversarial network as the first constraint item. The preliminary retinal artery and vein classification results are generated according to the retinal gold standard training. Then, the vascular labels are used as the second condition constraint item, and are respectively spliced with the artery and vein gold standard and the retinal artery and vein classification results output by the generator as data pairs and sent to the discriminator. The local discriminator network is used to estimate whether the input data is a real sample. The generator and the discriminator are trained alternately and iteratively, and the trained generator model is saved.
[0037] During the testing phase, the test data was preprocessed, including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction, and data amplification. Then, the generator model in the conditional generative adversarial network saved after training was loaded and forward reasoning was performed to obtain the second classification results of the retinal arteries and veins of the fundus. All data were labeled into three categories, namely arteries, veins, and background, converting the artery and vein classification task into a multi-class semantic segmentation problem without the need to first segment the retinal blood vessels and then classify the arteries and veins.
[0038] Finally, relevant comparative experiments on the DRIVE dataset verified the efficiency of several model algorithms proposed in this invention in completing the task of fundus retinal artery and vein classification, and can provide technical support for disease diagnosis based on fundus images.
[0039] The present invention also discloses a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute an operation corresponding to any one of the above-mentioned methods for classifying fundus retinal arteries and veins.
[0040] The present invention also discloses a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute an operation corresponding to any one of the above-mentioned methods for classifying fundus retinal arteries and veins.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The present invention takes color fundus retinal images as the research object, utilizes deep learning technology, combines the characteristics of retinal images, and classifies the arteries and veins of color fundus retinal images through a multi-scale deep supervised fully convolutional neural network, or an improved conditional generative adversarial network, or a multi-scale deep supervised fully convolutional network and a deep supervised high-resolution network fusion algorithm. This not only improves the accuracy and efficiency of classification, but also eliminates the need for manual feature extraction, greatly reducing labor intensity.
[0043] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0045] Figure 1 A partial image display diagram of the DRIVE dataset used in the present invention;
[0046] Figure 2 The flowchart of the method for classifying fundus retinal arteries and veins based on MDRDU-Net of the present invention;
[0047] Figure 3 It is the overall structure diagram of MDRDU-Net of the present invention;
[0048] Figure 4 A flow chart of the method for classifying fundus retinal arteries and veins based on an improved conditional generative adversarial network of the present invention;
[0049] Figure 5 The flowchart of the FFNet-based fundus retinal artery and vein classification method of the present invention;
[0050] Figure 6 This is a flow chart of the fundus retinal artery and vein classification method based on decision-level fusion of the present invention.
[0051] Figure 7 This is a comparison chart of the classification results of fundus retinal arteries and veins by the method of the present invention and the popular network. DETAILED DESCRIPTION
[0052] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. The descriptions herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0053] The present invention uses the public database DRIVE and the third-party annotated arteriovenous gold standard for experiments; the DRIVE dataset is a color fundus image library established by the Niemeijer team in 2004 based on the fundus lesion data of 400 diabetic patients aged 25 to 90. The database images were taken with a Canon CR5 non-mydriatic 3CCD camera at a 45-degree field of view, and the pixels of each image are 565×584; the DRIVE dataset contains a total of 40 color fundus images, consisting of 33 normal images and 7 pathological images, which are divided into 20 training sets and 20 test sets, and provide expert segmented binary labels and corresponding masks.
[0054] See also Figure 1 As shown, Figure 1 a represents the color fundus image contained in the DRIVE dataset, Figure 1 b represents the gold standard of arteries and veins contained in the DRIVE dataset. Figure 1 c represents the gold standard for blood vessel segmentation contained in the DRIVE dataset. Figure 1 d represents the mask contained in the DRIVE dataset.
[0055] Since the color brightness contrast of diabetic retinopathy images in the DRIVE dataset is very low and the color of blood vessels is very similar to the background, it is very difficult to classify the arteries and veins in the images. Therefore, all training samples must be preprocessed first and the dataset size must be expanded to increase the difference in features such as the color brightness of arteries and veins and the background. Only in this way can an algorithm network with strong learning and generalization capabilities be trained, and ultimately a relatively accurate retinal arteriovenous classification result can be obtained.
[0056] To address the problem of segmenting small arteries and veins in the retina, this paper first proposes a multi-scale deeply supervised fully convolutional network (MDRDU-Net), which introduces deformable convolutional blocks in U-Net to adapt to the shape and size of blood vessels, better fuses shallow detail features and deep global features through residual connections, and calculates the loss function of classification results of different resolutions by fusing deep supervision modules, so that the network can learn feature representation more accurately.
[0057] MDRDU-Net uses U-Net as the basic structure of the network. In the U-Net network, the jump connection between the encoder and the decoder fuses features at different levels to achieve end-to-end pixel-level classification. However, there are still certain defects when it is applied to the classification task of retinal arteries and veins. This is because the number of U-Net layers is too shallow, resulting in a too simple way of extracting feature information, which cannot adapt to the complex and changeable vascular geometry and color changes in retinal images. In addition, U-Net directly splices and fuses high-level features and low-level features through jump connections, while ignoring the semantic differences between them.
[0058] Therefore, in order to better segment and classify the small arteries and veins in the retina, the present invention makes corresponding changes to the basic network U-Net, replaces some of the original convolutional layers of the basic network U-Net with deformable convolutional layers (Defconv), and models retinal blood vessels of different shapes and diameters by adaptively learning local dense receptive fields; uses residual path (Respath) connection to replace the simple jump connection in the original basic network U-Net, which is composed of a series of convolutional layers with residual connections to eliminate the semantic gap between the encoder and decoder; adds a multi-scale fusion deep supervision module (Dsfusion) to constrain the feature expression of each layer, and at the same time concatenates the features of different levels of the decoder to form the final feature map, so as to fuse the feature information at different scales.
[0059] The overall structure of MDRDU-Net is as follows Figure 3 As shown in , the main network contains 10 convolutional layers, four of which are deformable convolutional layers. Each convolutional layer includes two 3×3 convolutions, batch normalization (BN) and a ReLU. Different numbers of residual modules are used to connect features between the encoder and decoder, and additional upsampling layers and 1×1 convolutional layers are added to each layer of the decoder. Figure 2 As shown, the specific algorithm flow is as follows:
[0060] In the training phase, the training set of the color fundus image dataset is first preprocessed through operations including grayscale, contrast-limited adaptive histogram equalization, normalization, and gamma correction. The dataset is then expanded by random cropping to obtain training samples, which are then sent to MDRDU-Net for training, the network parameters are adjusted, and the best model is selected from the validation set. MDRDU-Net uses a multi-class cross entropy loss function.
[0061] During the testing phase, the test set of the color fundus image dataset is first preprocessed through operations including grayscale, contrast-limited adaptive histogram equalization, normalization, and gamma correction, and then the dataset is expanded by random cropping to obtain test samples, which are then sent to the trained optimal model to obtain the final fundus retinal artery and vein classification results.
[0062] In response to the problem that the contrast between arteries and veins in the retina is low and difficult to distinguish, the present invention secondly proposes an improved conditional generative adversarial network based on MDRDU-Net. Through a unique game training method between the generator and the discriminator, the learning process of the difference features of arteries and veins is optimized; the binary vascular label image is used as the constraint condition of the discriminator, so that the discriminator can learn more vascular features.
[0063] In retinal images, especially in the optic disc area where all microvessels are gathered, the characteristics of arteries and veins are very similar, and MDRDU-Net cannot distinguish arteries and veins well; therefore, in order to adapt to the complex and changeable vascular structure of the retina and realize the task of arteriovenous blood vessel classification in color fundus images, the present invention can also improve MDRDU-Net and construct an improved conditional generative adversarial network.
[0064] This improved conditional generative adversarial network uses MDRDU-Net as the generative network; adds binary vascular segmentation labels to the generative network as conditional constraints of the discriminator, so that the generative network can extract more characteristic information about the blood vessels themselves while learning the difference between arteries and veins; adds multi-class cross entropy loss function and Dice loss function to the loss function of the generative network to solve the problem of sample category imbalance.
[0065] This embodiment of the present invention innovatively adds binary blood vessel segmentation labels to the network, so that the network can extract more characteristic information about the blood vessels themselves while learning the difference between arteries and veins. Figure 4 As shown, the specific algorithm flow is as follows:
[0066] In the training phase, the collected retinal images are preprocessed including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction and data amplification, and then input into the generator network of the conditional generative adversarial network as the first constraint item. The preliminary retinal artery and vein classification results are generated according to the retinal gold standard training. Then, the vascular labels are used as the second condition constraint item, and are respectively spliced with the artery and vein gold standard and the retinal artery and vein classification results output by the generator as data pairs and sent to the discriminator. The local discriminator network is used to estimate whether the input data is a real sample. The generator and the discriminator are trained alternately and iteratively, and the trained generator model is saved.
[0067] During the testing phase, the test data was preprocessed, including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction, and data amplification. Then, the generator model in the conditional generative adversarial network saved after training was loaded and forward reasoning was performed to obtain the second classification results of the retinal arteries and veins of the fundus. All data were labeled into three categories, namely arteries, veins, and background, converting the artery and vein classification task into a multi-class semantic segmentation problem without the need to first segment the retinal blood vessels and then classify the arteries and veins.
[0068] When using MDRDU-Net to segment low-contrast areas and small edge blood vessels in retinal images, although the effect has been improved to a certain extent, there are still misclassified and disappeared blood vessels; therefore, this patent attempts to use information fusion technology to combine two neural networks according to feature-level fusion and decision-level fusion strategies to segment and classify arteries and veins in color fundus images.
[0069] In order to simultaneously segment and classify the overall structure and small blood vessels of the retinal arteries and veins, the present invention finally proposes two information fusion methods at the feature level and decision level respectively, and fuses the multi-scale deep supervised fully convolutional network (MDRDU-Net) and the deep supervised high-resolution network (DHRNet) and applies them to the classification of retinal arteries and veins; by adding a deep supervision module to the high-resolution network to constrain the network middle layer to better learn the feature expression; by combining the advantages of the two networks through the feature fusion network and the adaptive weight decision fusion.
[0070] A method for classifying fundus retinal arteries and veins, comprising:
[0071] Acquire color fundus retinal images;
[0072] The fundus retinal images are input into MDRDU-Net and DHRNet respectively to obtain the processing results required for information fusion;
[0073] The two processing results obtained by MDRDU-Net and DHRNet are fused according to the feature-level fusion or decision-level fusion strategy and using the corresponding information fusion technology, so as to achieve the segmentation and classification of the overall structure of the retinal artery and vein and the small blood vessels at the same time.
[0074] See also Figure 5 As shown, the classification methods based on feature-level fusion strategy include:
[0075] Combine MDRDU-Net and DHRNet to build a multi-scale feature fusion network (FFNet);
[0076] MDRDU-Net and DHRNet learn features of different resolutions from different aspects and perform multi-scale fusion.
[0077] MDRDU-Net recovers high-resolution representation from low-resolution representation generated by high-to-low-resolution network and connects intermediate-resolution representation through residual path; DHRNet maintains high-resolution representation in the whole process and gradually adds high-low resolution switching subnets and multi-resolution subnets to be connected in parallel; in this respect, DHRNet is essentially different from MDRDU-Net;
[0078] FFNet combines the feature representations of different resolutions learned by these two neural networks to form complementary information, so the two are integrated to improve the generalization ability and algorithm performance of the entire fusion network;
[0079] FFNet combines some of the framework structures of MDRDU-Net and DHRNet, and serially fuses the feature map extracted by MDRDU-Net focusing on edge information and the feature map extracted by DHRNet focusing on global information to obtain fused features, and then passes through a 1×1 convolutional layer to obtain the final retinal artery and vein classification results;
[0080] At the same time, the output features of MDRDU-Net and DHRNet are passed through an additional 1×1 convolutional layer to obtain the retinal artery and vein classification results of the two MDRDU-Net and DHRNet, and their loss functions are calculated. Then, the learning process is constrained together with the loss of the fusion feature output, and the optimized parameters are back-propagated to the entire FFNet.
[0081] The more commonly used methods of classifier fusion are feature-level fusion and decision-level fusion. The above methods are different for two neural networks to learn features at different resolutions. The network frameworks of MDRDU-Net and DHRNet are combined into a new network to train parameters. The fused network learns more comprehensive details and global features of the retina, but it also makes the network more complex and requires higher computer data processing capabilities. Therefore, this patent also attempts to fuse two neural networks at the decision level. The probabilities of the three pixel categories on the retinal image learned by the neural network are fused according to the weighted average strategy to achieve more accurate retinal pixel classification results. The decision-level fusion method is easier to implement and does not increase computational overhead.
[0082] See also Figure 6 As shown in Figure 2, the classification methods based on decision-level fusion strategy include:
[0083] First, for MDRDU-Net and DHRNet, the color fundus image datasets were used to train the parameters of the two network models, respectively, so that the two neural networks were statistically optimal.
[0084] Then, weights are adaptively assigned to the two neural networks according to the specific classification of different pixels in the test image, and the decision probability outputs of the two neural networks are fused through adaptive weighting to obtain the final retinal artery and vein classification result, thereby achieving decision performance that is better than that of a single MDRDU-Net or DHRNet.
[0085] The traditional weighted decision-level fusion method only assigns a fixed weight to each neural network according to the final classification effect of each neural network, ignoring the specific classification of each pixel on the image. This patent adopts a fusion algorithm with adaptive weight adjustment, which can adaptively adjust the weights of the pixels on the retinal image one by one and fuse them to obtain the optimal classification result.
[0086] In order to further demonstrate the high efficiency of the several model algorithms proposed in the present invention for retinal arteriovenous classification, the several methods proposed in the present invention are compared with more popular medical image segmentation networks, including U-Net, Attention U-Net, DenseUNet, R2U-Net, etc., on the DRIVE dataset.
[0087] See also Figure 7 As shown, Figure 7 a is the classification result diagram of the arteriovenous gold standard, Figure 7 b is the classification result diagram of U-Net, Figure 7 c is the classification result diagram of R2U-Net, Figure 7 d is the classification result diagram of Attention U-Net. Figure 7 e is the classification result diagram of DenseUNet, Figure 7 f is the classification result graph of MDRDunet, Figure 7 g is the classification result diagram of Improved cGAN, Figure 7 h is the classification result diagram of DHRNet, Figure 7 i is the classification result diagram of FFNet, Figure 7 j is the classification result diagram of the decision-level fusion network.
[0088] As shown in Table 1, Table 1 gives the various index values of different networks for the retinal artery and vein classification task. It can be seen that the various classification methods proposed in the present invention have achieved good results in the classification task of small blood vessels of retinal arteries and veins, and the classification performance is better than the existing more popular networks. Among them, the artery and vein classification method based on decision-level fusion of weight adaptive fusion has the best performance, reaching an accuracy of 96.15%, so that the blood vessel classification results in the vicinity of the retinal optic disc have clearer image texture details, thereby verifying the efficiency of the several model algorithms proposed in the present invention in completing the task of fundus retinal artery and vein classification, and can provide technical support for disease diagnosis based on fundus images.
[0089] Table 1 Comparison of experimental results of various methods (mean ± standard deviation)
[0090] method network Acc(%) Prec(%) Rec(%) F1(%) 1 U-Net 94.83±0.60 79.03±6.81 72.45±4.18 75.59 2 R2U-Net 92.03±0.80 58.57±4.46 55.34±4.21 56.91 3 Attention U-Net 95.68±0.63 79.32±6.90 72.89±4.47 75.97 4 Denseunet 95.78±0.59 79.91±7.06 73.00±4.53 76.30 5 MDRDunet 95.86±0.58 81.45±6.69 73.65±4.93 77.35 6 Improved cGAN 95.99±0.60 82.05±6.46 74.21±4.57 77.93 7 DHRNet 95.81±0.57 79.93±6.34 73.44±3.96 76.55 8 FFNet 96.03±0.51 82.26±6.97 76.59±4.43 79.32 9 Decision-level fusion 96.15±0.47 83.05±7.05 76.90±5.01 79.86
[0091] The present invention also discloses a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to execute an operation corresponding to any one of the above-mentioned methods for classifying fundus retinal arteries and veins.
[0092] The present invention also discloses a computer storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to execute an operation corresponding to any one of the above-mentioned methods for classifying fundus retinal arteries and veins.
[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for classifying retinal arteries and veins of the fundus, characterized in that: include: 1) Obtain color fundus retinal images; 2) Input the fundus retinal image into the multi-scale deep supervised fully convolutional network and the deep supervised high-resolution network respectively to obtain the processing results required for information fusion; 3) The two processing results obtained by the multi-scale deep supervision full convolutional network and the deep supervision high-resolution network are fused according to the feature level fusion or decision level fusion strategy and using the corresponding information fusion technology, so as to achieve the segmentation and classification of the overall structure of the retinal artery and vein and the small blood vessels at the same time; Classification methods based on feature-level fusion strategies include: Combining the multi-scale deep supervision full convolutional network and the deep supervision high-resolution network to construct a multi-scale feature fusion network; The multi-scale deep supervision full convolutional network and the deep supervision high-resolution network learn features of different resolutions from different aspects and perform multi-scale fusion, wherein: The multi-scale deep supervised fully convolutional network recovers the high-resolution representation from the low-resolution representation generated by the high-to-low resolution network, and connects the intermediate resolution representation through the residual path; the deep supervised high-resolution network maintains the high-resolution representation throughout the process, and gradually increases the high-low resolution switching subnetwork and the multi-resolution subnetwork parallel connection; The multi-scale feature fusion network fuses the feature representations of different resolutions learned by the two neural networks to form complementary information, so the two are fused to improve the generalization ability and algorithm performance of the entire fusion network; The multi-scale feature fusion network combines the multi-scale deep supervision full convolutional network and part of the framework structure of the deep supervision high-resolution network, and serially fuses the feature map focusing on edge information extracted by the multi-scale deep supervision full convolutional network and the feature map focusing on global information extracted by the deep supervision high-resolution network to obtain fusion features, and then passes through a 1×1 convolutional layer to obtain the final retinal artery and vein classification result; At the same time, the output features of the multi-scale deep supervision full convolutional network and the deep supervision high-resolution network are subjected to an additional 1×1 convolutional layer to obtain the retinal artery and vein classification results of the two multi-scale deep supervision full convolutional networks and the deep supervision high-resolution network, and the loss function is calculated. The loss function is then used together with the loss of the fusion feature output to constrain the learning process, and the optimized parameters are back-propagated to the entire multi-scale feature fusion network; Classification methods based on decision-level fusion strategies include: Firstly, for the multi-scale deep supervised fully convolutional network and the deep supervised high-resolution network, two network model parameters are trained using color fundus image datasets respectively, so that the two neural networks are optimized in a statistical sense; Then, weights are adaptively assigned to the two neural networks according to the specific classification conditions of different pixels in the test image, and the decision probability outputs of the two neural networks are fused by adaptive weighting to obtain the final retinal artery and vein classification result, thereby obtaining a decision performance that is better than that of a single multi-scale deep supervision full convolutional network or a single deep supervision high-resolution network; The method of improving the multi-scale deep supervised fully convolutional network to an improved conditional generative adversarial network is: Using the multi-scale deep supervised fully convolutional network as a generating network; Add binary blood vessel segmentation labels to the generative network as conditional constraints for the discriminator, so that the generative network can extract more feature information about the blood vessels themselves while learning the difference between arteries and veins; Add multi-class cross entropy loss function and Dice loss function to the loss function of the generated network to solve the problem of unbalanced sample categories; The overall process of the fundus retinal artery and vein classification method using the improved conditional generative adversarial network is as follows: In the training phase, the collected retinal images are preprocessed including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction and data amplification, and then input into the generator network of the conditional generative adversarial network as the first constraint item. The preliminary retinal artery and vein classification results are generated according to the retinal gold standard training. Then, the vascular labels are used as the second condition constraint item, and are respectively spliced with the artery and vein gold standard and the retinal artery and vein classification results output by the generator as data pairs and sent to the discriminator. The local discriminator network is used to estimate whether the input data is a real sample. The generator and the discriminator are trained alternately and iteratively, and the trained generator model is saved. During the testing phase, the test data was preprocessed, including grayscale processing, contrast-constrained adaptive histogram equalization, gamma correction, and data amplification. Then, the generator model in the conditional generative adversarial network saved after training was loaded and forward reasoning was performed to obtain the second classification results of the retinal arteries and veins of the fundus. All data were labeled into three categories, namely arteries, veins, and background, converting the artery and vein classification task into a multi-class semantic segmentation problem without the need to first segment the retinal blood vessels and then classify the arteries and veins.
2. The method for classifying fundus retinal arteries and veins according to claim 1, characterized in that: The construction method of the multi-scale deep supervision full convolutional network is: Some original convolutional layers of the basic network U-Net are replaced with deformable convolutional layers, and retinal blood vessels of different shapes and diameters are modeled by adaptively learning local dense receptive fields. Use residual path connections to replace the simple skip connections in the original base network U-Net, which consists of a series of convolutional layers with residual connections to eliminate the semantic gap between the encoder and decoder; A multi-scale fusion deep supervision module is added to constrain the feature expression of each layer, and the features of different levels of the decoder are cascaded to form the final feature map, so as to fuse the feature information at different scales; The main network of the multi-scale deep supervised fully convolutional network contains a total of 10 convolutional layers, four of which are deformable convolutional layers. Each convolutional layer includes two 3×3 convolutions, batch normalization and a ReLU. Different numbers of residual modules are used to connect features between the encoder and decoder, and additional upsampling layers and 1×1 convolutional layers are added to each layer of the decoder.
3. The method for classifying fundus retinal arteries and veins according to claim 2, characterized in that: The overall process of the fundus retinal artery and vein classification method using the multi-scale deep supervised fully convolutional network is as follows: In the training stage, the training set of the color fundus image dataset is first preprocessed and scaled up to obtain training samples, which are then sent to a multi-scale deep supervised fully convolutional network for training, the network parameters are adjusted, and the best model is selected from the validation set. The multi-scale deep supervised fully convolutional network uses a multi-class cross entropy loss function; In the testing phase, the test set of the color fundus image dataset is first preprocessed and expanded in the same manner to obtain a test sample, which is then sent to the trained optimal model to obtain the final fundus retinal artery and vein classification result.
4. The method for classifying fundus retinal arteries and veins according to claim 3, characterized in that: The specific operations for preprocessing and scaling the training and test sets of the color fundus image dataset are as follows: The dataset is augmented by random cropping after preprocessing including grayscale, contrast-constrained adaptive histogram equalization, normalization, and gamma correction.
5. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform an operation corresponding to the method for classifying fundus retinal arteries and veins as described in any one of claims 1 to 4.
6. A computer storage medium, characterized in that: The computer storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the method for classifying fundus retinal arteries and veins as described in any one of claims 1 to 4.
