Auxiliary diagnosis method and device for diabetic retinopathy with limited computing resources

Abstract

The present invention provides a method and device for auxiliary diagnosis of diabetic retinopathy with limited computing resources, which adopts a residual neural network model to classify and predict the fundus images of patients, and improves the residual neural network model, improves the residual block and extracts shallow features. The model has a strong relational inductive bias. Therefore, the improved model can learn more features, better express features, and perform auxiliary diagnosis tasks for diabetic retinopathy, which can ultimately improve the accuracy of classification. In addition, the model structure of the present invention is simple and does not require the use of methods such as model mixing, multi-task training, and metric learning. Therefore, compared with existing high-precision models, the model construction of this embodiment is fast and convenient, and the amount of computing consumed by the training process is also small, which is suitable for environments with limited memory and computing resources.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and specifically relates to a method and device for auxiliary diagnosis of diabetic retinopathy with limited computing resources. Background Art

[0002] Diabetic retinopathy (DR), one of the three leading causes of blindness recognized by the World Health Organization, is an eye complication of diabetes and is known as diabetic retinopathy. Early screening for DR is crucial because, while symptoms may be absent in the early stages, by the time patients experience any discomfort, irreversible damage has already occurred. Currently, diagnosis relies on ophthalmologists identifying lesions in fundus images to determine the severity of lesions. However, given the large number of diabetic patients in my country and limited medical resources, regular screening for DR by ophthalmologists is insufficient. With the advancement of deep learning theory, computer-assisted diagnosis (CAD) technologies based on deep learning are increasingly being applied to early screening. Fundus images contain a wealth of information about DR lesions. Deep learning can be used to extract and analyze lesion information and disease characteristics from these images, enabling large-scale, regular CAD screening.

[0003] The recent success of deep neural networks has fueled research in many computer vision tasks, such as image classification, object detection, and semantic segmentation. However, the power of deep models is due in part to their depth and breadth, a prerequisite that not only limits the applications of these models but also hinders their computational complexity and depth, making them difficult to implement on small devices. Therefore, in environments with limited computing resources and memory, a diagnostic method is needed that requires minimal computational resources and memory while maintaining acceptable diagnostic accuracy. Currently, such a method is lacking. Summary of the Invention

[0004] The present invention is made to solve the above problems, and aims to provide a diabetic retinopathy auxiliary diagnosis method and corresponding device that can be applied to environments with limited computing resources and memory and can ensure the accuracy of diagnosis. The present invention adopts the following technical solutions:

[0005] The present invention provides a method for auxiliary diagnosis of diabetic retinopathy with limited computing resources, characterized in that it includes: step S1, collecting fundus images of subjects; step S2, preprocessing the collected fundus images; step S3, using a trained improved residual neural network to classify and predict the preprocessed fundus images, wherein the improved residual neural network includes: multiple densely connected convolutional layers; multiple side output layers for extracting shallow features of the fundus images; and a fusion layer for combining the shallow features from multiple side output layers.

[0006] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such a technical feature, wherein the convolutional layer includes multiple improved residual blocks, the improved residual blocks divide the input into multiple low-dimensional embeddings, and then transform the multiple low-dimensional embeddings separately, and finally aggregate the multiple low-dimensional embeddings.

[0007] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such a technical feature, wherein the fusion layer has at least a softmax layer, the side output layer is composed of an average pooling layer, a fully connected layer and a softmax layer, and the average output sum of multiple side output layers is input into the softmax layer of the fusion layer.

[0008] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such technical features, wherein the number of the side output layers is three, the filter sizes of the three average pooling layers are 7×7, 5×5, and 7×7, respectively, and the step sizes of the three average pooling layers are 3, 3, and 1, respectively.

[0009] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such a technical feature, wherein the multiple convolutional layers include at least a BatchNorm layer and a ReLU layer, and the side output layer is inserted after the BatchNorm layer or the ReLU layer.

[0010] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have a technical feature, wherein the outputs of all the convolutional layers are interconnected.

[0011] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such a technical feature, wherein the label in the training set is the lesion level of diabetic retinopathy, and the trained improved residual neural network model is used to classify and predict the lesion level of the diabetic retinopathy in the fundus image.

[0012] The computing resource-limited diabetic retinopathy auxiliary diagnosis method provided by the present invention may also have such a technical feature, wherein, in step S2, the preprocessing includes: cropping the region of interest, resizing, color and brightness normalization, and data enhancement.

[0013] The present invention provides a diabetic retinopathy auxiliary diagnosis device with limited computing resources, characterized in that it includes: a fundus image acquisition module for acquiring fundus images of a subject; a preprocessing module for preprocessing the acquired fundus images; and a classification prediction module for classifying and predicting the preprocessed fundus images through a trained improved residual neural network, wherein the improved residual neural network includes: multiple densely connected convolutional layers; multiple side output layers for extracting shallow features of the fundus images; and a fusion layer for combining the shallow features from multiple side output layers.

[0014] Functions and effects of the invention

[0015] According to the computing resource-limited diabetic retinopathy auxiliary diagnosis method of the present invention, a residual neural network model is used to classify and predict the patient's fundus images, and the residual neural network model is improved to extract shallow features. The model has a strong relational inductive bias. Therefore, the improved model can learn more features, better express features, and better perform diabetic retinopathy auxiliary diagnosis tasks, which can ultimately improve the classification accuracy. In addition, the model structure is simple and does not require the use of model mixing, multi-task training, and metric learning methods. Therefore, compared with existing high-precision models, the model of the present invention is fast and convenient to build, and the amount of computation consumed by the training process is also small, which is suitable for environments with limited memory and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for auxiliary diagnosis of diabetic retinopathy with limited computing resources according to an embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of an improved residual block in an embodiment of the present invention;

[0018] Figure 3 2 is a schematic diagram of a partial structure of the improved ResNet model in an embodiment of the present invention;

[0019] Figure 4 Schematic diagram of the structure of the improved ResNet model in an embodiment of the present invention;

[0020] Figure 5 This is a structural block diagram of a device for assisting in diabetic retinopathy diagnosis with limited computing resources according to an embodiment of the present invention;

[0021] Figure 6Schematic diagram of the residual block in the prior art.

[0022] Reference numerals:

[0023] Diabetic retinopathy auxiliary diagnosis device 10; fundus image acquisition module 11; preprocessing module 12; classification prediction module 13; storage module 14; control module 15. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the computing resource-limited diabetic retinopathy auxiliary diagnosis method and device of the present invention in combination with embodiments and drawings.

[0025] <Example>

[0026] This embodiment provides a method for auxiliary diagnosis of diabetic retinopathy with limited computing resources, which is applied in an environment with limited computing resources and memory, wherein the operating environment is a single NVIDIA2080Ti.

[0027] Figure 1 This is a flowchart of the auxiliary diagnosis method for diabetic retinopathy with limited computing resources in this embodiment.

[0028] like Figure 1 As shown in FIG, the auxiliary diagnosis method for diabetic retinopathy with limited computing resources specifically includes the following steps:

[0029] Step S1: Acquire fundus images of a subject, wherein the subject is a patient with diabetic retinopathy.

[0030] In this embodiment, the data set of the Kaggle DR competition is used. The data set comes from the diabetic retinopathy detection competition held on the Kaggle website. In this data set, the original training set contains 35,000 fundus images, and the original test set contains 45,000 fundus images. These fundus images are all collected in a real environment and contain most of the imaging conditions during the collection process, which is consistent with the actual diabetic retinopathy screening situation. However, the image quality in this data set is highly differentiated, and there are a large number of fundus image quality problems such as insufficient light input, severe exposure abnormalities, excessive reflections, and inaccurate shooting positions. Therefore, the data set was screened for image quality, and in order to maintain a balance of samples of each lesion level, 20,000 fundus images were selected in the training set as the experimental training set, and 5,000 fundus images were selected in the test set as the experimental test set. In this data set, the label is the lesion level of diabetic retinopathy.

[0031] Step S2: pre-processing the acquired fundus image.

[0032] The fundus images in the dataset come from patients with diabetic retinopathy of different races and age groups. Based on this characteristic of the dataset, in this embodiment, all selected images were preprocessed as follows:

[0033] Cropping Region of Interest (ROI): Since the original images have large black borders which are unnecessary for classification, the central ROI of each image is obtained by cropping.

[0034] Resizing: Most images in the dataset have a resolution of around 3000×3000, so the cropped images were resized to 512x512. This is a trade-off between GPU processing power and the complex features of color fundus images. If the image size is too small, it will be difficult to extract the features of some tiny discriminative regions, such as microaneurysms.

[0035] Color and brightness normalization: Fundus images captured by different imaging environments and camera devices can have significant variations in brightness and color, which is irrelevant to DR classification. Therefore, in this embodiment, color and brightness normalization is performed on all images. This process normalizes images with varying colors and brightness, thereby reducing this noise to a certain extent during network training.

[0036] In order to prevent overfitting, the following data augmentation processing is applied to the experimental training set during training:

[0037] Flip: Randomly flip the fundus image horizontally or vertically, or not flip it.

[0038] Smoothing: Fundus images are randomly smoothed using a 3×3 Gaussian filter.

[0039] Contrast Adjustment: Randomly adjust the contrast of the fundus image.

[0040] Rotate: Rotate the fundus image randomly by an angle.

[0041] Cropping: The fundus image of size 512 × 512 was cropped to a fundus image of size 448 × 448 with random offsets.

[0042] Step S3: Use the trained improved residual neural network model to perform classification prediction on the preprocessed fundus image.

[0043] In this embodiment, the existing residual neural network model ResNet is improved. Due to the limitations of actual environmental memory and computing resources, the premise of the improvement is not to increase the complexity of the model and to improve the classification accuracy of the model.

[0044] At the same time, by utilizing the superficial features of fundus images and the clinical manifestations of diabetic retinopathy lesions, we can propose an adaptive improvement strategy for the baseline model to further improve the diagnostic accuracy of the model. The specific improvement strategies mainly include the following three aspects:

[0045] 1. Improve the residual block.

[0046] Figure 2 Schematic diagram of the improved residual block in this embodiment.

[0047] Figure 6 Schematic diagram of the residual block in the prior art.

[0048] like Figure 2 and Figure 6 As shown in the figure, this embodiment draws on the improvements made to the residual block by ResNeXt and uses the Inception split-transform-aggregation algorithm to transform the residual block of ResNet. The convolution in the residual block is converted into grouped convolution. This saves parameters by reducing the input channels of the convolution kernel and widens the output channels of the intermediate layer. That is, with the same number of parameters, the improved residual block reduces the number of input channels and widens the number of output channels, making the network wider and improving classification accuracy.

[0049] The improved residual block first splits the input into C (cardinality) low-dimensional embeddings, then transforms each of the C low-dimensional embeddings separately, and finally aggregates the C low-dimensional embeddings. Experiments show that increasing the number of cardinality splits improves ResNet model accuracy more significantly than increasing network width and depth. Furthermore, this improved approach maintains virtually no change in hyperparameters, model complexity, or computational cost.

[0050] 2. Adopt dense connection (DC) structure.

[0051] Traditional diabetic retinopathy diagnostic algorithms typically carefully design and manually extract shallow features, such as color, texture, and spatial information, and then use traditional machine learning methods to classify lesions. Auxiliary diagnostic algorithms based on deep learning can automatically learn image features, but the final classification results depend only on the deep semantic features learned by the model, while ignoring shallow image features. This embodiment improves the model to address this problem.

[0052] Figure 3 Schematic diagram of a portion of the structure of the improved ResNet model in this embodiment.

[0053] like Figure 3As shown, in this embodiment, the outputs of all convolutional layers are interconnected, meaning that the input of each layer receives the outputs of all previous layers. By establishing connections between layers, the input of each layer includes shallow image features extracted by all previous layers, allowing the features extracted in each layer to be integrated with the features of all previous layers. The channel-wise connection of features from multiple layers enables feature reuse. As a result, superior performance compared to existing ResNets can be achieved while maintaining the same parameters and computational overhead.

[0054] 3. Adding a side output layer

[0055] Because the clinical manifestations of diabetic retinopathy lesions vary, the lesion grade auxiliary diagnosis model needs to have the ability to extract multi-scale features. For example, bleeding foci appear in large numbers on fundus images, with small areas of individual bleeding foci and a wide coverage area; microaneurysms are usually attached to blood vessels and have small areas; and the lesion areas of exudates are continuous and vary in size. However, the original version of ResNet cannot extract multi-scale features, and its feature extraction capabilities are somewhat lacking. This embodiment also improves the model to address this issue.

[0056] Figure 4 Schematic diagram of the structure of the improved ResNet model in this embodiment.

[0057] like Figure 4 As shown in Figure 1, this embodiment employs a side output layer structure, inserting three additional side output layers after the third, fourth, and fifth residual blocks, respectively. A fusion layer is also added to combine features from the multi-scale side output layers. Side output layers are not added after the first two residual units because the feature maps produced by these units are large. More importantly, the layers in these units primarily learn low-level visual features, which are insufficient for comprehensive classification.

[0058] The specific structure of the side output layer is as follows: All side output layers consist of an average pooling layer, a fully connected layer, and a softmax layer. The filter sizes of the three average pooling layers are 7×7, 5×5, and 7×7, respectively, and the strides of the three average pooling layers are 3, 3, and 1, respectively. Furthermore, all three side output layers are inserted after the BatchNorm layer and the ReLU layer. The average output sum of the three fully connected layers is input to the softmax layer of the fusion layer.

[0059] The improved model has higher classification accuracy than the baseline model and can respond to diagnostic requests in minutes while keeping the model complexity and computational consumption basically unchanged.

[0060] The improved ResNet model was trained using the experimental training set described above, with the following hyperparameter settings: learning rate (0.2), batch_size (64), momentum (0.9), weight decay (0.0001), and number of training iterations (1000). In this example, the learning rate was halved after every 50 iterations. The training strategy used was stochastic gradient descent with Nesterov acceleration to train the quality screening model on the training dataset, and the model performance was verified on the test dataset. The model was trained on a single NVIDIA 2080Ti.

[0061] Since the labels in the experimental training set are the lesion levels of diabetic retinopathy, the trained ResNet model is a lesion level diagnosis model, which is used to classify and predict the lesion level of fundus images.

[0062] In this embodiment, an ablation experiment was performed using 5,000 fundus images in the experimental test set. The experimental results are shown in Table 1.

[0063] Table 1 Ablation experiment data

[0064]

[0065] As can be seen from Table 1, compared with the existing ResNet or the ResNet using one of the above improvements, the ResNet model of this embodiment that integrates the above three improvements has better performance.

[0066] In addition, a comparative experiment was conducted between the improved model of this embodiment and other excellent models in the prior art. The comparative experiment results are shown in Table 2.

[0067] Table 2 Comparison experimental data of various classification diagnosis models

[0068]

[0069] As shown in Table 2, four existing diabetic retinopathy classification and diagnosis models were selected: Inception-v3, ResNet, InceptionResNet, and Xception. It can be found that ResNet-152, Inception-v3, InceptionResNet, and Xception all have lower classification accuracy than the model of this embodiment. At the same time, the model of this embodiment can quickly fit the training data after 3000 training iterations, with an accuracy improvement of 1.4% compared to the best comparison model. The diagnostic speed is significantly faster, with a runtime of only 1 / 4 to 1 / 5 of the comparison model.

[0070] This embodiment also provides a diabetic retinopathy auxiliary diagnosis device with limited computing resources.

[0071] Figure 5 This is a structural block diagram of the diabetic retinopathy auxiliary diagnosis device with limited computing resources in this embodiment.

[0072] like Figure 5 As shown, the diabetic retinopathy auxiliary diagnosis device 10 includes a fundus image acquisition module 11, a preprocessing module 12, a classification prediction module 13, a storage module 14, and a control module 15.

[0073] Among them, the fundus image acquisition module 11 adopts the method of the above-mentioned step S1 to acquire the fundus images of patients with diabetic retinopathy; the preprocessing module 12 adopts the method of the above-mentioned step S2 to preprocess the acquired fundus images; the classification prediction module 13 adopts the method of the above-mentioned step S3 to classify and predict the preprocessed fundus images; the storage module 14 stores the above-mentioned trained and improved ResNet model; and the control module 15 coordinates and controls the work of the above-mentioned modules.

[0074] Example Function and Effect

[0075] According to the computing resource-limited diabetic retinopathy auxiliary diagnosis method and device provided by this embodiment, a residual neural network model is used to classify and predict the patient's fundus images, and the residual neural network model is improved, the residual block is improved, and shallow features are extracted. The model has a strong relational inductive bias. Therefore, the improved model can learn more features, better express features, and better assist in the diagnosis of diabetic retinopathy, which can ultimately improve the accuracy of classification. In addition, the model structure is simple and does not require the use of model mixing, multi-task training, and metric learning methods. Therefore, compared with existing high-precision models, the model of this embodiment is fast and convenient to build, and the amount of computation consumed by the training process is also small, which is suitable for environments with limited memory and computing resources.

[0076] Specifically, the convolution in the residual block is converted into grouped convolution, which saves parameters by reducing the input channels of the convolution kernel and widens the output channels of the intermediate layer. Therefore, the improved residual block can improve the classification accuracy of the model without changing the number of parameters, and the complexity and computational consumption of the model remain basically unchanged.

[0077] Furthermore, due to the use of a dense connection structure, the features extracted from each layer in the improved ResNet model are fused with the features of all previous layers. The connection of features from multiple layers on the channel can achieve feature reuse. Therefore, the improved model can achieve better performance than the existing ResNet model without changing the parameters and computational consumption.

[0078] Furthermore, due to the adoption of the side output layer structure, the improved model has the ability to extract multi-scale features, thereby improving the classification accuracy.

[0079] In the embodiment, the model of this embodiment is compared with four excellent classification diagnosis models in the prior art. The comparison results show that the model of this embodiment not only has higher classification accuracy than the existing four classification diagnosis models, but also has a significantly faster diagnosis speed.

[0080] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A method for auxiliary diagnosis of diabetic retinopathy with limited computing resources, characterized in that: include: Step S1, collecting fundus images of the subject; Step S2, preprocessing the collected fundus image; Step S3: using the trained improved residual neural network model to perform classification prediction on the pre-processed fundus image. In step S2, the preprocessing includes: cropping the region of interest, resizing, color and brightness normalization, and data enhancement, and the data enhancement includes: Randomly flipping the fundus image horizontally or vertically; Randomly smoothing the fundus image using a Gaussian filter; Randomly adjusting the contrast of the fundus image; Randomly rotating the fundus image by an angle; and The fundus image of size 512×512 is cropped to a fundus image of size 448×448 with random offsets, The improved residual neural network model includes: Multiple densely connected convolutional layers, where the outputs of all the convolutional layers are connected to each other; a plurality of side output layers for extracting shallow features of the fundus image; and A fusion layer for combining the shallow features from multiple side output layers, The convolutional layer includes multiple improved residual blocks, BatchNorm layers, and ReLU layers. The Inception split-transform-aggregation method is used to transform the residual block of ResNet, and the convolution in the residual block of ResNet is converted into grouped convolution. By reducing the input channels of the convolution kernel, parameters are saved, and the output channels of the intermediate layer are widened, thereby obtaining the improved residual block. The improved residual block splits the input into multiple low-dimensional embeddings, and then transforms the multiple low-dimensional embeddings separately, and finally aggregates the multiple low-dimensional embeddings. The three side output layers are respectively inserted after the third, fourth and fifth improved residual blocks, and the three side output layers are all inserted after the BatchNorm layer and the ReLU layer, Each of the side output layers consists of an average pooling layer, a fully connected layer, and a softmax layer, wherein the filter sizes of the three average pooling layers are 7×7, 5×5, and 7×7, respectively, and the step sizes of the three average pooling layers are 3, 3, and 1, respectively. The fusion layer has at least a softmax layer, and the average output sum of the three fully connected layers is input to the softmax layer of the fusion layer. When training the improved residual neural network model, the hyperparameters are set as follows: learning rate is 0.2, batch_size is 64, momentum is 0.9, weight decay is 0.0001, the number of training iterations is 1000, and the learning rate is halved after every 50 iterations. The training strategy adopted is stochastic gradient descent with Nesterov acceleration to train the quality screening model on the training dataset, and the performance of the model is verified on the test dataset.

2. The computing resource-constrained diabetic retinopathy auxiliary diagnosis method according to claim 1, characterized in that: in, The labels in the training set are the lesion levels of diabetic retinopathy. The trained improved residual neural network model is used to classify and predict the lesion grade of the diabetic retinopathy in the fundus image.

3. A device for assisting in the diagnosis of diabetic retinopathy with limited computing resources, characterized in that: include: A fundus image acquisition module, used for acquiring fundus images of subjects; A preprocessing module, configured to preprocess the acquired fundus image; as well as The classification prediction module performs classification prediction on the pre-processed fundus image through the trained improved residual neural network model. The preprocessing includes: cropping the region of interest, resizing, color and brightness normalization, and data enhancement, and the data enhancement includes: Randomly flipping the fundus image horizontally or vertically; Randomly smoothing the fundus image using a Gaussian filter; Randomly adjusting the contrast of the fundus image; Randomly rotating the fundus image by an angle; and The fundus image of size 512×512 is cropped to a fundus image of size 448×448 with random offsets, The improved residual neural network model includes: Multiple densely connected convolutional layers, where the outputs of all the convolutional layers are connected to each other; a plurality of side output layers for extracting shallow features of the fundus image; and A fusion layer for combining the shallow features from multiple side output layers, The convolutional layer includes multiple improved residual blocks, BatchNorm layers, and ReLU layers. The Inception split-transform-aggregation method is used to transform the residual block of ResNet, and the convolution in the residual block of ResNet is converted into grouped convolution. By reducing the input channels of the convolution kernel, parameters are saved, and the output channels of the intermediate layer are widened, thereby obtaining the improved residual block. The improved residual block splits the input into multiple low-dimensional embeddings, and then transforms the multiple low-dimensional embeddings separately, and finally aggregates the multiple low-dimensional embeddings. The three side output layers are respectively inserted after the third, fourth and fifth improved residual blocks, and the three side output layers are all inserted after the BatchNorm layer and the ReLU layer, Each of the side output layers consists of an average pooling layer, a fully connected layer, and a softmax layer, wherein the filter sizes of the three average pooling layers are 7×7, 5×5, and 7×7, respectively, and the step sizes of the three average pooling layers are 3, 3, and 1, respectively. The fusion layer has at least a softmax layer, and the average output sum of the three fully connected layers is input to the softmax layer of the fusion layer. When training the improved residual neural network model, the hyperparameters are set as follows: learning rate is 0.2, batch_size is 64, momentum is 0.9, weight decay is 0.0001, the number of training iterations is 1000, and the learning rate is halved after every 50 iterations. The training strategy adopted is stochastic gradient descent with Nesterov acceleration to train the quality screening model on the training dataset, and the performance of the model is verified on the test dataset.

Images