Small sample-oriented diabetic retinopathy classification method and system

Abstract

The invention discloses a small sample-oriented diabetic retinopathy classification method and system, and belongs to the field of image processing and medicines.The classification method comprises a pre-training stage based on lightweight RexNet channel optimization, a fundus image data preprocessing stage, a data enhancement stage and a model fine-tuning training stage based on transfer learning. System implementation stages are formed; according to the method, the problems of few medical image data samples and patient privacy are solved, and the barrier of insufficient DR diagnosis capability of a grassroots community is broken through. Meanwhile, in combination with transfer learning, pre-training is carried out on ImageNet, and a pre-training model is obtained; data are enhanced, so that the accuracy of a diabetic retinopathy classification detection system is improved, the calculation cost is reduced, and a grassroots community is assisted in clinical diagnosis of diabetic retinopathy to perform prevention, screening and diagnosis of diabetic retinopathy.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a small sample oriented diabetic retinopathy classification method and system. Background Art

[0002] Diabetic retinopathy (DR) is a serious complication of diabetes and is more common in the elderly population. With age, the risk of DR in diabetic patients increases significantly. As the disease progresses, patients may experience symptoms such as blurred vision, reduced visual field, and decreased contrast sensitivity. In more severe cases, retinal neovascularization, vitreous hemorrhage, preretinal hemorrhage, and tractional retinal detachment may occur, which may eventually lead to blindness.

[0003] In the elderly population, regular eye examinations are particularly important for the screening and early diagnosis of DR. Regular eye examinations are the key to discovering DR. At present, DR screening in my country also faces severe challenges.

[0004] In the medical field, due to the need to protect patient privacy, medical data marked with personal privacy information is often not publicly available, which leads to a certain degree of isolation of medical data. In addition, the cost of medical data annotation is high and requires professional knowledge, which limits the large-scale expansion of medical data sets.

[0005] In terms of treatment, in addition to traditional laser therapy and drug therapy, emerging medical technologies, such as the application of deep learning algorithms in medical images, provide new possibilities for the detection and diagnosis of DR. Deep learning has been widely used in medical image processing and disease-assisted diagnosis, but it usually requires a large amount of medical data support. However, it is difficult and unrealistic to fully annotate a large amount of data, and there are also challenges in the collection of image samples. In view of these characteristics of medical data, the transfer learning method in small sample learning provides a new solution for medical image processing.

[0006] The samples were reviewed and annotated according to the International Clinical Diabetic Retinopathy Disease Severity Scale (ICDRSS). The samples were divided into five categories: no diabetic retinopathy (No DR), mild (Mild), moderate (Moderate), severe (Severe) and proliferative (Proliferative), corresponding to 0-4 respectively.

[0007] The emergence of the diabetic retinopathy detection system has a significant effect on promoting healthy aging and bridging the "gap" of primary medical resources, and can assist in the discovery and diagnosis of early DR. Therefore, it is necessary to consider a diabetic retinopathy classification method and system for small samples. The lightweight model RexNet with optimized channel configuration is combined with data enhancement. Through transfer learning, it mainly solves the problem of small medical image data samples and insufficient DR diagnosis capabilities in primary communities. Summary of the invention

[0008] The purpose of the present invention is to provide a small sample-oriented diabetic retinopathy classification method and system to solve the problem of small medical image data samples and patient privacy, and break the barrier of insufficient DR diagnosis capabilities in grassroots communities. At the same time, combined with transfer learning, pre-training is performed on ImageNet to obtain a pre-trained model. The data is enhanced to improve the accuracy of the diabetic retinopathy classification detection system, reduce the computing cost, and assist grassroots communities in clinical diagnosis of diabetic retinopathy and preventive screening and diagnosis of diabetic retinopathy.

[0009] The technical solution proposed in the present invention includes a pre-training stage based on lightweight RexNet channel optimization, fundus image data preprocessing, a data enhancement stage, a model fine-tuning training stage based on transfer learning, and a system implementation stage.

[0010] Pre-training phase based on lightweight RexNet channel optimization:

[0011] Inverted Residual Structure and Linear Bottleneck are introduced to optimize the network channel design based on MobileNetV2: ①Progressively increase the number of channels in each layer ②Use nonlinear activation function Swish-1 to increase the rank. Pre-training is performed on the large image dataset ImageNet to obtain a powerful pre-training model.

[0012] Fundus image data preprocessing stage:

[0013] Preprocess the APTOS dataset. The fundus image dataset comes from the APTOS2019 Blindness Detection dataset released by Aravind Hospital in India on the Kaggle public platform. Fundus photos are collected under different conditions and environments. Some images are very dark and difficult to visualize. In order to reduce the impact of lighting conditions. Grayscale processing and cropping are used to try to convert the image to grayscale to reduce color interference. Crop the areas with less information and perform automatic cropping.

[0014] Data augmentation stage:

[0015] The preprocessed APTOS dataset is subjected to data enhancement, including random image scaling, cropping and interpolation, random image horizontal flipping, automatic contrast adjustment, histogram equalization, image flipping, rotation, increasing image tonal separation, increasing image exposure, increasing exposure effect, increasing image color saturation, contrast, brightness, and sharpness; relative application of shear transformation and relative translation on the XY axis; normalization; and random image erasing.

[0016] The preprocessed APTOS dataset is augmented with data, including Mixup and Cutmix, to generate new samples by combining different images and labels to improve generalization. Mixup creates a new image by linearly interpolating between two images, and the labels are mixed accordingly. Cutmixup cuts and replaces specific areas of the image (usually rectangular areas) to increase the model's robustness to local features of the image, and controls the enhancement effect by setting different parameters.

[0017] Model fine-tuning training phase based on transfer learning:

[0018] The image data after data preprocessing and data enhancement was used as input, and the pre-trained model RexNet was called based on transfer learning to fine-tune the model. The activation function ReLu6 and AdamP optimizer in the reverse linear residual block, which are most suitable for retinal disease applications, were used to optimize the lightweight RexNet with channel configuration optimization, and the step size was adaptively adjusted; all models were trained using cross entropy loss, 1000 ecpoches were trained, and the batch size was 32. The original learning rate was set to 0.001 using the cosine annealing scheduling rate, and 20 linear preheatings were performed. In order to prevent overfitting, a weight decay rate of 0.05 was used to obtain the DR classification model.

[0019] Compared with the existing invention, the beneficial effects of the present invention are:

[0020] (1) The method proposed in the present invention is based on the lightweight framework RexNet, which is trained on ImageNet to form a powerful pre-trained model. RexNet helps improve the accuracy of diabetic retinopathy classification prediction while maintaining a low computational cost by optimizing channel configuration.

[0021] (2) The method proposed in the present invention uses a large amount of data enhancement to find the most discriminative local feature representation to extract DR lesions. By combining different images and labels to generate new samples, the generalization ability is improved, thereby assisting in diagnosis, prediction and treatment.

[0022] (3) The method proposed in the present invention is based on model fine-tuning of transfer learning, and inputs image data after preprocessing and data enhancement. Based on a powerful pre-trained model, it can solve the problems of small number of medical image data samples and insufficient DR diagnosis capabilities in grassroots communities, promote healthy aging and bridge the "gap" of grassroots medical resources, and assist in the discovery and diagnosis of early DR. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the process of the method proposed in the present invention.

[0024] Figure 2 It is a comparison diagram before and after data preprocessing in the method proposed in the present invention, wherein the left diagram is before preprocessing and the right diagram is after preprocessing.

[0025] Figure 3 It is the result of the processing of steps S31-S34 of the present invention, wherein, from left to right, they are the original image, the image randomly mirrored, scaled, cropped and interpolated, the image horizontally flipped by 50%, the image enhanced by the RandAugment algorithm, the image of the standardization operation and the randomly erased image.

[0026] Figure 4 It is the result of the processing of step S35 of the present invention, wherein, from left to right, there are randomly selected image 1, randomly selected image 2, and an image after randomly applying Mixup and Cutmix data enhancement with a probability of 50%.

[0027] Figure 5 This is a model structure diagram based on transfer learning in the method proposed in the present invention.

[0028] Figure 6 It is a graphical interface of the diabetic retinopathy classification system provided by the present invention. DETAILED DESCRIPTION

[0029] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0030] like Figure 1 As shown, the present invention provides a small sample-oriented diabetic retinopathy classification method, the detailed steps are as follows:

[0031] Step 1: Pre-training based on lightweight RexNet channel optimization

[0032] The present invention first pre-trains the RexNet model on the ImageNet dataset to obtain the initial network parameters for subsequent DR tasks. Model pre-training on large-scale datasets can enable the model to learn some common features and prior knowledge, reduce dependence on the target training task dataset and improve the training efficiency of the target task. The RexNet model is a popular network designed for the ImageNet classification task. In view of the expression bottleneck problem of the MobileNetV2 network, a new network design principle is proposed. RexNet uses multiple extension layers to gradually improve the rank of the output dimension and reduce the layer expression bottleneck problem caused by the leap improvement. In addition, based on the characteristic that the Swish function has a higher rank, the activation function is introduced in both the first convolutional layer and the extension layer.

[0033] Step 2: Preprocess fundus image data

[0034] In the case of redundant black edges in the image, a method based on brightness threshold is used to solve it. The specific method is to convert the color image into a grayscale image, which can simplify the image data and facilitate calculation. Based on the obtained grayscale image and brightness threshold, a mask matrix is ​​established, and the three color channels of R / G / B of the image are cropped by the mask matrix respectively, and then recombined to generate a new color image.

[0035] In view of the irregular fundus image, a circular cropping method is used to make it conform to the fundus shape characteristics. The specific method is to establish a circular mask matrix M(x, y) and perform a bitwise AND operation with the original image I(x, y). The calculation formula is:

[0036] I cropped (x,y)=I(x,y)·M(x,y)

[0037] Among them, I cropped (x, y) is the pixel value of the cropped image at position (x, y). M(x, y) is 1 when the distance from the pixel point (x, y) to the center of the circle is less than or equal to the radius r. The images before and after processing are as follows Figure 2 shown.

[0038] Step 3: Perform data enhancement on the preprocessed APTOS dataset

[0039] First, randomly mirror, scale, crop and interpolate the image. Random cropping refers to randomly sampling a rectangular area from the original image, and the size of the area is within a certain range (for example, between 0.08 and 1). The sampled area is then scaled to a fixed size (224x224) as the input of the network. Interpolation is a technique for converting a low-resolution image into a high-resolution image. The interpolation algorithm approximates the missing pixel values ​​in the original image to reconstruct a clearer image. The present invention uses bicubic interpolation, which can provide a smoother effect when the image is scaled and reduce image distortion.

[0040] Secondly, randomly flip the image horizontally (with a probability of 50%), which can increase the randomness of the data. Since it is performed randomly, it can increase the diversity and generalization ability of the data and reduce the risk of overfitting of the model.

[0041] Next, the RandAugment algorithm is used to enhance the image. The algorithm randomly selects N operations from a predefined set of enhancement operations and applies a fixed amplitude M to each operation. The configuration used by the present invention is rand-m9-mstd0.5-inc1, which randomly selects 2 operations from 15 operations (inversion / flipping / contrast enhancement, etc.), and the amplitude of each operation conforms to the normal distribution with a mean of 9 and a standard deviation of 5.

[0042] Next, the image is normalized and the pixel values ​​are adjusted to be centered at 0 and have a unit variance distribution, which helps the model learn and generalize better. The present invention also uses a random erasing method to randomly select a rectangular area in the image and replace its pixel values ​​with random values ​​or preset values ​​to simulate the situation where the fundus image is incomplete. The data enhancement effect based on a single image is as follows Figure 3 Shown

[0043] Finally, the present invention randomly applies Mixup and Cutmix methods to the images in the batch with a probability of 50%, and the processed images are as follows Figure 4 Mixup can create new samples with smooth transitions, which helps the model learn the boundaries between categories and reduce overfitting. Its operation can be expressed as:

[0044] x ′ =λx i +(1-λ)x i

[0045] y ′ =λy j +(1-λ)y j

[0046] Here, λ is a number randomly sampled from a Beta distribution, usually in the range [0,1].

[0047] Cutmix selects partial images for “pasting” and mixes labels accordingly. This method can force the model to learn different parts of the image and improve the model’s ability to recognize local features. Its operation can be expressed as:

[0048] x ′ =M⊙x A +(1-M)⊙x B

[0049] y ′ =λy A +(1-λ)y B

[0050] Where M is a binary mask indicating which regions should be retained from x A , which regions should come from x B . λ is also a number sampled from the Beta distribution and is used to determine the mixing ratio of the labels.

[0051] Step 4: Fine-tune the pre-trained model

[0052] All the data processed in step 3 are randomly divided into training set and test set in a ratio of 4:1. The weights and biases of the model are optimized on the training set, and the model performance is evaluated on the test set to verify its effectiveness and generalization ability. For neural network models, choosing the appropriate optimizer, loss function, and formulating a reasonable learning rate scheduling strategy are crucial for efficient learning and good performance of the model.

[0053] The present invention adopts the AdamP optimizer to adjust the learning rate of each parameter. It introduces weight perspective momentum and adaptive learning rate scheduling, so that it can intelligently adjust the learning speed at different training stages. Weight momentum helps the optimizer maintain direction and speed to avoid detours during the optimization process; while the adaptive learning rate ensures that each parameter is updated at a speed suitable for it. The weight decay relative ratio parameter of the AdamP optimizer set by the present invention is 0.01, and Nesterov momentum is used.

[0054] Due to Mixup and Cutmix processing, the label data is converted from hard labels to soft labels. The conventional Softmax cross loss function is no longer applicable. The present invention adopts the soft label cross loss function. Its formula is as follows:

[0055]

[0056] Where N is the number of samples, M is the number of categories, and t ij is the probability of the jth category in the true label (soft label) of the i-th sample, p ijis the probability that the model predicts that the i-th sample belongs to the j-th category.

[0057] For the selection of learning rate scheduling strategy, the present invention adopts the cosine annealing algorithm. Starting from a larger learning rate, the model can avoid oscillating and searching for the local optimal solution. As the training progresses, the learning rate gradually decreases in the form of a cosine function. This process is repeated periodically. In addition, in order to start the model stably, the present invention also adopts a preheating strategy, with an initial learning rate of 0.000001, which is gradually increased to 0.001, and a preheating cycle of 20.

[0058] In addition, in order to further reduce the overfitting of the model and enhance the representation ability of the single-layer network, a spatial Dropout operation is added to the model. This operation randomly discards all activation values ​​in the entire feature map. The network model structure used in the present invention is as follows: Figure 5 shown.

[0059] The present invention is trained on an Nvidia RTX 3090 GPU with 24G memory, and simulation experiments are performed to evaluate the effectiveness of the model. All models are trained for 1000 epochs, with a batch size of 32 and an image output size of 320*320. The accuracy evaluation indicators used are the area under the characteristic curve (AUC), accuracy (ACC), statistic (Kappa) and F1 value (F1-score). The experimental results are as follows.

[0060] It is not difficult to find from the table that the performance of the present invention using data augmentation and fine-tuning in terms of four evaluation indicators, AUC, ACC, Kappa, and F1score, is higher than that of the model using only the pre-trained model or the model without data augmentation.

[0061]

[0062] Step 5: Graphical detection interface.

[0063] The present invention also builds a set of simple graphical interfaces to provide users with friendly detection operations and store the user's detection data to facilitate tracking of the user's health status. The graphical system is built based on the Tkinter library.

[0064] This interface provides a registration and login function. Users use passwords to enter the system and conduct tests, ensuring the isolation of user data. It also provides a fundus image upload function. Users click the upload image button to open the local directory and complete the upload of fundus images. After the background program completes the analysis, the image and the inferred lesion type will be displayed on the front end. This interface records the user's usage (lesion type) through the MySQL database to facilitate subsequent analysis and tracking of the disease. The specific interface is as follows Figure 6 shown.

Claims

1. A classification method for diabetic retinopathy for small samples, characterized in that: The classification method includes: S1. In the pre-training phase based on lightweight RexNet channel optimization, the inverted residual structure and linear bottleneck are introduced. Based on MobileNetV2, the network channel design is optimized to obtain a pre-training model based on lightweight RexNet channel optimization. S2, fundus image data preprocessing stage, the data comes from the public APOTS dataset, and the preprocessed dataset is obtained after preprocessing the image data of light source inhomogeneity; S3, data enhancement stage, through a large amount of data enhancement, find the most discriminative local feature representation to extract diabetic retinopathy and improve the performance of retinal blood vessel segmentation; generate new samples by combining different images and labels to improve generalization ability; S4: In the model fine-tuning training phase based on transfer learning, the image data after data preprocessing and data enhancement is used as input, and the pre-trained model RexNet is called based on transfer learning to perform model fine-tuning to obtain a classification model for diabetic retinopathy for small samples.

2. The classification method for diabetic retinopathy for small samples according to claim 1, characterized in that: In step S1, the inverted residual structure and linear bottleneck are introduced, and the network channel design is optimized based on MobileNetV2; pre-training is performed on the large image dataset ImageNet to obtain a pre-trained model; wherein, the optimized network channel design includes progressively increasing the number of channels in each layer and using the nonlinear activation function Swish-1 to increase the rank.

3. The classification method for diabetic retinopathy for small samples according to claim 2, characterized in that: In step S2, the fundus image data comes from the APTOS2019 Blindness Detection dataset released by the Aravind Hospital in India on the Kaggle public platform; the fundus photos with redundant black edges and irregularities are preprocessed respectively.

4. The classification method for diabetic retinopathy for small samples according to claim 3, characterized in that: In order to solve the problem of redundant black edges in the image, a method based on brightness threshold is adopted. The specific method is to convert the color image into a grayscale image to simplify the image data. A mask matrix is ​​established based on the obtained grayscale image and brightness threshold, and the mask matrix is ​​applied to the three color channels of the image R / G / B for cropping, and then recombined to generate a new color image. In view of the irregular fundus image, a circular cropping method is used to make it conform to the fundus shape characteristics; the specific method is to establish a circular mask matrix M(x, y) and perform a bitwise AND operation with the original image I(x, y). The calculation formula is: I cropped (x,y)=I(x,y)·M(x,y) Among them, I cropped (x, y) is the pixel value of the cropped image at position (x, y). M(x, y) is 1 when the distance from the pixel point (x, y) to the center of the circle is less than or equal to the radius r.

5. The classification method for diabetic retinopathy for small samples according to claim 4, characterized in that: Step S3 includes randomly scaling, cropping and interpolating the image, randomly flipping the image horizontally, automatically adjusting contrast, histogram equalization, flipping the image, rotating, increasing the tonal separation of the image, increasing the exposure of the image, increasing the exposure effect, increasing the color saturation, contrast, brightness, and sharpness of the image; relatively applying shear transformation and relative translation on the XY axis; Normalization processing; random erasing operation is performed on the image.

6. The classification method for diabetic retinopathy for small samples according to claim 5, characterized in that: Step S3 also includes Mixup and Cutmix, which generate new samples by combining different images and labels to improve generalization ability; Mixup creates a new image by linear interpolation between two images, and the labels are mixed accordingly; Cutmixup cuts and replaces specific areas of the image to increase the robustness of the model to local features of the image, and controls the enhancement effect by setting different parameters.

7. The classification method for diabetic retinopathy of small sample size according to claim 6, characterized in that: Step S3 specifically includes the following steps: S31, randomly mirror, scale, crop and interpolate the image; random cropping is to randomly sample a rectangular area from the original image; then scale the sampled area to a fixed size of 224x224 as the input of the network; and use bicubic interpolation to provide a smoother effect when scaling the image and reduce image distortion; S32, randomly flip the image horizontally with a probability of 50% to increase the diversity and generalization ability of the data and reduce the risk of overfitting of the model; S33, uses the RandAugment algorithm to enhance the image, randomly selects N operations from a predefined set of enhancement operations, and applies a fixed amplitude M to each operation; the configuration used is rand-m9-mstd0.5-inc1, randomly selects 2 operations from 15, and the amplitude of each operation conforms to the normal distribution with mean 9 and standard deviation 5; S34, performing a standardization operation on the image, wherein the pixel values ​​are adjusted to be distributed with a unit variance and centered at 0; and using a random erasing method, randomly selecting a rectangular area in the image, and replacing its pixel values ​​with random values ​​or preset values, to simulate a situation where the fundus image is incomplete; S35, randomly applies Mixup and Cutmix methods to images in the batch with a probability of 50%; Mixup creates new samples with smooth transitions, which helps the model learn the boundaries between categories and reduce overfitting; its operation is expressed as: x ′ =λx i +(1-λ)x i and ′ =ly j +(1-λ)y j Where λ is a number randomly sampled from a Beta distribution, in the range [0,1]; Cutmix selects local images for "pasting" and mixes labels accordingly; it forces the model to learn different parts of the image and improves the model's ability to recognize local features. Its operation is expressed as: x ′ =M⊙x A +(1-M)⊙x B and ′ =ly A +(1-λ)y B Where M is a binary mask indicating which regions should be retained from x A , which regions should come from x B ; λ is the number sampled from the Beta distribution and is used to determine the mixing ratio of the labels.

8. The classification method for diabetic retinopathy for small samples according to claim 1, characterized in that: In step S4, the pre-trained model based on lightweight RexNet channel optimization in step S1 and steps 2 and 3 are used to complete the preprocessing and data enhancement of the APOTS dataset. The image data after data preprocessing and data enhancement is used as input, and the pre-trained model RexNet is called based on transfer learning to perform model fine-tuning to obtain a classification model for diabetic retinopathy for small samples.

9. A classification system for diabetic retinopathy for small samples, characterized in that: The system comprises: A pre-training module based on lightweight RexNet channel optimization, which is used to introduce the inverted residual structure and linear bottleneck. Based on MobileNetV2, the network channel design is optimized to obtain a pre-training model based on lightweight RexNet channel optimization. The fundus image data preprocessing module is based on the public APOTS data set, and preprocesses the light source uneven image data to obtain the preprocessed data set; The data augmentation module is used to find the most discriminative local feature representation to extract diabetic retinopathy through a large amount of data augmentation, and to generate new samples by combining different images and labels; The model fine-tuning training module based on transfer learning is used to take the image data after data preprocessing and data enhancement as input, call the pre-trained model RexNet based on transfer learning, perform model fine-tuning, and obtain a classification model for diabetic retinopathy DR for small samples.

Images