Fundus photograph classification system and storage medium for chronic kidney disease detection

Abstract

This invention provides a fundus image classification system and storage medium for chronic kidney disease detection, addressing the technical problems of low efficiency and large errors in existing fundus image classification methods. The system includes: acquiring and preprocessing fundus images to generate standardized fundus images; dividing the processed fundus images into training and testing sets, and augmenting the training set; preprocessing the training and testing sets to ensure uniform class distribution in each training batch input to the model during training; integrating learning strategies and training branched neural network models; and fusing the models to obtain the final detection model, achieving fundus image classification. This invention offers high efficiency and excellent automation; it extracts features such as the optic cup / disc ratio and arteriovenous ratio from fundus images and combines them with machine learning methods for classification. This invention also boasts high computational speed and low computer resource consumption during operation.

Description

[0001] This patent application is a divisional application of Chinese Patent 202011450625.X entitled “Method for classifying fundus photographs, method and system for processing fundus images”, the full text of which is incorporated herein by reference. Technical Field

[0002] This invention relates to the field of medical image classification and detection technology, specifically to a method for classifying fundus photographs, a method for processing fundus images, and a system. Background Technology

[0003] Chronic kidney disease (CKD) is a chronic disease affecting global public health. The prevalence and incidence of CKD are increasing annually, characterized by high morbidity, difficulty in treatment, high mortality, and low awareness. CKD is often accompanied by cardiovascular diseases such as hypertension and diabetes. As CKD gradually worsens, patients may require kidney transplantation or long-term dialysis to sustain their lives.

[0004] Studies have shown that early diagnosis and treatment of chronic kidney disease can effectively prevent further deterioration of the condition. One means of prevention and treatment is through fundus photography and regular checkups to monitor disease progression and allow for timely intervention. However, relying solely on doctors to visually examine fundus images presents several challenges. Firstly, experience alone makes accurate judgment and classification difficult; secondly, it requires a significant workload for interpreting the images. How to classify and select potential target fundus images for doctors' reference, thereby improving classification accuracy, has been a pressing issue that needs to be addressed.

[0005] In recent years, with the development of computer vision technology, many deep learning models have been widely used in the field of medical imaging. Therefore, in order to reduce the time and effort of manual screening while ensuring recognition accuracy, it is essential to design an image preprocessing and deep learning classification method for fundus photographs that can be used for chronic kidney disease detection.

[0006] Currently, classification methods based on fundus images are mainly divided into three categories: the first category involves doctors manually marking disease-related areas through visual observation; the second category uses image segmentation, feature extraction, and other methods to extract features such as the optic cup / optic disc ratio and arteriovenous ratio from fundus images, and combines these with machine learning methods for classification; and the third category uses end-to-end deep learning models to make predictions directly.

[0007] Methods involving doctors' visual observation and manual labeling rely heavily on their subjective experience, are extremely labor-intensive, and inefficient. Using image segmentation and feature extraction techniques to extract features such as the optic cup / optic disc ratio and arteriovenous ratio from fundus photographs, combined with machine learning methods for classification, generally involves complex algorithm design, slow computation speed, and is highly dependent on the correlation between the disease and the extracted features. If the correlation between the extracted features and the disease is poor, or if the feature extraction algorithm is unstable, the classification results may be unsatisfactory. Using end-to-end deep learning models for classification requires ensuring the quality control of the input fundus photographs; if the quality of the input fundus photographs is poor, the deep learning model will not achieve normal classification results. Summary of the Invention

[0008] The present invention proposes a method and system for classifying fundus images, which can solve the technical problems of low efficiency and low accuracy of existing classification methods based on fundus images in the detection of chronic kidney disease.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] According to one aspect of the present invention, a method for classifying fundus photographs that can be used for the detection of chronic kidney disease is provided.

[0011] Includes the following steps:

[0012] S100: Acquire fundus photos and perform preprocessing to generate standardized fundus images;

[0013] S200. Divide the fundus images into a training set and a test set, and augment the training set with data.

[0014] S300: Perform data preprocessing on the training and test sets of S200 to ensure that each batch of classes input into the model is uniform during the training phase.

[0015] S400, based on the S300 ensemble learning strategy, and trained branch neural network models;

[0016] S500 performs model fusion to obtain the final detection model, enabling the classification of fundus images.

[0017] Furthermore, in step S100, fundus photographs are acquired and preprocessed to generate standardized fundus images;

[0018] include:

[0019] 1.8) For each fundus image, first extract the region of interest (ROI), extract the R channel component of the fundus image, and perform binarization.

[0020] 1.9) The binarized R channel components are closed using a 5x5 convolution kernel. Seed points are selected (int(h / 2), int(w / 2)), where h is the height of the image, w is the width of the image, and int indicates rounding of the floating-point number. The connected component analysis algorithm is used to extract the circular / circular cross-sectional region located in the center of the fundus image as mask 1.

[0021] 1.10) Find the minimum bounding rectangle of the binarized mask 1 image, and expand it outward by 10 pixels to form mask 2;

[0022] 1.11) Perform pixel-wise operations on the original fundus image based on the mask 1 image. That is, for corresponding points in the original image and the binarized mask 1 image, if the value of the corresponding point on the mask 1 image is 1, then retain the values ​​of the three RGB channels of the original image; if the value of the corresponding point on the mask 1 image is 0, then set the values ​​of the three RGB channels of the original image to 0.

[0023] 1.12) Based on the binarized mask 2 image, the original fundus photo is cropped to obtain the intermediate image, thereby achieving the purpose of extracting the ROI region;

[0024] 1.13) Extract the G channel component of the intermediate image. For pixels with non-zero values ​​in the corresponding mask 1 image, use histogram equalization to enhance image contrast. Where N is the total number of pixels, v is the original gray level, s is the gray level after histogram equalization (ranging from 0 to 255), cdf is the cumulative distribution function of v, and round indicates rounding to the nearest integer; cdf min The minimum value of CDF;

[0025] 1.14) Merge the intermediate image with the R channel component, the G channel component after histogram equalization, and the B channel component to obtain the preprocessed fundus image.

[0026] Furthermore, in step S200, the processed fundus images are divided into a training set and a test set, and the training set is augmented with data.

[0027] The processed fundus images were divided into training and testing sets, specifically including:

[0028] The original fundus image dataset was randomly divided into training and test sets using stratified sampling at a ratio of 4:1.

[0029] In the training set, stratified sampling is used again to randomly divide the data into five disjoint subsets: training set 1, training set 2, training set 3, training set 4, and training set 5.

[0030] Furthermore, in step S200, the processed fundus images are divided into a training set and a test set, and the training set is augmented with data.

[0031] This includes data augmentation of the training set, specifically including:

[0032] For each subset of fundus images in the training set, data augmentation is performed by randomly using any of the following methods:

[0033] 3.1) Flip horizontally;

[0034] 3.2) After cropping the image at a random ratio between 0 and 0.1, resize it to the original image size;

[0035] 3.3) Smoothing is performed using a 5x5 Gaussian filter with random σ.

[0036] Furthermore, in S300, the training and test sets of S200 are preprocessed to ensure that each batch of categories input into the model during the training phase is uniform.

[0037] Specifically, it includes:

[0038] 4.1) The dimensions of the fundus image after image preprocessing are: (h, w, c), where h is the height of the image, w is the width of the image, and c is the number of channels; the pixel value range of each channel is 0 to 255, and the pixel value range is normalized to map it to the interval of 0 to 1.

[0039] 4.2) Set the hyperparameters for model training;

[0040] The batch size and size l are set as follows: batch size = 16 and l = 600 for each training set.

[0041] For the training set, input data is prepared in a 12:4 ratio of negative samples (no chronic kidney disease) and positive samples (with chronic kidney disease), where negative samples are sampled without replacement and positive samples are sampled with replacement; the batch size of the test set is 1, and both negative and positive samples are sampled without replacement.

[0042] The input fundus image is resized to have dimensions (l,l,c), resulting in m tensor data in the format (l,l,c,n), where n is the batch size and m is the total number of batches.

[0043] Furthermore, the S400 is based on the S300 ensemble learning strategy and trains a branch neural network model;

[0044] The ensemble learning strategy specifically includes:

[0045] Five branch models are trained on each of five disjoint training subsets.

[0046] Each model uses four training subsets as the training set and leaves one training subset as the test set to adjust the parameters and branch the model.

[0047] The training and parameter tuning processes for each branch model are independent of each other.

[0048] Furthermore, the S400 is based on the S300 ensemble learning strategy and trains a branch neural network model;

[0049] This includes training branch neural network models; specifically:

[0050] 6.1) For each branch model, DenseNet-101 is used as the basic network structure. The last fully connected layer of DenseNet-101 is removed and a fully connected layer with an output dimension of 2 is added. The model parameters are transferred learning or retrained using parameters pre-trained based on ImageNet.

[0051] 6.2) The loss function used in deep learning models is the focal loss function, the specific formula of which is:

[0052]

[0053] Where α and γ are manually set hyperparameters, specifically α = 0.6 and γ = 0.25, p is the probability value of the softmax output image being a positive sample, and y is the true label of the image, y = 1 indicates having chronic kidney disease, and y = 0 indicates not having chronic kidney disease. In order to prevent overfitting, L2 regularization is used to constrain the model.

[0054] 6.3) The model optimization strategy uses the Adam optimizer; the Adam optimization steps are as follows:

[0055]

[0056] m t =β1m t-1 +(1-β1)g t

[0057]

[0058]

[0059] Among them, g tLet be the gradient of the loss function L with respect to the parameter θ at time t, β1 and β2 are hyperparameters, β1 = 0.9 and β2 = 0.999, and m0 = 0, v0 = 0, α is the learning rate, and e is the denominator protection parameter; α = 0.0001, e = 1 * 10 -8 θ t The updated parameters;

[0060] 6.4) Dropout strategy: During training, 40% of the random parameters in the later layers of the model do not participate in the backpropagation of gradients.

[0061] 6.5) Early-Stop strategy: During the training of the branch model, if the absolute value of the loss on the training set and the loss on the test set becomes significantly larger, training is stopped and the model before the excessive bias occurs is retained.

[0062] Furthermore, the S500 performs model fusion to obtain the final detection model, thereby enabling the classification of fundus images;

[0063] Specifically, it includes:

[0064] For the trained branch model, the final classification result of the model is obtained by using soft-voting during the testing process.

[0065] On the other hand, the present invention also discloses a fundus photograph classification system, comprising the following units:

[0066] The fundus image preprocessing unit is used to acquire fundus images and perform preprocessing to generate standardized fundus images;

[0067] The fundus image segmentation unit is used to divide the processed fundus images into training and testing sets, and to augment the training set.

[0068] The training and test set processing unit performs data preprocessing on the training and test sets to ensure that each batch of categories input into the model during the training phase is evenly distributed.

[0069] The model training unit is used to train branch neural network models based on ensemble learning strategies.

[0070] The model fusion determination unit is used to perform model fusion to obtain the final detection model, thereby enabling the classification of fundus images.

[0071] According to another aspect of the present invention, a method for processing fundus images is also provided, characterized by comprising the following steps:

[0072] The first dataset is constructed, including preprocessing of the original fundus images and data filtering of the preprocessed fundus images. The preprocessing of the original fundus images includes data desensitization, ROI extraction and contrast enhancement. The data filtering of the preprocessed fundus images includes using the MobileNet v3 large model and soft voting method to filter the preprocessed fundus images and remove fundus images that do not meet the quality standards, thus constructing the first dataset.

[0073] A convolutional neural network model with a CCAM module is constructed. The convolutional neural network model includes a backbone network and the CCAM module. The backbone network includes an input, a first convolutional layer, an MLP, and a first sigmoid function. The CCAM module includes an upper branch and a lower branch. The upper branch is used to extract CAM based on a first feature map using a second convolutional layer, and uses a second sigmoid function to differentiate the importance of features. The lower branch uses activation and squeezing operations to extract the channel weights of the first feature map, and combines them with CAM to perform a scaling operation to obtain a channel CAM map with a channel attention mechanism. The channel CAM map is then subjected to a Hadamard product operation with the first feature map and added to the first feature map before being input into a fully connected layer or a GAP. The first feature map is the output of the first convolutional layer.

[0074] Using Focal Loss as the loss function, the convolutional neural network model with CCAM module is trained using the first dataset;

[0075] The trained convolutional neural network model with CCAM module is used to process the user's fundus images and output classification results.

[0076] According to another aspect of the invention, MLP is a fully connected layer or GAP (Global Average Pooling).

[0077] According to another aspect of the present invention, images of the user's left and right fundus are acquired and the data is desensitized, ROI is extracted, and contrast is enhanced.

[0078] According to another aspect of the present invention, the trained convolutional neural network model with CCAM module is used to process the left and right fundus images in parallel, and outputs the second feature map of the left eye and the second feature map of the right eye respectively. Then, they are aggregated together with the channel as the axis and used as the feature tensor of the fully connected layer or GAP. Finally, the classification result is output.

[0079] According to another aspect of the present invention, the decision-making basis region of the convolutional neural network model is visualized and a heatmap is generated using the SmoothGrad-CAM++ method.

[0080] As can be seen from the above technical solution, the fundus image classification method of the present invention is implemented based on fundus images. The present invention includes a method for fundus image preprocessing and a deep learning classification model; specifically, it includes a complete image preprocessing step to generate standardized fundus images through fundus image preprocessing; proportional allocation of the number of positive and negative samples in each batch during input data preprocessing; and ensemble learning combined with focal loss to solve the class imbalance problem in fundus image classification.

[0081] Compared to methods relying on doctors' visual observation and manual calibration, this invention offers superior efficiency and automation. Compared to methods that use image segmentation and feature extraction to extract features such as the optic cup / disc ratio and arteriovenous ratio from fundus photographs and then combine them with machine learning for classification, this invention boasts faster computation speed and consumes fewer computer resources during operation. Compared to using end-to-end deep learning models for classification, this invention ensures quality control of the input fundus photographs and, to some extent, addresses the problem of overfitting. Attached Figure Description

[0082] Figure 1 This is a schematic diagram of the method of the present invention;

[0083] Figure 2 This is a schematic diagram illustrating the generation of standardized fundus images through fundus image preprocessing in the method of the present invention;

[0084] Figure 3 This is a schematic diagram of the network model training of the present invention;

[0085] Figure 4 This is a schematic diagram of the model fusion of the present invention;

[0086] Figure 5 This is a schematic diagram of the convolutional neural network model with CCAM module of the present invention;

[0087] Figure 6 This is a schematic diagram of the left and right eye feature fusion network of the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0089] Example 1

[0090] The fundus image classification method described in this invention mainly utilizes image morphology and deep learning methods. It uses morphological methods to preprocess fundus images, and then, based on the DenseNet-101 network model, it introduces a focal loss function more suitable for morbidity problems and a training strategy for class imbalance problems, thus designing an automatic classification method based on the DenseNet-101 network model.

[0091] To achieve the above-mentioned objectives, this invention proposes an image preprocessing and automatic classification method for fundus photographs based on deep learning algorithms, comprising the following steps:

[0092] (1) Standardized fundus images are generated through fundus image preprocessing, such as... Figure 2 As shown

[0093] 1.15) For each fundus image, first extract the region of interest (ROI), extract the R channel component of the fundus image, and perform binarization.

[0094] 1.16) The binarized R channel components are closed using a 5x5 convolution kernel. Seed points are selected (int(h / 2), int(w / 2)), where h is the height of the image, w is the width of the image, and int indicates rounding of the floating-point number. The connected component analysis algorithm is used to extract the circular / circular cross-sectional region located in the center of the fundus image as mask 1.

[0095] 1.17) Find the minimum bounding rectangle of the binarized mask 1 image, and expand it outward by 10 pixels to form mask 2;

[0096] 1.18) Perform pixel-wise operations on the original fundus image based on the mask 1 image. That is, for corresponding points in the original image and the binarized mask 1 image, if the value of the corresponding point on the mask 1 image is 1, then retain the values ​​of the three RGB channels of the original image; if the value of the corresponding point on the mask 1 image is 0, then set the values ​​of the three RGB channels of the original image to 0.

[0097] 1.19) Based on the binarized mask 2 image, the original fundus photo is cropped to obtain the intermediate image, thereby achieving the purpose of extracting the ROI region;

[0098] 1.20) Extract the G channel component of the intermediate image. For pixels with non-zero values ​​in the corresponding mask 1 image, use histogram equalization to enhance image contrast. Where N is the total number of pixels, v is the original gray level, s is the gray level after histogram equalization (ranging from 0 to 255), cdf is the cumulative distribution function of v, and round indicates rounding to the nearest integer; cdf min The minimum value of CDF;

[0099] 1.21) Merge the intermediate image with the R channel component, the G channel component after histogram equalization, and the B channel component to obtain the preprocessed fundus image.

[0100] (2) Divide the training set and the test set.

[0101] The original fundus image dataset was randomly divided into training and test sets using stratified sampling at a ratio of 4:1. Within the training set, stratified sampling was again used to randomly divide the data into five disjoint subsets: Training Set 1, Training Set 2, Training Set 3, Training Set 4, and Training Set 5.

[0102] (3) Expand the training set through data augmentation.

[0103] For each subset of fundus images in the training set, data augmentation is performed by randomly using one or more of the following methods:

[0104] 3.1) Flip horizontally;

[0105] 3.2) After cropping the image at a random ratio between 0 and 0.1, resize it to the original image size;

[0106] 3.3) Smoothing is performed using a 5x5 Gaussian filter with random σ.

[0107] (4) Input data preprocessing

[0108] 4.1) The dimensions of the fundus image after image preprocessing are (h, w, c), where h is the height of the image, w is the width of the image, and c is the number of channels. The pixel value range of each channel is 0 to 255. Here, the pixel value range is normalized and mapped to the interval of 0 to 1.

[0109] 4.2) Set the hyperparameters for model training—batch size and size l. Here, the batch size in each training set is set to 16, and l to 600. For the training set, input data is prepared in a 12:4 ratio of negative samples (without chronic kidney disease) to positive samples (with chronic kidney disease), where negative samples are sampled without replacement and positive samples are sampled with replacement. The batch size of the test set is 1, and both negative and positive samples are sampled without replacement. Maintaining a constant ratio of positive to negative samples in the training set ensures that the gradient will not be trapped in a local minimum due to class imbalance during each update. The input fundus images are resized to (l, l, c), resulting in m tensors of (l, l, c, n) dimensions, where n is the batch size and m is the total number of batches.

[0110] (5) Integrated learning strategy

[0111] The main strategy for training deep learning models is ensemble learning, which involves training five branch models on each of five disjoint training subsets. Each model uses four training subsets as its training set and the remaining subset as its test set to fine-tune its parameters. For example, branch model 1 might be trained using training sets 2, 3, 4, and 5, and its parameters fine-tuned using training set 1 as its validation set. The training and parameter tuning processes for each branch model are independent of each other.

[0112] (6) Network model training, such as Figure 3 As shown

[0113] 6.1) For each branch model, DenseNet-101 is used as the basic network structure. The last fully connected layer of DenseNet-101 is removed and a fully connected layer with an output dimension of 2 is added. The model parameters can be transferred learning based on the parameters pre-trained on ImageNet, or they can be retrained.

[0114] The network model structure can also adopt VGG, ResNet, etc.;

[0115] 6.2) The loss function used in deep learning models is the focal loss function, the specific formula of which is:

[0116]

[0117] Where α and γ are manually set hyperparameters, α = 0.6 and γ = 0.25, p is the probability value of the softmax output image being a positive sample, and y is the true label of the image, y = 1 indicates having chronic kidney disease and y = 0 indicates not having chronic kidney disease. In order to prevent overfitting, L2 regularization is used to constrain the model.

[0118] 6.3) The model is optimized using the Adam optimizer. The Adam optimization steps are as follows:

[0119]

[0120] m t =β1m t-1 +(1-β1)g t

[0121]

[0122]

[0123] Among them, g tLet be the gradient of the loss function L with respect to the parameter θ at time t. β1 and β2 are hyperparameters, β1 = 0.9 and β2 = 0.999, and m0 = 0, v0 = 0. α is the learning rate, and e is the denominator protection parameter. Here, α = 0.0001, e = 1 * 10^25. -8 θ t The updated parameters;

[0124] 6.4) Dropout strategy: During training, 40% of the random parameters in the later layers of the model do not participate in the backpropagation of gradients.

[0125] 6.5) Early-Stop strategy: During the training of the branch model, if the absolute value of the deviation between the loss on the training set and the loss on the test set becomes significantly larger, training is stopped and the model before the deviation became too large is retained.

[0126] (7) Model fusion on the test set, such as Figure 4 As shown

[0127] For the five trained branches, during the testing process, soft-voting is used to obtain the final classification result of the model, and this final classification result is provided to, for example, a doctor for reference.

[0128] In this step, Hard-voting can be used instead of Soft-voting.

[0129] It is understandable that the training model determined by the method of this embodiment of the invention can directly input fundus photos and then automatically output classification results, which is fast and accurate.

[0130] In summary, the fundus image classification method of this invention is highly effective and has a high degree of automation compared to methods that rely on doctors' visual observation and manual calibration. Compared to methods that use image segmentation and feature extraction to extract features such as the optic cup / optic disc ratio and arteriovenous ratio from fundus images and then combine them with machine learning for classification, this invention has a faster computation speed and consumes less computer resources during operation. Compared to using end-to-end deep learning models for classification, this invention ensures the quality control of the input fundus images and, to some extent, solves the problem of overfitting.

[0131] On the other hand, the present invention also discloses a fundus photograph classification system, comprising the following units:

[0132] The fundus image preprocessing unit is used to acquire fundus images and perform preprocessing to generate standardized fundus images;

[0133] The fundus image segmentation unit is used to divide the processed fundus images into training and testing sets, and to augment the training set.

[0134] The training and test set processing unit performs data preprocessing on the training and test sets to ensure that each batch of categories input into the model during the training phase is evenly distributed.

[0135] The model training unit is used to train branch neural network models based on ensemble learning strategies.

[0136] The model fusion determination unit is used to perform model fusion to obtain the final detection model, thereby enabling the classification of fundus images.

[0137] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0138] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0139] Memory, used to store computer programs;

[0140] The processor, when executing the program stored in memory, implements the above-mentioned fundus image classification method;

[0141] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described fundus photograph classification methods.

[0142] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the fundus photograph classification methods described above.

[0143] Example 2

[0144] In existing technologies, convolutional neural network models employing attention branch modules are used to process fundus image data, but these modules still have several shortcomings. First, the attention branch module, referencing the CAM approach, replaces the fully connected layer connected to the softmax layer with a pooling algorithm (GAP). Compared to fully connected layers with trainable parameters, GAP is merely a pooling operation and cannot be parameterized, increasing the training pressure on the preceding convolutional layer and slowing down global convergence. Second, the attention branch module receives feature maps from the previous convolutional layer and compresses the feature map information into a CAM using ordinary convolutions and 1x1x1 convolutions. On another branch, the attention branch module uses 1x1 convolutions and GAP to constrain the loss function. In fact, these two components themselves can directly derive the CAM; the CAM derived from GAP can be considered the one that actually constrains the loss function, while the CAM derived from 1x1x1 convolutions acts on the original feature map to enhance the effective feature representation. This ambiguity in the CAM caused by the structural design leads to potential differences in the focus of the two branches of the attention branch module. Then, the loss function constraints added to the backbone network by the attention branch module only apply to the training phase. In other words, during the testing phase, a large portion of the parameters in the attention branch module are passively frozen, occupying storage space without participating in computation, thus increasing unnecessary hardware overhead.

[0145] To improve the accuracy of classifying fundus photographs or images and solve one or more of the aforementioned technical problems, this invention further proposes a novel convolutional neural network model to address the issues of slow model training speed and CAM ambiguity in existing technologies. Simultaneously, by integrating modules into the network, constraints on the loss function are removed, reducing unnecessary hardware overhead.

[0146] Specifically, see Figure 5 According to a preferred embodiment of the present invention, a method for processing fundus images is provided, characterized by comprising the following steps:

[0147] The first dataset is constructed, including preprocessing of the original fundus images and data filtering of the preprocessed fundus images. The preprocessing of the original fundus images includes data desensitization, ROI extraction and contrast enhancement. The data filtering of the preprocessed fundus images includes using the MobileNet v3 large model and soft voting method to filter the preprocessed fundus images and remove fundus images that do not meet the quality standards, thus constructing the first dataset.

[0148] A convolutional neural network model with a CCAM module is constructed. The convolutional neural network model includes a backbone network and the CCAM module. The backbone network includes an input, a first convolutional layer, an MLP, and a first sigmoid function. The CCAM module includes an upper branch and a lower branch. The upper branch is used to extract CAM based on a first feature map using a second convolutional layer, and uses a second sigmoid function to differentiate the importance of features. The lower branch uses activation and squeezing operations to extract the channel weights of the first feature map, and combines them with CAM to perform a scaling operation to obtain a channel CAM map with a channel attention mechanism. The channel CAM map is then subjected to a Hadamard product operation with the first feature map and added to the first feature map before being input into a fully connected layer or a GAP. The first feature map is the output of the first convolutional layer.

[0149] Using Focal Loss as the loss function, the convolutional neural network model with CCAM module is trained using the first dataset;

[0150] The trained convolutional neural network model with CCAM module is used to process the user's fundus images and output classification results.

[0151] Preferably, the MLP is a fully connected layer or a GAP (Global Average Pooling). CCAM specifically consists of CAM and a channel attention module (Class activation mapping and Channel Attention Module).

[0152] Advantageously, the MobileNet v3 large model achieved good data filtering results, with an average accuracy of 0.9800, an average Kappa coefficient of 0.9686, and an average Macro F1 of 0.9769 under 5-fold cross-validation. At the same time, it has only 299.88M FLOPs, a parameter size of 4,199,259, a size of 16.11Mb, and an average task execution time of 0.1221 seconds.

[0153] Advantageously, Focal Loss can significantly improve the model's sensitivity, while CCAM improves AUC and model classification accuracy.

[0154] Advantageously, convolutional neural network models with CCAM modules achieved average AUC of 0.8450, average accuracy of 0.8024, average sensitivity of 0.7138, and average specificity of 0.8261 on datasets such as TRCKDS. Results across multiple metrics demonstrate strong generalization ability with minimal performance loss on cross-ethnic datasets. CCAM can adaptively enhance the expression of local features associated with chronic kidney disease and their contribution to the final classification results.

[0155] Specifically, the TongRen Chronic Kidney Disease Study (TRCKDS) dataset mainly includes data on chronic kidney disease from 12,740 subjects who underwent ophthalmological examinations and physical examinations at Beijing Tongren Hospital, affiliated with Capital Medical University in Beijing, China, and 20,461 fundus images.

[0156] Preferably, see Figure 5 The upper-layer branch includes a second convolutional layer, Batch Normalization (BN), 1x1 convolution, ReLU function, 1x1 convolution (with 1 tip channel), BN, and a second Sigmoid function. The lower-layer branch includes Global Average Pooling (GAP), Batch Normalization (BN), Fully Connected (FC), FC, and a first Sigmoid function.

[0157] According to another preferred embodiment of the present invention, the classification result can be used to express the relevance of chronic kidney disease, for example, by outputting a score. Preferably, the classification result is used to provide a doctor with a reference. Preferably, the classification result can also be a probability or predicted probability of chronic kidney disease.

[0158] According to another preferred embodiment of the present invention, images of the user's left and right fundus are acquired, and data anonymization, ROI extraction, and contrast enhancement are performed. ROI refers to the region of interest.

[0159] According to another preferred embodiment of the present invention, see [link to previous document]. Figure 6 The trained convolutional neural network model with CCAM module processes the left and right fundus images in parallel, outputting the second feature maps for the left and right eyes respectively. These second feature maps are then aggregated along channels as the input feature tensor to a fully connected layer or GAP layer, finally outputting the classification result (first classification result). Advantageously, this invention integrates the features of both eyes, enabling feature fusion and effectively improving the classification accuracy for patients with chronic kidney disease.

[0160] According to another preferred embodiment of the present invention, the trained convolutional neural network model with the CCAM module is further used to process the left and right fundus images respectively, outputting the second feature map of the left eye and the second feature map of the right eye respectively, which are used as the feature tensors of the fully connected layer or GAP respectively, and finally outputting the corresponding second classification result (left eye) and third classification result (left eye). Preferably, the second classification result and the third classification result are compared to obtain a first difference. When the first difference is within a first predetermined range, the first classification result is output as the final classification result. Further, when the first difference is outside the first predetermined range, the larger value between the second classification result and the third classification result is taken as the final classification result. This is because when the classification results of the left and right eyes differ greatly, there is a potential error. At this time, the first classification result (fusion result) will deviate from the actual situation. In view of this situation, the present invention adopts the larger value between the second classification result and the third classification result (i.e., the one with a higher correlation with chronic kidney disease) as the final classification result, which is beneficial to locate the user for further examination by doctors, without the risk of omission or false detection, which may be quite disadvantageous to the user.

[0161] Preferably, when the first difference is within a first predetermined range, the one closer to the first classification result between the second and third classification results is determined as the nearest neighbor result, and the final classification result is output based on the first classification result and the nearest neighbor result. For example, the average of the first classification result and the nearest neighbor result can be taken. This method can further improve the accuracy of detection.

[0162] According to another preferred embodiment of the present invention, the decision-making basis region of the convolutional neural network model is visualized and a heatmap is generated using the SmoothGrad-CAM++ method. Advantageously, the heatmap obtained by the SmoothGrad-CAM++ method can improve the accuracy of the focusing region and can focus on abnormal regions.

[0163] The experimental results show that the convolutional neural network model (ResNet-CCAM) with CCAM module of this invention is compared with other mainstream classification network models SKNet, DenseNet, and Inception V4.

[0164] Performance of different models on TRCKDS

[0165]

[0166]

[0167] Note 1: AUC, accuracy, sensitivity, and specificity are expressed as mean (standard deviation), with bolded results representing the best results. * indicates that the paired t-test results between the best results and the best results are statistically significant at a significance level of 0.05.

[0168] It is evident that this invention has achieved the best results in multiple evaluation metrics, including AUC, accuracy, sensitivity, and specificity.

[0169] Finally, it should be noted that the convolutional neural network model with CCAM module in Example 2 can be used in conjunction with Example 1.

[0170] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fundus photograph classification system, characterized in that... include: The fundus image preprocessing unit is used to acquire fundus images and perform preprocessing to generate standardized fundus images; The fundus image segmentation unit is used to divide the processed fundus images into training and testing sets, and to augment the training set. The training and test set processing unit performs data preprocessing on the training and test sets to ensure that each batch of categories input into the model during the training phase is uniform. The model training unit is used to train branch neural network models based on ensemble learning strategies. The model fusion determination unit is used to perform model fusion to obtain the final detection model, enabling the classification of fundus images. The training branch neural network model specifically includes: 6.1) For each branch model, DenseNet-101 is used as the basic network structure. The last fully connected layer of DenseNet-101 is removed and a fully connected layer with an output dimension of 2 is added. The model parameters are transferred learning or retrained using parameters pre-trained based on ImageNet. 6.2) The loss function used in deep learning models is the focal loss function, the specific formula of which is: Where α and γ are manually set hyperparameters, specifically α = 0.6 and γ = 0.25, p is the probability value of the softmax output image being a positive sample, and y is the true label of the image, y = 1 indicates having chronic kidney disease, and y = 0 indicates not having chronic kidney disease. In order to prevent overfitting, L2 regularization is used to constrain the model. 6.3) The model optimization strategy uses the Adam optimizer; the Adam optimization steps are as follows: m t =β1m t-1 +(1-β1)g t Among them, g t Let be the gradient of the loss function L with respect to the parameter θ at time t, β1 and β2 are hyperparameters, β1 = 0.9 and β2 = 0.999, and m0 = 0, v0 = 0, α is the learning rate, and e is the denominator protection parameter; α = 0.0001. e = 1 * 10 -8 θ t The updated parameters; 6.4) Dropout strategy: During training, 40% of the random parameters in the later layers of the model do not participate in the backpropagation of gradients. 6.5) Early-Stop strategy: During the training of the branch model, if the absolute value of the deviation between the loss on the training set and the loss on the test set becomes significantly larger, training is stopped and the model before the deviation became too large is retained.

2. The fundus photograph classification system according to claim 1, characterized in that... The process of acquiring and preprocessing fundus photographs to generate standardized fundus images specifically includes: 1.1) For each fundus image, first extract the region of interest (ROI), extract the R channel component of the fundus image, and perform binarization. 1.2) The binarized R channel components are closed using a 5x5 convolution kernel. Seed points are selected (int(h / 2), int(w / 2)), where h is the height of the image, w is the width of the image, and int indicates rounding of the floating-point number. The connected component analysis algorithm is used to extract the circular or circular cross-sectional area located in the center of the fundus image as mask 1. 1.3) Find the minimum bounding rectangle of the binarized mask 1 image, and expand it outward by 10 pixels to form mask 2; 1.4) Perform pixel-wise operations on the original fundus image based on the mask 1 image. That is, for corresponding points in the original image and the binarized mask 1 image, if the value of the corresponding point on the mask 1 image is 1, then retain the values ​​of the three RGB channels of the original image; if the value of the corresponding point on the mask 1 image is 0, then set the values ​​of the three RGB channels of the original image to 0. 1.5) Based on the binarized mask 2 image, the original fundus photo is cropped to obtain an intermediate image, thereby achieving the purpose of extracting the ROI region; 1.6) Extract the G channel component of the intermediate image. For pixels with non-zero values ​​in the corresponding mask 1 image, use histogram equalization to enhance image contrast. Where N is the total number of pixels, v is the original gray level, s is the gray level after histogram equalization (ranging from 0 to 255), cdf is the cumulative distribution function of v, and round indicates rounding to the nearest integer; cdf min The minimum value of CDF; 1.7) Merge the intermediate image with the R channel component, the G channel component after histogram equalization, and the B channel component to obtain the preprocessed fundus image.

3. The fundus photograph classification system according to claim 2, characterized in that... The processed fundus images were divided into training and testing sets, specifically including: The original fundus image dataset was randomly divided into training and test sets using stratified sampling at a ratio of 4:

1. In the training set, stratified sampling is used again to randomly divide the data into five disjoint subsets: training set 1, training set 2, training set 3, training set 4, and training set 5.

4. The fundus photograph classification system according to claim 3, characterized in that... Data augmentation of the training set specifically includes: For each subset of fundus images in the training set, data augmentation is performed by randomly using any of the following methods: 3.1) Flip horizontally; 3.2) After cropping the image at a random ratio between 0 and 0.1, resize it to the original image size; 3.3) Smoothing is performed using a 5x5 Gaussian filter with random σ.

5. The fundus photograph classification system according to claim 4, characterized in that... Data preprocessing is performed on the training and test sets to ensure that each batch of classes input into the model during the training phase is evenly distributed. Specifically, this includes: 4.1) The dimensions of the fundus image after image preprocessing are: (h, w, c), where h is the height of the image, w is the width of the image, and c is the number of channels; the pixel value range of each channel is 0 to 255, and the pixel value range is normalized to map it to the interval of 0 to 1. 4.2) Set the hyperparameters for model training; The batch size and size l are set as follows: batch size = 16 and l = 600 for each training set. For the training set, input data is prepared in a 12:4 ratio of negative samples (no chronic kidney disease) and positive samples (with chronic kidney disease), where negative samples are sampled without replacement and positive samples are sampled with replacement; the batch size of the test set is 1, and both negative and positive samples are sampled without replacement. The input fundus image is resized to have dimensions (l,l,c), resulting in m tensor data in the format (l,l,c,n), where n is the batch size and m is the total number of batches.

6. The fundus photograph classification system according to claim 5, characterized in that... Ensemble learning strategies specifically include: Five branch models are trained on each of five disjoint training subsets. Each model uses four training subsets as the training set and leaves one training subset as the test set to adjust the parameters and branch the model. The training and parameter tuning processes for each branch model are independent of each other.

7. The fundus photograph classification system according to claim 6, characterized in that... Model fusion is performed to obtain the final detection model, which enables the classification of fundus images. Specifically, this includes: For the trained branch model, the final classification result of the model is obtained by using soft-voting during the testing process.

8. A method for processing fundus images, characterized in that... Includes the following steps: The first dataset is constructed, including preprocessing of the original fundus images and data filtering of the preprocessed fundus images. The preprocessing of the original fundus images includes data desensitization, ROI extraction and contrast enhancement. The data filtering of the preprocessed fundus images includes using the MobileNet v3 large model and soft voting method to filter the preprocessed fundus images and remove fundus images that do not meet the quality standards, thus constructing the first dataset. A convolutional neural network model with a CCAM module is constructed. The convolutional neural network model includes a backbone network and the CCAM module. The backbone network includes an input, a first convolutional layer, an MLP, and a first sigmoid function. The CCAM module includes an upper branch and a lower branch. The upper branch is used to extract CAM based on a first feature map using a second convolutional layer, and uses a second sigmoid function to differentiate the importance of features. The lower branch uses activation and squeezing operations to extract the channel weights of the first feature map, and combines them with CAM to perform a scaling operation to obtain a channel CAM map with a channel attention mechanism. The channel CAM map is then subjected to a Hadamard product operation with the first feature map and added to the first feature map before being input into a fully connected layer or a GAP. The first feature map is the output of the first convolutional layer. Using Focal Loss as the loss function, the convolutional neural network model with CCAM module is trained using the first dataset; Collect images of the user's left and right fundus and perform data anonymization, ROI extraction, and contrast enhancement; The trained convolutional neural network model with CCAM module is used to process the left and right fundus images in parallel, and output the second feature map of the left eye and the second feature map of the right eye respectively. Then, they are aggregated together with the channel as the axis and used as the feature tensor of the fully connected layer or GAP. Finally, the classification result is output, which is used to express the correlation of chronic kidney disease. The decision-making region of the convolutional neural network model is visualized and a heatmap is generated using the SmoothGrad-CAM++ method.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor, such that a computer device having the processor performs the method of claim 8.

Images