Multi-factor diabetic nephropathy progress prediction method based on fundus image focus segmentation

Through a deep learning model based on fundus images, the DKD progress is predicted, combined with clinical data, and the problem of strong subjectivity and time-consuming in DKD progress prediction is solved, and efficient and accurate prediction results are achieved, helping clinical diagnosis and treatment and improving patient prognosis.

CN118918122APending Publication Date: 2024-11-08TIANJIN POLYTECHNIC UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410950482.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art has a strong subjective and time-consuming problem in the prediction of progress of diabetic nephropathy (DKD), and it is impossible to effectively perform early renal microvascular changes in asymptomatic patients, resulting in irreversible nephropathy in some patients.

Method used

Using a multi-factor DKD progress prediction method based on fundus image lesion segmentation, the deformation hybrid Vision Transformer deep learning model is used to segment the fundus images. Combined with the patient's clinical data, a DKD progress prediction model is established through machine learning algorithms to achieve efficient and accurate predict.

Benefits of technology

It improves the accuracy and reliability of DKD progress prediction, reduces manual intervention, helps doctors develop more effective treatment plans, improves patient prognosis, and verifies the clinical applicability of the model through a rigorous training and testing process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918122A_ABST
    Figure CN118918122A_ABST
Patent Text Reader

Abstract

The invention provides a multi-factor diabetic nephropathy progress prediction method based on fundus image focus segmentation. The method comprises the following steps: 1) collecting a color fundus image and clinical information of a patient as sample data; 2) preprocessing the color eye fundus image, and extracting an eye fundus area; 3) normalizing the size of the image to adapt to a network structure; 4) constructing and training a deep learning model, and performing focus segmentation on the fundus image; 5) utilizing a segmentation result to extract a focus area proportion feature; (6) combining the area proportion of the focus with clinical data of the patient, forming high-risk and low-risk tags of nephropathy progression according to the change of urine protein, and constructing a multi-factor data set; 7) training a multi-factor diabetic nephropathy progress prediction model by using the multi-factor data set, and modeling by using a random forest algorithm; and 8) performing focus segmentation and feature extraction on the fundus image of the diabetic nephropathy patient, inputting focus features and clinical information into the trained multi-factor diabetic nephropathy progress prediction model, and outputting a diabetic nephropathy progress prediction result. The method can be widely applied to predicting the progress of diabetic nephropathy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a multi-factor DKD (diabetic nephropathy) progression prediction method based on fundus image lesion segmentation. The method is superior to traditional algorithms in terms of the accuracy of predicting DKD progression, has significant performance advantages, and belongs to the fields of image processing, medical image analysis, and deep learning. Background Art

[0002] Diabetic nephropathy (DKD) is one of the most common complications of diabetes and one of the main causes of end-stage renal disease (ESRD). In current clinical practice, the progression of DKD mainly depends on clinicians to conduct a comprehensive analysis of the patient's clinical indicators and then develop a personalized treatment plan. Renal puncture biopsy, which can clearly assess the degree of kidney damage and is the gold standard for diagnosing diabetic nephropathy, is an invasive examination with high cost and low patient acceptance. At the same time, current technology cannot achieve in vivo detection of early renal microvascular changes in asymptomatic diabetic patients. Once clinical proteinuria occurs, the kidney disease of some patients is irreversible.

[0003] In recent years, fundus images have been gradually applied to the study of diabetic complications due to their advantages of being non-invasive and able to reflect microvascular lesions. By analyzing fundus images, not only can retinal lesions be detected, but also the progression of diabetic nephropathy can be indirectly reflected. However, the analysis of fundus images currently still mainly relies on manual reading, which is limited by the experience and technical level of ophthalmologists and has problems such as strong subjectivity and long time consumption.

[0004] Deep learning is a branch of machine learning. Its application in image recognition, semantic segmentation and other fields has developed rapidly and has shown amazing accuracy. Compared with traditional algorithms, methods based on deep learning are efficient, reliable and flexible. In view of the current problem of subjective and time-consuming prediction of DKD progression in hospital treatment due to the limitations of ophthalmologists' experience and technical level, establishing an accurate DKD progression prediction model is a necessary and urgent task for both patients and hospitals.

[0005] In view of the above problems, the present invention proposes a multi-factor DKD (diabetic nephropathy) progression prediction method based on fundus image lesion segmentation. The method uses a deep learning model to accurately segment lesions in fundus images, obtains the proportion of fundus lesions, and then uses a machine learning method to build a model in combination with the patient's clinical data (such as age, gender, blood pressure, etc.), thereby predicting the progression of DKD. Summary of the invention

[0006] In order to achieve the above-mentioned purpose and improve the accuracy of predicting the progression of DKD, the present invention proposes a multi-factor DKD progression prediction method based on fundus image lesion segmentation. The method introduces quantitative information of the proportion of fundus lesions that has not been used in previous DKD progression prediction models, and accurately predicts the progression of DKD by combining individual clinical indicators of patients; accurately segmenting fundus lesions using a deformable hybrid Vision Transformer deep learning model to obtain accurate lesion area proportion data as an important characteristic variable for predicting the progression of DKD; combining with the patient's clinical data (such as age, gender, blood pressure, etc.) to form a multi-factor data set; using a machine learning algorithm to train the multi-factor data set to establish a DKD progression prediction model; implementing the technical solution of the present invention includes the following steps:

[0007] Step 1: Search for eligible patients by medical record number in the cooperating hospital, obtain their clinical information and fundus photos, annotate the lesions in the fundus photos at the pixel level, and then conduct follow-up examinations on the patients' conditions. At the same time, experienced doctors will annotate the patients' DKD progression;

[0008] Step 2: Preprocess the collected color fundus images, including cropping black edges to suppress unnecessary noise and image enhancement;

[0009] Step 3: Normalize the preprocessed fundus images to the same size of 512×512 as training and testing data for the subsequent segmentation network;

[0010] Step 4: Select fundus image samples for network training, construct a deformable hybrid Vision Transformer network model, and introduce dual-branch feature extraction, cross-attention feature fusion, and skip connection decoder to construct a deformable hybrid Vision Transformer network model. After training on a public data set, fine-tune the collected color fundus images to obtain a network training model. During training, observe the trend of the loss function. If the loss function of training and testing continues to rise, it means that the network structure is improperly designed. If the loss function of training and testing tends to remain unchanged, it means that the learning rate or batch size needs to be reduced. The network structure and learning rate are continuously adjusted to make the loss function continue to decrease, so as to obtain a better network training model.

[0011] Step 5: Use the segmentation results to extract the important feature of lesion area ratio, calculate the area of ​​the lesion region and divide it by the area of ​​the entire fundus image to obtain the key feature of lesion area ratio, and store it together with other clinical data;

[0012] Step 6: Combine the extracted features with the patient’s clinical data (such as age, gender, blood pressure, etc.) according to the medical record number to form a multi-factor data set;

[0013] Step 7: Manually divide the multi-factor dataset into a training set and a test set, use the training set to train the DKD progression prediction model, select the random forest algorithm for modeling, and evaluate the model performance on the test set;

[0014] Step 8: Perform lesion segmentation and feature extraction on fundus images of new patients, combine these features with clinical information to form a multi-factor dataset, input it into the trained prediction model, output the DKD progression prediction results, and feed the results back to doctors and patients, in this way, assist clinical decision-making;

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] By introducing quantitative information about the proportion of fundus lesions and combining it with the individual clinical indicators of the patient, the progression of the patient's condition can be more comprehensively reflected. The deformable hybrid Vision Transformer deep learning model is used to accurately segment the fundus lesions to ensure accurate identification of the lesion area. This innovative network structure achieves efficient feature extraction and fusion through dual-branch feature extraction, cross-attention feature fusion and jump connection decoder, thereby improving the performance of the model. From fundus image preprocessing, lesion segmentation, feature extraction to training and application of the prediction model, the entire process is highly automated, reducing manual intervention and improving work efficiency and consistency. The present invention can not only accurately predict the progression of DKD, but also help doctors formulate more effective treatment plans, assist clinical decision-making, and thus improve the prognosis of patients. Through data cooperation with cooperative hospitals, the method of the present invention has been verified by a large amount of real case data and has strong clinical applicability. The clinically verified prediction model can be promoted and applied in different medical environments and has broad practical application prospects. Through manual division of multi-factor data sets and strict training and testing processes, the prediction model of the present invention shows good performance indicators on the test set, such as high accuracy, sensitivity and specificity, ensuring the reliability of the model in practical applications. In summary, the present invention significantly improves the accuracy and reliability of DKD progression prediction through innovative technical means and rigorous process design, provides important technical support for clinical diagnosis and treatment, and has broad application value and promotion prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the overall framework, i.e., the abstract drawing;

[0018] Figure 2 Public dataset DDR;

[0019] Figure 2 (a) is the fundus image;

[0020] Figure 2 (b) is the EX lesion annotation of image (a);

[0021] Figure 2 (c) is the HE lesion annotation of image (a);

[0022] Figure 3 Fundus images from partner hospitals;

[0023] Figure 3 (a) is the fundus image;

[0024] Figure 3 (b) is the EX lesion annotation of image (a);

[0025] Figure 3 (c) is the HE lesion annotation of image (a);

[0026] Figure 4 Extract the original image and result image of the region of interest of the fundus image;

[0027] Figure 4 (a) is the original fundus image;

[0028] Figure 4 (b) is the fundus image after extracting the retinal area;

[0029] Figure 5 Image enhancement comparison chart;

[0030] Figure 5 (a) is the fundus image without image enhancement;

[0031] Figure 5 (b) is the fundus image after image enhancement;

[0032] Figure 6 Schematic diagram of the deformable hybrid Vision Transformer network;

[0033] Figure 7 (a) is a schematic diagram of deformed self-attention;

[0034] Figure 7 (b) is a schematic diagram of cross-attention feature fusion;

[0035] Figure 7 (c) is a schematic diagram of the skip connection decoder. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with specific implementation modes.

[0037] The overall framework schematic diagram of the present invention is as follows Figure 1 As shown, first, the clinical information and fundus photographs obtained from the cooperative hospital are annotated; the black edges of the image are removed by the region of interest algorithm, only the retinal area is retained, and image enhancement is performed to make the image more suitable for network training; the preprocessed image is resized to adapt to the scale of network training data and enhance accuracy; a deformable hybrid Vision Transformer network model is constructed, and the network model is pre-trained on a large public dataset DDR and then fine-tuned on the collected fundus images of patients to obtain a training model; the key feature of the lesion area ratio is calculated after segmenting the patient's fundus image; this feature is merged with the patient's clinical data into a multi-factor dataset; the multi-factor dataset is manually divided into a training set and a test set, the training set is used to train the DKD progression prediction model, and the model performance is evaluated on the test set.

[0038] The specific implementation process of the technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0039] 1. Experimental subjects

[0040] The fundus images used in the present invention for pre-training the fundus lesion segmentation network come from the public dataset DDR dataset, which contains 13673 color retinal fundus images with sizes ranging from 1088×1920 to 3456×5184. Among them, 757 images are pixel-level labeled with EX (hard exudate), HE (bleeding spot), MA (microaneurysm) and SE (soft exudate), such as Figure 2 As shown in the figure. Specifically, 486 images are labeled with EX, 570 images are labeled with MA, 239 images are labeled with SE, and 601 images are labeled with HE. The DDR dataset provides the original division, 383 images are used for training, 149 images are used for verification, and the remaining 225 images are used for testing. Because the shooting machines are different, in order to better adapt to the characteristics of the fundus images of the cooperative hospitals, the segmentation model needs to be fine-tuned on the images of the cooperative hospitals, so the hospital images are also labeled with lesions, such as Figure 3 The included patients will also be followed up, and their DKD progression will be marked by senior doctors after follow-up.

[0041] 2. Image Preprocessing

[0042] 2.1 Extracting Region of Interest

[0043] As shown in the attached figure Figure 4As shown in the figure, the region of interest extraction algorithm is designed to obtain the fundus area in the image, making the image training content clearer and preparing for subsequent work. The collected fundus image has a large area of ​​black edges. In order to reduce the calculation time and suppress irrelevant noise information, the region of interest extraction algorithm is used to remove the black edges of the image and only retain the retinal structure area.

[0044] 2.2 Image Enhancement

[0045] The image is enhanced using contrast-limited adaptive histogram equalization (CLAHE), with processing parameters set to ClipLimit=2 and GridSize=8. Figure 5 As shown, after image enhancement, the lesion area becomes more prominent.

[0046] 2.3 Size Adjustment

[0047] The data scale has a huge impact on the performance of the training network. In order to make the fundus images more suitable for network training requirements, each fundus image is resized and normalized to the same size of 512×512. Finally, the image pixel values ​​are normalized from 0 to 255 to 0 to 1 for subsequent network processing and as training data for subsequent networks.

[0048] 3. Lesion segmentation

[0049] The deformable hybrid Vision Transformer network can accurately segment fundus lesions by introducing dual-branch feature extraction, cross-attention feature fusion and skip connection decoder, providing support for subsequent DKD progression prediction. The network structure diagram is shown in the figure. Figure 6 shown.

[0050] 3.1 Dual-branch feature extraction

[0051] In order to make the segmentation network take into account both local and global features when extracting features, CNN and deformable Vision Transformer are used to extract image features simultaneously.

[0052] The CNN branch includes a pre-processing block, a pooling layer, a residual block, and an output processing block. Each pooling layer reduces the feature map by half, and the residual block performs in-depth extraction and fusion of the feature map. The specific implementation is as follows: 1) First, the input image passes through the pre-processing block, which performs preliminary processing on the input image, including two 3×3 convolutional layers, each followed by a BatchNormalization (BN) and ReLU activation function. The pre-processing block converts the number of channels of the input image from 3 (RGB) to 24, and enhances the feature representation through a nonlinear activation function; 2) Then the pooling layer and the residual block are performed alternately to gradually reduce the feature map and extract features. The pooling layer uses Max Pooling, the convolution kernel size is 2, and the stride is 2. This design can reduce the size of the feature map by half after the pooling layer. When the feature map is input into the residual block, it will first pass through conv(3×3)->BN->ReLU->conv(3×3)->BN, and then the output result will be directly added to the original input to form a residual connection. After addition, the result will be transformed nonlinearly by the ReLU activation function; 3) The output processing block contains two 3×3 convolutional layers, each followed by a Batch Normalization and ReLU activation function for further processing and refining the previously extracted features; 4) Finally, the prediction module is passed, which consists of two 1×1 convolutional layers and a Sigmoid activation function to output a probability map.

[0053] The deformable Vision Transformer can extract the global features of the image and adapt to the variable shape of fundus lesions. The method of implementing the deformable Vision Transformer is as follows: 1) replace the self-attention mechanism in the Vision Transformer with the deformable self-attention mechanism proposed in the present invention, as shown in the schematic diagram Figure 7 As shown in the middle figure (a); 2) deformable self-attention introduces the learned offset offset into self-attention, adds the offset offset to the q in the self-attention, and then performs self-attention calculation with k and v to enhance the model's adaptability to geometric deformation; 3) The offset offset learned in deformable self-attention is calculated by the convolutional layer with kernel1 size = 3, pad = 1, stride = 1 and the output channel number is 2*kernel size*kernel size.

[0054] 3.2 Cross-Attention Feature Fusion

[0055] The present invention aims at the problem of fusing local features and global features, and introduces cross-attention feature fusion, which can effectively fuse the features extracted by the two branches. The schematic diagram is shown in Figure 7 As shown in the middle figure (b).

[0056] In the feature fusion process, the self-attention idea in Transformer is used for reference. Specifically, the global feature is used as the query (query, denoted as q), and the local feature is used as the key (key, denoted as k) and value (value, denoted as v). The feature fusion is performed through the following formula:

[0057]

[0058] Among them, d k is the dimension of the key. Through the above formula, local features and global features are effectively fused to generate hierarchical features: stage1, stage2, stage3 and stage4.

[0059] 3.3 Skip Connection Decoder

[0060] During the decoding process, the decoder retains high-resolution features by directly connecting the fusion features and decoder features at the same level. These connections can help the decoder better restore the detailed information of the image. Assume that the feature of a certain layer of the decoder is D i , the corresponding level fusion feature is stage i , D is connected through skip connections i The fusion with stage is recorded as D′ i =concat(D i , stage i ), and then upsampled by transposed convolution with kernel size = 2 and stride = 2 to obtain the decoder feature D of the previous layer i -1, schematic diagram as Figure 7 As shown in the middle figure (c).

[0061] 4. Constructing a multi-factor dataset

[0062] The segmentation results are used to extract the important feature of lesion area ratio. The area of ​​the lesion region is calculated and divided by the area of ​​the entire fundus image to obtain the lesion area ratio. The lesion area ratio is then combined with the patient's clinical data (such as age, gender, blood pressure, etc.) according to the medical record number to form a multi-factor data set.

[0063] 5. Build a predictive model

[0064] The multifactorial dataset was manually divided into a training set and a test set. The random forest algorithm was selected for modeling. The training set was used to train the DKD progression prediction model, and the model performance was evaluated on the test set. Lesion segmentation and feature extraction were performed on the fundus images of new patients. These features were combined with clinical information to form a multifactorial dataset, which was input into the trained prediction model. The DKD progression prediction results were output and fed back to doctors and patients. In this way, clinical decision-making was assisted.

[0065] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It should be understood that the present invention is not limited to the implementation schemes described herein, and the purpose of describing these implementation schemes is to help those skilled in the art to practice the present invention. Any person skilled in the art can easily make further improvements and perfections without departing from the spirit and scope of the present invention, so the present invention is only limited by the content and scope of the claims of the present invention, which is intended to cover all alternatives and equivalents included in the spirit and scope of the present invention defined by the appended claims.

Claims

1. A multi-factorial diabetic nephropathy progression prediction method based on fundus image lesion segmentation, comprising the following steps: Step 1: Collect and annotate color fundus images and clinical information of patients with diabetic nephropathy; Step 1-1: At the beginning of the study, color fundus images and clinical information of patients with diabetic nephropathy were collected. The clinical information included age at visit, gender, disease duration, systolic blood pressure (SBP), diastolic blood pressure (DBP), urea (Urea), uric acid (UA), 24-hour urine microalbumin (mAlb), 24-hour urine total protein (TP), urine albumin / creatinine ratio (UACR), and estimated glomerular filtration rate (eGFR); Step 1-2: Annotate the lesions in the color fundus image at the pixel level; Step 1-3: Follow up the patients and collect their urine protein data after six months. Patients with increased urine protein are marked as having a high risk of renal disease progression, while patients with no change in urine protein or decreased urine protein are marked as having a low risk of renal disease progression; Step 2: Preprocessing the color fundus image; Step 2-1: Crop the black border in the fundus image; Step 2-2: Use contrast-limited adaptive histogram equalization to enhance the image using CLAHE; Step 2-3: Normalize the image pixel values ​​from 0 to 255 to 0 to 1; Step 3: Normalize the fundus image preprocessed in step 2 to the same size of 512×512; Step 4: Introduce dual-branch feature extraction, cross-attention feature fusion, and skip connection decoder to construct a deformable hybrid Vision Transformer network model for lesion segmentation and perform model training; Step 4-1: Construct a deformable hybrid Vision Transformer lesion segmentation network model. 1) Use the CNN network and the deformable Vision Transformer network as two branches to extract local features and global features of the image respectively; 2) Taking the global feature as the query q, the local feature as the key k and the value v, cross attention is introduced to perform feature fusion to obtain hierarchical features stage1, stage2, stage3 and stage4; 3) Introducing skip connections in the decoder stage, directly connecting the fused features and the decoder features at the same level to retain high-resolution features to better restore detail information; Step 4-2: Train the deformable hybrid Vision Transformer lesion segmentation network model, first train it on the public dataset, then train and fine-tune it on the color fundus image processed in step 3 to obtain a trained lesion segmentation model; Step 5: Use the lesion segmentation model trained in step 4 to segment the lesions in the fundus image, calculate the area of ​​the lesion region and divide it by the area of ​​the entire fundus image to obtain the feature of the lesion area ratio; Step 6: Combine the lesion area ratio with the patient's age at visit, gender, disease course, systolic blood pressure (SBP), diastolic blood pressure (DBP), urea (Urea), uric acid (UA), 24-hour urine microalbumin (mAlb), 24-hour urine total protein (TP), urine albumin / creatinine ratio (UACR), and estimated glomerular filtration rate (eGFR) to form a multifactorial diabetic nephropathy progression dataset; Step 7: Using the random forest method, a diabetic nephropathy progression prediction model was trained based on the multi-factor diabetic nephropathy progression dataset; Step 8: Predict the risk of diabetic nephropathy in patients with diabetic nephropathy; Step 8-1: Collect color fundus images and clinical information of patients with diabetic nephropathy, including age at visit, gender, disease course, systolic blood pressure (SBP), diastolic blood pressure (DBP), urea (Urea), uric acid (UA), 24-hour urine microalbumin (mAlb), 24-hour urine total protein (TP), urine albumin / creatinine ratio (UACR), and estimated glomerular filtration rate (eGFR); Step 8-2: Segment the lesions on the fundus image of the patient and calculate the lesion ratio. Combine the clinical information and input it into the diabetic nephropathy progression prediction model trained in step 7. The model outputs the prediction result of the diabetic nephropathy progression of the patient.

2. According to claim 1, a multi-factor diabetic nephropathy progression prediction method based on fundus image lesion segmentation is characterized in that: In step 4, the deformed Vision Transformer replaces the self-attention mechanism in the Vision Transformer with deformable self-attention, and adds the learned offset offset to the self-attention q.

3. According to claim 1, a multi-factor diabetic nephropathy progression prediction method based on fundus image lesion segmentation is characterized in that: In step 4, the cross-attention feature fusion method is to add the learned offset to q and then perform self-attention calculation with k and v to enhance the model's adaptability to geometric deformation: Among them, d k is the dimension of key k. Through the above formula, local features and global features are effectively integrated to generate hierarchical features: stage 1 , stage 2 , stage 3 and stage 4 .

Citation Information

Cited By

  • Method and device for predicting diabetic nephropathy, electronic equipment and storage medium

    CN119517383A

  • Methods, apparatus, electronic devices, and storage media of predicting diabetic nephropathy

    CN119517383B

  • Non-invasive diabetic nephropathy grading and predicting method supporting convenient screening

    CN120452803A

  • Evaluation method and device for fundus lesion of chronic kidney disease

    CN121483579A