A non-invasive method for diabetic nephropathy grading and prediction with convenient screening

Abstract

The application relates to a non-invasive diabetic nephropathy grading and prediction method supporting convenient screening, which comprises the following steps: collecting color fundus images and non-invasive clinical characteristics of diabetic patients, pre-processing to obtain a multi-modal data set, and setting annotations of multiple severity degrees of diabetic nephropathy; constructing a deep learning model A1 to obtain a diabetic nephropathy severity prediction probability distribution based on the color fundus images according to the multi-modal data set; constructing a deep learning model A2 to obtain a diabetic nephropathy severity prediction probability distribution based on clinical data according to the multi-modal data set; constructing a fusion model, the fusion model fuses the obtained diabetic nephropathy prediction probabilities through an adaptive weight distribution algorithm, dynamically adjusts the weights according to the prediction results output by each model, calculates a comprehensive diabetic nephropathy severity score, and outputs a final diabetic nephropathy prediction grading result according to the score.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image processing and artificial intelligence, and particularly relates to a fine prediction method for non-invasive diabetic nephropathy severity based on fundus images and non-invasive clinical features. BACKGROUND

[0002] Diabetic nephropathy is one of the most common and most threatening complications of type 2 diabetes, and is also the main cause of end-stage diabetic nephropathy. Diabetic nephropathy significantly increases the morbidity and mortality risk of patients, and also causes a heavy economic and social burden to the global health care system. Early detection and intervention of diabetic nephropathy is crucial for delaying disease progression and improving patient prognosis. However, the current screening coverage of diabetic nephropathy is low, and the main reasons include the lack of convenience of traditional screening methods and the low patient compliance. For patients who have been diagnosed with diabetes for a certain period of time, regular urine albumin / creatinine ratio testing and estimated glomerular filtration rate evaluation should be performed to improve the early diagnosis rate of diabetic nephropathy and develop precise disease management strategies, so as to reduce the risk of renal failure and high mortality. Despite this, there are still significant limitations in traditional diagnostic methods.

[0003] Traditional diagnostic methods for diabetic nephropathy mainly include urine tests, blood tests, imaging examinations, and kidney biopsy. Among them, imaging examinations are expensive, kidney biopsy is invasive and has potential complication risks, while urine tests and blood tests, although they have certain universality as routine methods, are limited in convenience due to the dependence on specific test conditions (such as fasting or early drug withdrawal). In addition, since patients with early diabetic nephropathy usually have no obvious clinical symptoms, the willingness of patients to actively screen is low, resulting in a high rate of missed diagnosis, which delays the diagnosis and treatment opportunity of the disease. Therefore, developing a non-invasive, convenient and low-cost diagnostic method for diabetic nephropathy has become an important requirement for current medical research.

[0004] With the development of medical imaging technology and deep learning methods, researchers have begun to explore diabetic nephropathy prediction methods based on fundus images. However, existing research still has significant limitations. First, existing methods usually only achieve binary classification (with or without nephropathy) or ternary classification (early, medium, and late stage) of diabetic nephropathy, which is difficult to meet the clinical needs of continuous grading and fine management of disease progression. Second, existing methods mainly rely on traditional convolutional neural networks or general convolutional neural network architectures (such as ResNet, VGG), which are limited by local receptive fields and fixed weight mechanisms, making it difficult to accurately identify complex pathological changes in fundus images. In addition, current methods rely on urine tests and blood tests as feature inputs, which have certain prediction capabilities, but in actual application, their invasiveness and preparation requirements seriously affect patient screening compliance. SUMMARY

[0005] The application aims to overcome the deficiencies of the prior art and provide a non-invasive diabetic nephropathy grading and prediction method supporting convenient screening, which is a diabetic nephropathy severity prediction method based on fundus images and clinical characteristics to realize non-invasive, efficient and accurate disease screening and severity assessment.

[0006] The application solves the technical problems by the following technical solutions:

[0007] A non-invasive diabetic nephropathy grading and prediction method supporting convenient screening comprises the following steps:

[0008] Step 1: Collecting color fundus images and non-invasive clinical characteristics of diabetic patients and pre-processing to obtain a multi-modal data set, and setting labels of multiple severities of diabetic nephropathy;

[0009] Step 2: Constructing a deep learning model A1 to train the color fundus images in the multi-modal data set of step 1 to obtain a diabetic nephropathy severity prediction probability distribution based on color fundus images ;

[0010] Step 3: Constructing a deep learning model A2 to train the non-invasive clinical characteristics in the multi-modal data set of step 1 to obtain a diabetic nephropathy severity prediction probability distribution based on clinical data ;

[0011] Step 4: Constructing a fusion model, which fuses the diabetic nephropathy prediction probabilities obtained in steps 2 and 3 by an adaptive weight distribution algorithm, dynamically adjusts the weights according to the prediction results output by each model, calculates a comprehensive diabetic nephropathy severity score, and outputs the final diabetic nephropathy prediction grading result according to the score.

[0012] Furthermore, the step 1 comprises the following steps:

[0013] Step 1.1: Collecting sample data of diabetic patients and removing abnormal sample data of diabetic patients;

[0014] Step 1.2: Pre-processing the color fundus images in the sample data;

[0015] Step 1.3: Pre-processing the non-invasive clinical characteristics in the sample data;

[0016] Step 1.4, the pre-processed sample data is divided into 6 grades of diabetic nephropathy labels: including grade 0: normal; grade 1: early warning; grade 2: mild diabetic nephropathy; grade 3: moderate diabetic nephropathy; grade 4: severe diabetic nephropathy; and grade 5: end-stage diabetic nephropathy;

[0017] Step 1.5, a multi-modal data set is constructed, including color fundus images of diabetic patients, non-invasive clinical features, and corresponding grading labels, wherein the non-invasive clinical features include: age, gender, diabetes duration, and history of hypertension.

[0018] Moreover, the deep learning model A1 in the step 2 includes a double-branch Transformer, a multi-scale feature extraction module, a double-sparse attention module, and a hierarchical feature interaction module, the input ends of the double-branch Transformer branch one and the double-branch Transformer branch two are respectively connected with the multi-scale feature extraction module, the double-sparse attention module, the Transformer module, and the hierarchical feature interaction module connected in sequence; and the specific working method of the deep learning model A1 is:

[0019] Step 2.1, the color fundus images are respectively input to the double-branch Transformer branch one and the double-branch Transformer branch two at different resolutions;

[0020] Step 2.2, the multi-scale feature extraction module extracts features from the images of the double-branch Transformer branch one and the double-branch Transformer branch two and obtains two feature sequences X1 and X2;

[0021] Step 2.3, the double-sparse attention module fuses and filters the results before and after the multi-scale feature extraction module, and obtains features and , and inputs the features and to the double-branch Transformer module to obtain deep feature representations and ;

[0022] Step 2.4, the hierarchical feature interaction module respectively connects the features and output by the double-sparse attention module with and obtained after the Transformer module to obtain and , and completes the double-branch information interaction and the fusion of features at different levels to obtain the final feature information .

[0023] Moreover, the multi-scale feature extraction module in the constructed deep learning model A1 includes: an initial convolution module, a multi-scale convolution enhancement module, a size unification module and an attention fusion output module, wherein the initial convolution module, the multi-scale convolution enhancement module, the size unification module and the attention fusion output module are connected in sequence; the initial convolution module is used to extract the initial features, the multi-scale convolution enhancement module extracts the multi-scale structural features based on the initial features, the size unification module unifies the size of the multi-scale structural features, and the attention fusion output module introduces a hybrid attention mechanism to unify the modeling of the multi-scale features. and .

[0024] Moreover, the specific working method of the dual sparse attention module in the deep learning model A1 is as follows: the extracted dual-branch multi-scale features X1 and X2 are respectively subjected to attention analysis with the original branch features Y1 and Y2 to obtain the attention analysis result M. Combined with the attention analysis result M, the dynamic threshold screening mechanism and the adaptive Top-K screening mechanism are used to perform sparse screening respectively, and the overlapping verification module is used to verify the results, and a fusion attention matrix is ​​generated. The feature channel weights are adjusted accordingly, and the output features of the two branches are obtained as follows: and .

[0025] Moreover, the working method of the hierarchical feature interaction module in the deep learning model A1 network is to combine the image features from different resolution branches and After Transformer feature extraction and , divide out and Global characteristics of 、 and local features 、 ;

[0026] The global features of different branches obtained by Transformer feature extraction 、 Local features of another branch 、 Splicing to get and ;calculate and 、 and The similarity weight matrix between them is obtained, and the enhanced global features are obtained. and ; Then, and 、 and The information interaction is completed by cross attention mechanism respectively, and finally the global feature and is added and fused to obtain feature information and :

[0027]

[0028]

[0029] The obtained and respectively extract global information and , and then splice the global information with the obtained to obtain , and splicing , finally and are spliced to obtain the final feature information ; the is operated by softmax function to obtain the normalized probability distribution :

[0030] .

[0031] Moreover, the deep learning model A2 in step 6 includes an input layer, a hidden layer and an output layer, wherein the hidden layer adopts ReLU activation function and normalization to process data, and the output layer calculates the prediction probability through a fully connected layer .

[0032] Moreover, the adaptive weight distribution method adopted in step 4 is:

[0033]

[0034] wherein, represents the prediction probability distribution based on fundus image calculated by the deep learning model A1; represents the prediction probability distribution based on non-invasive clinical features calculated by the deep learning model A2; is the weight coefficient adaptively calculated based on the input sample features;

[0035] The final prediction class is obtained by maximum value index operation on the fused probability distribution :

[0036] ​

[0037] Predicted class The value range of the predicted class is , respectively corresponding to normal, early warning, mild diabetic nephropathy, moderate diabetic nephropathy, severe diabetic nephropathy, and end-stage diabetic nephropathy.

[0038] The beneficial effects of the present application are:

[0039] 1. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening of the present application realizes the precise prediction of the severity of completely non-invasive diabetic nephropathy by extracting image lesion features by using a deep learning model A1, combining a multilayer perceptron to process non-invasive clinical features, and finally realizing disease severity grading through adaptive weight distribution.

[0040] 2. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening of the present application has convenient screening and improves patient compliance: the method uses fundus image analysis combined with non-invasive clinical features to replace traditional urine tests, blood tests, and kidney biopsy methods, and provides patients with a more convenient, safe, and comfortable screening method.

[0041] 3. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening of the present application designs a model architecture for targeted tasks: the method innovatively combines convolutional neural networks and Transformers to construct a deep learning model A1, extracts multiscale lesion features of fundus images, combines a multilayer perceptron to process non-invasive clinical features, enhances the comprehensive recognition ability of the disease, and through adaptive weight distribution, makes the color fundus image and non-invasive clinical features work together to improve the prediction accuracy and generalization ability.

[0042] 4. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening of the present application realizes fine-grained grading and precise personalized management: traditional methods are mostly limited to binary classification (with or without nephropathy) or ternary classification (early, middle, and late) of diabetic nephropathy, and the present application can realize six-level refined grading (0 to 5) of the severity of diabetic nephropathy for precise description of the disease progression state. This grading system is helpful for personalized treatment decisions, and doctors can adjust intervention strategies according to the prediction results to optimize the treatment plan for patients and delay disease progression.

[0043] 5、The application is a non-invasive diabetic nephropathy grading and prediction method supporting convenient screening, the deep learning model A1 is specially designed for diabetic nephropathy fundus image lesion patterns, can effectively identify typical pathological features such as micro-exudation and micro-aneurysm, and is significantly superior to the adaptability and prediction effect of a general image classification model in this task, the application combines multi-modal data through deep learning technology, without invasive and urine test indicators, and conveniently realizes early screening and 6-grade severity prediction of diabetic nephropathy, and is suitable for clinical auxiliary diagnosis and disease progression prediction. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is a schematic diagram of the overall framework of the application;

[0045] Figure 2 It is a schematic diagram of the deep learning model A1 for identifying fundus image lesions of different severity of diabetic nephropathy;

[0046] Figure 3 It is a schematic diagram of different models for identifying fundus image lesions of diabetic nephropathy,

[0047] Wherein a is the true label, b is the deep learning model A1 for identifying fundus image lesions of diabetic nephropathy, c is the model VIT for identifying fundus image lesions of diabetic nephropathy, and d is the model ResNet34 for identifying fundus image lesions of diabetic nephropathy;

[0048] Figure 4 It is a schematic diagram of the deep learning model A1 of the application;

[0049] Figure 5 It is a schematic diagram of the multi-scale feature extraction module of the application;

[0050] Figure 6 It is a schematic diagram of the double sparse attention module of the application;

[0051] Figure 7 It is a schematic diagram of the hierarchical feature interaction module of the application. DETAILED DESCRIPTION

[0052] The application will be further described in detail through specific embodiments below, the following embodiments are only descriptive and not limiting, and the protection scope of the application cannot be limited by this.

[0053] A non-invasive diabetic nephropathy grading and prediction method supporting convenient screening, as shown in Figure 1 The application comprises the following steps:

[0054] Step 1, collect color fundus images and non-invasive clinical features of diabetic patients and preprocess to obtain a multi-modal data set, and set the labels of various severities of diabetic nephropathy.

[0055] Step 1.1, collect sample data of diabetic patients, and exclude abnormal data sample data of diabetic patients. Exclude sample data of patients who are not type 2 diabetic nephropathy, have active malignancy, have a history of kidney transplantation, or have a follow-up interval of less than 3 months.

[0056] Step 1.2, pre-process the color fundus images in the sample data.

[0057] Step 1.3, pre-process the non-invasive clinical features in the sample data. Non-invasive clinical features are obtained through an electronic medical record system: blood test indicators, urine test indicators, diabetes duration, history of hypertension, patient gender and age;

[0058] Step 1.4, divide the pre-processed sample data into 6 levels of diabetic nephropathy labels: 0 level: normal; 1 level: early warning; 2 level: mild diabetic nephropathy; 3 level: moderate diabetic nephropathy; 4 level: severe diabetic nephropathy; 5 level: end-stage diabetic nephropathy.

[0059] Step 1.5, construct a multi-modal data set, including color fundus images of diabetic patients, non-invasive clinical features and corresponding classification labels, wherein the non-invasive clinical features include: age, gender, diabetes duration and history of hypertension.

[0060] Step 2, construct a deep learning model A1 to train the color fundus images in the multi-modal data set of step 1, and obtain a diabetic nephropathy severity prediction probability distribution based on color fundus images .

[0061] As shown in Figure 4 , the deep learning model A1 includes a dual-branch Transformer, a multi-scale feature extraction module, a dual-sparse attention module, and a hierarchical feature interaction module. The input ends of the dual-branch Transformer branch one and the dual-branch Transformer branch two are connected to the multi-scale feature extraction module, the dual-sparse attention module, the Transformer module, and the hierarchical feature interaction module in sequence. The specific working method of the deep learning model A1 is:

[0062] Step 2.1, input the color fundus images at different resolutions to the dual-branch Transformer branch one and the dual-branch Transformer branch two respectively;

[0063] Step 2.2, the multi-scale feature extraction module extracts features from the image of the double-branch Transformer branch one and the image of the double-branch Transformer branch two and obtains two feature sequences X1 and X2;

[0064] Step 2.3, the double sparse attention module fuses and screens the results before and after the multi-scale feature extraction module to obtain features and , and inputs the features and to the double-branch Transformer module to obtain deep feature representations and .

[0065] Step 2.4, the hierarchical feature interaction module performs residual connection on the features and output by the double sparse attention module and the and obtained after the Transformer module to obtain and , and completes the double-branch information interaction and the fusion of different hierarchical features to obtain the final feature information .

[0066] As shown in Figure 5 , the multi-scale feature extraction module in the deep learning model A1 includes an initial convolution module, a multi-scale convolution enhancement module, a size unification module, and an attention fusion output module, wherein the initial convolution module, the multi-scale convolution enhancement module, the size unification module, and the attention fusion output module are connected in sequence.

[0067] The initial convolution module first uses a 7x7 deep convolution to capture the large receptive field intermediate feature map of the obvious lesion, and then uses channel normalization to strengthen the consistency and stability of the channel dimension features, followed by pointwise convolution and GELU activation for channel mapping and nonlinear transformation, and then uses a standard 3x3 convolution to extract small receptive field information of blood vessels and microaneurysms, and finally combines batch normalization and ReLU activation function to obtain the final output:

[0068]

[0069]

[0070] wherein, is a convolution operation, is layer normalization, is an activation function, is batch normalization, an activation function, is an intermediate feature map, is an initial feature extracted by the initial convolution module;

[0071] The multi-scale convolution enhancement module enhances the initial feature extracted by performing a pooling operation, and then extracts multi-scale structural features at different spatial resolutions layer by layer to enhance the semantic perception ability of the network for local lesion regions at different scales;

[0072]

[0073] wherein, is the maximum pooling layer, and the output preliminary feature is represented as , represents the number of channels;

[0074]

[0075] wherein, is a multi-scale structural feature, , , , ;

[0076] The size unification module unifies the size of the multi-scale structural feature and maps it to a fixed dimension through 1×1 convolution layer by layer;

[0077]

[0078] wherein, is a multi-scale feature with the same number of channels, , , , ;

[0079] As shown in Figure 6 , the attention fusion output module introduces a hybrid attention mechanism to uniformly model the multi-scale feature , and obtains feature information of three scales of two branches and , which are spliced into two one-dimensional sequences to obtain and :

[0080]

[0081]

[0082] wherein, is spatial attention, is channel attention, ​represents element-wise product, are the features after spatial attention and channel attention, respectively and , It is a splicing operation.

[0083] like Figure 7 As shown, the dual sparse attention module in deep learning model A1 is responsible for feature screening and sparse modeling. Through joint modeling with dynamic thresholds and an adaptive Top-K mechanism, it effectively eliminates redundant background, highlights key regional features of diabetic nephropathy lesions, and improves expressiveness and discriminability. At the same time, the module possesses dynamic selection capabilities under heterogeneous structures, enhancing the model's adaptability to complex lesions and cross-regional recognition stability. The sparse attention matrix it outputs serves as a high-confidence guide, helping to improve the feature quality of Transformer's global modeling and the accuracy of hierarchical predictions. Its specific working method is as follows:

[0084] Perform attention analysis on the extracted dual-branch multi-scale features X1 and X2 and the original branch features Y1 and Y2 respectively:

[0085]

[0086] in, is a scaling parameter, multi-head attention is performed on n groups of Q, K and V, and an output with d=C / n channel dimension is obtained, Different scale features, It is the original branch feature that has not been processed by multiple scale feature extraction modules. ;

[0087] Dynamic threshold screening mechanism generates adaptive thresholds through lightweight networks , performs masking operations on low response areas in the input features

[0088]

[0089]

[0090] in, represents the Sigmoid function, and is the learning parameter, It means taking the mean of the input matrix. As a nonlinear activation function, it is only activated when the response in the feature exceeds the dynamic threshold will be retained only when It is a dynamic threshold screening mechanism;

[0091] The adaptive Top-K screening mechanism dynamically determines the sparse retention factor based on the statistical characteristics of feature responses, and selects the top K feature information with the highest response value:

[0092]

[0093]

[0094] wherein, and are the attention matrices The mean and standard deviation on each sequence ensure that the sparsity factor is moderate, is the length of the attention score vector, Downward rounding ensures that the result of adaptive sparse dimension selection is an integer;

[0095] The overlap verification module compares the consistency of the above two sparse screening results, generates a fused attention matrix, and adjusts the feature channel weight accordingly:

[0096]

[0097]

[0098] wherein, is an element-wise multiplication used to select the overlapping area of the two matrices; Softmax is a normalization that ensures the attention meets the probability distribution characteristics, and the output features of the two branches are and .

[0099] The working method of the hierarchical feature interaction module in the deep learning model A1 network is as follows: the image features and from different resolution branches are subjected to Transformer feature extraction to obtain and , and the global features and , , and the local features , are divided.

[0100] The global features , of different branches obtained by Transformer feature extraction are spliced with the local features , of the other branch to obtain and ; and and 、 and The similarity weight matrix between them is obtained, and the enhanced global features are obtained. and ; Then, and 、 and The information interaction is completed through the cross attention mechanism, and finally the enhanced global features are and Add and fuse to obtain feature information and :

[0101]

[0102]

[0103] To get and Extract global information separately and , and then the global information and obtained Splicing , and Splicing , and finally and Splice to get the final feature information ;Will Perform softmax function operation to obtain normalized probability distribution :

[0104] .

[0105] The fundus image for training the deep learning model A1 is from a multi-modal data set, the fundus photos and clinical features of diabetic patients are obtained from cooperative hospitals, non-type 2 diabetic nephropathy patients, active malignant tumor patients, patients with a history of kidney transplantation and samples with a follow-up interval of less than 3 months are excluded, and at the same time, it is ensured that the corresponding color fundus photos are clear and unobstructed. Then, under the guidance of doctors, each sample is graded and assigned a corresponding label (0-5) according to the detailed clinical features of the patient. Finally, the data set contains 3807 color fundus images and corresponding non-invasive clinical features, and the size of the color fundus images ranges from 1024x1024 to 2992x2000. Among them, 1698 pictures are normal, 886 pictures are early warning, 335 pictures are mild diabetic nephropathy, 481 pictures are moderate diabetic nephropathy, 270 pictures are severe diabetic nephropathy, and 137 pictures are end-stage diabetic nephropathy. 3045 images in the data set are used for training, and 762 images are used for testing. The non-invasive clinical features corresponding to each fundus photo are: gender, age of visit, duration of diabetes, and presence or absence of hypertension history.

[0106] Step 3, constructing a deep learning model A2, training the non-invasive clinical features in the multi-modal data set in step 1, and obtaining the prediction probability distribution of diabetic nephropathy severity based on clinical data .

[0107] The deep learning model A2 includes an input layer, a hidden layer and an output layer, wherein the hidden layer adopts ReLU activation function and normalization to process data, and the output layer calculates the prediction probability through a fully connected layer .

[0108] Step 4, constructing a fusion model, the fusion model fuses the prediction probabilities of diabetic nephropathy obtained in steps 2 and 3 through an adaptive weight distribution algorithm, dynamically adjusts the weight according to the prediction results output by each model, calculates the comprehensive diabetic nephropathy severity score, and outputs the final diabetic nephropathy prediction classification result according to the score.

[0109] The adaptive weight distribution method adopted is:

[0110]

[0111] Wherein, represents the prediction probability distribution based on the fundus image calculated by the deep learning model A1; represents the prediction probability distribution based on the non-invasive clinical features calculated by the deep learning model A2; is the weight coefficient calculated adaptively based on the input sample features;

[0112] The fusion probability distribution is calculated by performing a maximum index operation to obtain a final prediction category :

[0113]

[0114] the prediction category ranges from 0 to 5 , respectively corresponding to normal, early warning, mild diabetic nephropathy, moderate diabetic nephropathy, severe diabetic nephropathy, and end-stage diabetic nephropathy.

[0115] the experimental comparison algorithm

[0116] To verify the effectiveness of the deep learning model A1 proposed in the present application in the six-level prediction task of diabetic nephropathy severity, the current mainstream image classification neural network models are selected for comparison, including:

[0117] ResNet34: ResNet34 contains 34 layers of convolutional layers, which can alleviate the gradient disappearance problem in deep neural networks, and is widely used in medical image analysis and other tasks.

[0118] Inception-v2: Inception-v2 is a multi-path convolutional neural network that extracts multiple scale features by using different scale convolution kernels in the same layer, improving the modeling ability of the model for local and global information of images.

[0119] VisionTransformer (ViT): ViT is an image classification model based on self-attention mechanism, which is good at capturing long-distance dependency information.

[0120] The above models are trained and tested using the same fundus image dataset, with consistent data division ratio (training set: 3045, test set: 762), and unified six-class labels. In order to ensure fairness, the input image size, training round number, and optimizer strategy of all comparison models are kept consistent, and only the backbone feature extraction network structure is replaced. In the test stage, the prediction accuracy (Accuracy), F1-score, and AUC indicators are used to compare the performance of each model.

[0121] The experimental results are shown in Table 1. The deep learning model A1 proposed in the present application outperforms the existing mainstream models in the diabetic nephropathy grading prediction task, and performs better in accuracy, F1-score, and AUC indicators. Compared with ResNet, Inception, and ViT models, the deep learning model A1, with its specially designed module structure for diabetic nephropathy lesion modeling, achieves an accuracy improvement of about 2%, and significantly optimizes F1-score and AUC indicators, verifying its effectiveness and practical value in fine-grained medical image prediction tasks.Figure 3 The different models are used for identifying lesions of diabetic nephropathy fundus images. It can be seen from the visualization results that the deep learning model A1 focuses more accurately on the key areas related to the diabetic nephropathy fundus lesions, and the attention areas are highly consistent with the true labeled lesions, showing superior discrimination ability and interpretability. In contrast, the attention distribution of other models has a certain degree of deviation or redundant attention, and it is difficult to accurately focus on the high-risk areas.

[0122] Table 1: Comparison of experimental results of different models in diabetic nephropathy grading task

[0123]

[0124] The above comparison results fully verify the effectiveness and superiority of the deep learning model A1 proposed in the present application in the diabetic nephropathy severity grading task. Compared with existing models, the present method has stronger feature expression ability and semantic distinction ability, and can more accurately and meticulously reflect the pathological characteristics of diabetic nephropathy at different stages, providing stronger support for clinical early screening and individualized treatment. In addition, Figure 2 The deep learning model A1 of the present application is used for identifying lesions of diabetic nephropathy fundus images of different severity, and it can be seen that as the severity of diabetic nephropathy increases, the fundus lesion area increases.

[0125] Although the embodiments of the present application and the drawings are disclosed for the purpose of illustration, those skilled in the art can understand that various alternatives, changes and modifications are possible without departing from the spirit and scope of the present application and the appended claims, therefore the scope of the present application is not limited to the disclosed content of the embodiments and drawings.

Claims

1. A non-invasive diabetic nephropathy grading and prediction method that supports convenient screening, characterized in that: The following steps are involved: Step 1: Collect and preprocess color fundus images and non-invasive clinical features of diabetic patients to obtain a multimodal dataset and set annotations for various severity levels of diabetic nephropathy; Step 2: Build a deep learning model A1 and train it on the color fundus images in the multimodal dataset of step 1 to obtain the predicted probability distribution of diabetic nephropathy severity based on color fundus images. ; Step 3: Build a deep learning model A2 and train the non-invasive clinical features in the multimodal dataset in step 1 to obtain the predicted probability distribution of diabetic nephropathy severity based on clinical data. ; Step 4: Construct a fusion model. The fusion model fuses the predicted probabilities of diabetic nephropathy obtained in steps 2 and 3 through an adaptive weight allocation algorithm, dynamically adjusts the weights according to the prediction results output by each model, calculates the comprehensive diabetic nephropathy severity score, and outputs the final diabetic nephropathy prediction grading result based on the score.

2. A non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1: Collect sample data from diabetic patients and remove abnormal sample data from diabetic patients; Step 1.2, preprocessing the color fundus image in the sample data; Step 1.3: Preprocess the non-invasive clinical features in the sample data; Step 1.4: Classify the preprocessed sample data into 6 levels of diabetic nephropathy: Level 0: Normal; Level 1: Warning; Level 2: Mild diabetic nephropathy; Level 3: Moderate diabetic nephropathy; Level 4: Severe diabetic nephropathy; Level 5: End-stage diabetic nephropathy; Step 1.5: Construct a multimodal dataset, including color fundus images, non-invasive clinical features, and corresponding classification labels of diabetic patients. Non-invasive clinical features include: age at presentation, gender, duration of diabetes, and history of hypertension.

3. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 1, characterized in that: In step 2, the deep learning model A1 includes a dual-branch Transformer, a multi-scale feature extraction module, a dual-sparse attention module, and a hierarchical feature interaction module. The input ends of the dual-branch Transformer branch 1 and the dual-branch Transformer branch 2 are respectively connected to the multi-scale feature extraction module, the dual-sparse attention module, the Transformer module, and the hierarchical feature interaction module in sequence. The specific working method of the deep learning model A1 is as follows: Step 2.1: Input the color fundus images at different resolutions into the first and second branches of the dual-branch Transformer respectively; Step 2.2: The multi-scale feature extraction module extracts features from the image of the dual-branch Transformer branch 1 and the image of the dual-branch Transformer branch 2 to obtain two feature sequences X1 and X2. Step 2.3: The dual sparse attention module fuses and filters the results before and after the multi-scale feature extraction module to obtain the feature and , and the features and Input to the dual-branch Transformer module to obtain deep feature representation and ; Step 2.4: The hierarchical feature interaction module takes the features output by the dual sparse attention module and and obtained after the Transformer module and Do residual connection to get and , and complete the dual-branch information interaction and the fusion of different levels of features to obtain the final feature information .

4. A non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 3, characterized in that: The multi-scale feature extraction module in the deep learning model A1 includes: an initial convolution module, a multi-scale convolution enhancement module, a size unification module and an attention fusion output module, wherein the initial convolution module, the multi-scale convolution enhancement module, the size unification module and the attention fusion output module are connected in sequence; the initial convolution module is used to extract the initial features, the multi-scale convolution enhancement module extracts the multi-scale structural features based on the initial features, the size unification module unifies the size of the multi-scale structural features, and the attention fusion output module introduces a hybrid attention mechanism to unify the modeling of the multi-scale features. and .

5. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 3, characterized in that: The specific working method of the dual sparse attention module in the deep learning model A1 is as follows: the extracted dual-branch multi-scale features X1 and X2 are respectively subjected to attention analysis with the original branch features Y1 and Y2 to obtain the attention analysis result M. The dynamic threshold screening mechanism and the adaptive Top-K screening mechanism are used to perform sparse screening respectively, and the overlapping verification module is used to verify the results. The fused attention matrix is ​​generated, and the feature channel weights are adjusted accordingly to obtain the output features of the two branches as follows: and .

6. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 1, characterized in that: The working method of the hierarchical feature interaction module of the deep learning model A1 is to combine the image features from different resolution branches and After Transformer feature extraction and , divide out and Global characteristics of 、 and local features 、 ; The global features of different branches obtained by Transformer feature extraction 、 Local features of another branch 、 Splicing to get and ;calculate and 、 and The similarity weight matrix between them is obtained, and the enhanced global features are obtained. and ; Then, and 、 and The information interaction is completed through the cross attention mechanism, and finally the enhanced global features are and Add and fuse to obtain feature information and : ； ； To get and Extract global information separately and , and then the global information and obtained Splicing , and Splicing , and finally and Splice to get the final feature information ; Next, we will fuse the features Input to the classifier, after Operation to obtain the final probability distribution : ； in, is a fully connected layer; Used to output the predicted probability of each category.

7. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 1, characterized in that: The deep learning model A2 in step 3 includes an input layer, a hidden layer, and an output layer, wherein the hidden layer uses the ReLU activation function and normalization to process the data, and the output layer calculates the prediction probability through the fully connected layer. .

8. The non-invasive diabetic nephropathy grading and prediction method supporting convenient screening according to claim 1, characterized in that: The adaptive weight allocation method used in step 4 is: ； in, represents the predicted probability distribution based on fundus images calculated by deep learning model A1; represents the predicted probability distribution based on non-invasive clinical features calculated by the deep learning model A2; is the weight coefficient adaptively calculated based on the input sample features; By fusion probability distribution Perform the maximum index operation to obtain the final predicted category : ； Prediction category The value range is , corresponding to six grading labels: normal, warning, mild diabetic nephropathy, moderate diabetic nephropathy, severe diabetic nephropathy and end-stage diabetic nephropathy.

Images