A Grading Method for Diabetic Retinopathy Based on Multimodal Fundus Imaging

By using multimodal fundus image fusion technology, combining color fundus images and optical coherence tomography images, a lesion relationship map is constructed, which solves the problem of misjudgment and missed judgment in the grading of diabetic retinopathy by single-modal images, improves the accuracy of grading and risk identification ability, and enhances the interpretability and safety of the model.

CN123118591APending Publication Date: 2026-11-17HANGZHOU UNIV OF ELECTRONIC SCI & TECH PINGHU DIGITAL TECH INNOVATION RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611457443.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, single-modal images are prone to misjudgment and omission in the grading of diabetic retinopathy, making it difficult to effectively express the spatial relationship between lesions and the risk of macular edema, and lacking confidence level indications, thus failing to meet clinical safety requirements.

Method used

Multimodal fundus image fusion technology is used to combine color fundus images and optical coherence tomography images. The spatial and semantic relationships between lesions are constructed through a lesion relationship map, cross-modal feature fusion is performed, and the grading results and risk warnings are output.

Benefits of technology

It improves the accuracy of diabetic retinopathy grading and the ability to identify macular edema risk, and enhances the interpretability of the model and the safety of its clinical application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118591A_ABST
    Figure CN123118591A_ABST
Patent Text Reader

Abstract

This invention discloses a method for grading diabetic retinopathy based on multimodal fundus images. The method involves detecting candidate lesions, structural abnormalities, and locating key anatomical regions of the fundus in preprocessed multimodal fundus image data, obtaining a candidate lesion set, structural abnormality regions, and key anatomical regions of the fundus, thus forming a lesion relationship map. Global visual features, structural thickness features, and lesion relationship features are then extracted and fused across modalities to obtain a fused feature representation. Based on this fused feature representation, diabetic retinopathy grading and macular edema risk assessment are performed. This invention constructs a multimodal intelligent analysis framework that fuses color fundus images and optical coherence tomography (OCT) images. By jointly utilizing fundus surface lesion information and retinal layer structure information, it achieves complementary expression of the pathological features of diabetic retinopathy, improving the accuracy of lesion grading and macular edema risk assessment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing, artificial intelligence-assisted diagnosis, fundus disease screening and deep learning technology, and specifically relates to a method for grading diabetic retinopathy based on multimodal fundus images. Background Technology

[0002] Diabetic retinopathy is a common microvascular complication of diabetes and a significant cause of vision impairment. Early symptoms may include microaneurysms, patchy hemorrhages, hard exudates, and cotton wool spots. As the disease progresses, it can be accompanied by neovascularization, macular edema, and changes in the retinal layer structure.

[0003] In current clinical screening, color fundus images can effectively reflect the surface morphology of lesions and the distribution of blood vessels, but they are insufficient in representing deep structural changes such as macular edema. Optical coherence tomography (OCT) images can reflect the layered structure and thickness changes of the retina, but their utilization of global lesion distribution and vascular background information is limited. Therefore, relying solely on a single modality of image for grading can easily lead to misdiagnosis and missed diagnosis.

[0004] Most existing deep learning-based intelligent classification methods for diabetic retinopathy use convolutional neural networks or visual Transformers to classify the entire image directly. Although they have good end-to-end learning capabilities, they usually focus on global appearance features and do not make sufficient use of spatial relationships between lesions, relative positional relationships between lesions and the macula, and co-occurrence patterns of multiple lesions.

[0005] Furthermore, the importance of different lesions in grading varies significantly. For example, exudative lesions located near the macula contribute differently to disease assessment than minor lesions far from the macula; simple global pooling cannot explicitly express these relationships. For screening scenarios, model predictions also need to provide certain confidence levels and verification guidelines to meet clinical safety requirements.

[0006] Therefore, there is an urgent need for an intelligent grading technology that can combine color fundus images with optical coherence tomography images, explicitly model the relationship between lesions, and simultaneously output grading results and risk warnings. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides a method for grading diabetic retinopathy based on multimodal fundus imaging, thereby improving the accuracy of diabetic retinopathy grading, the ability to identify macular edema risk, and clinical interpretability.

[0008] To achieve the above objectives, the present invention provides the following technical solution.

[0009] In a first aspect, embodiments of this application provide a method for grading diabetic retinopathy based on multimodal fundus imaging, comprising the following steps:

[0010] S1. Acquire multimodal fundus image data of the subject and preprocess the multimodal fundus image data.

[0011] The multimodal fundus image data includes paired color fundus images and optical coherence tomography (OCT) images; during the acquisition of the OCT images, a fundus reference image is acquired simultaneously. The fundus reference image is used to characterize the spatial position and range of the scanning area corresponding to the OCT image in the fundus plane, and to establish the spatial correspondence between the color fundus images and the OCT images.

[0012] S2. Based on the preprocessed multimodal fundus image data, perform lesion candidate detection, structural abnormality detection, and localization of key fundus anatomical regions to obtain the lesion candidate set, structural abnormality regions, and key fundus anatomical regions.

[0013] S3. Based on the obtained candidate lesion set, structural abnormality region and key anatomical region of the fundus, construct a lesion relationship graph, determine the node set in the lesion relationship graph and the node features corresponding to each graph node, and establish the connection relationship between nodes according to the spatial position relationship and category relationship between different graph nodes, calculate the corresponding edge weight, and thus form a lesion relationship graph for lesion relationship feature learning by graph neural network.

[0014] Specifically, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the established spatial mapping relationship, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node.

[0015] S4. Extract global visual features based on preprocessed color fundus images, extract structural thickness features based on retinal thickness statistical features, and extract lesion relationship features based on lesion relationship diagram.

[0016] S5. Perform cross-modal fusion on the global visual features, structural thickness features, and lesion relationship features to obtain a fused feature representation.

[0017] S6. Tiered output and risk assessment.

[0018] Diabetic retinopathy grading and macular edema risk assessment based on fusion feature representation.

[0019] S7. Conduct network training and use the trained network to classify diabetic retinopathy and assess the risk of macular edema.

[0020] In one possible implementation, the preprocessing includes image quality assessment, effective imaging region determination, brightness normalization, noise suppression, vascular enhancement, and cross-modal spatial alignment. First, image quality assessment is performed on the acquired color fundus images and optical coherence tomography (OCT) images. This assessment includes sharpness assessment and occlusion detection. Sharpness assessment is used to obtain an image sharpness score; occlusion detection is used to identify eyelashes, eyelids, and reflective areas, and to obtain the corresponding occlusion areas and occlusion ratios. An occlusion degree score is determined based on the occlusion ratio, and then an image quality score is calculated based on the image sharpness score and the occlusion degree score.

[0021] When the image quality score is lower than the preset quality threshold, the corresponding image is determined not to meet the subsequent intelligent grading requirements; when the image quality score is not lower than the preset quality threshold, the corresponding image is subjected to subsequent preprocessing.

[0022] For color fundus images that meet a preset quality threshold, the effective imaging region is determined, and invalid backgrounds and occluded areas are removed during the determination process to obtain a color fundus effective region image. For optical coherence tomography (OCT) images that meet the preset quality threshold, their effective imaging regions are also determined. Subsequently, brightness normalization and noise suppression are performed sequentially on the color fundus effective region image and the OCT image with the determined effective imaging region. Based on this, vascular enhancement is performed on the color fundus effective region image with brightness normalization and noise suppression. Finally, cross-modal spatial alignment is performed on the color fundus image and the OCT image with the above preprocessing to establish a unified spatial coordinate system and a spatial mapping relationship between the two modal images. The preprocessed color fundus image, the preprocessed OCT image, and the spatial mapping relationship are then obtained.

[0023] In one possible implementation, the lesion candidate set is obtained by detecting the preprocessed color fundus image through a lesion detection network, including the lesion category, detection confidence, location coordinates and bounding box information of each candidate lesion;

[0024] The structural abnormality regions are obtained by detecting the preprocessed optical coherence tomography images through a structural abnormality detection network. These regions include cystic edema areas, abnormal reflection areas, and retinal thickening areas. Each structural abnormality region corresponds to a structural abnormality category, detection confidence level, location coordinates, and region range.

[0025] The location of key anatomical regions of the fundus refers to the identification of key anatomical structures in the preprocessed color fundus image and the determination of the location coordinates and range of the corresponding anatomical regions; wherein, the key anatomical regions of the fundus include the macular region, the optic disc region, the main vascular trunk region, and the main vascular junction region.

[0026] A lesion detection network is used to locate and classify lesions in the preprocessed color fundus images, detecting microaneurysms, hemorrhages, hard exudates, and cotton wool spots. The network outputs the lesion category, detection confidence score, and bounding box information for each detected target. Based on the bounding box information, the location coordinates of the detected target and the corresponding candidate lesion region are determined. Detected targets with a detection confidence score not lower than a preset confidence threshold are identified as candidate lesions, and all candidate lesions together form the lesion candidate set.

[0027] A structural anomaly detection network is used to detect structural anomaly regions in the preprocessed optical coherence tomography (OCT) images, identifying cystic edema areas, abnormal reflection areas, and retinal thickening areas. The network outputs the structural anomaly category, detection confidence level, location coordinates, and region extent for each detection result. Detection results with a confidence level not lower than a preset confidence threshold are defined as structural anomaly regions.

[0028] Simultaneously, the preprocessed color fundus images were analyzed using a macular region localization method, an optic disc detection model, and a vessel segmentation model. Specifically, the macular region localization method was used to determine the extent of the macular region; the optic disc detection model was used to determine the coordinates of the optic disc center and the extent of the optic disc region; and the vessel segmentation model was used to obtain a vessel segmentation mask, and based on this mask, to determine the main vessel trunks and major vessel intersections. Using spatial mapping relationships, the structurally abnormal regions were mapped to the color fundus image coordinate system, placing these regions and key anatomical areas of the fundus under a unified spatial reference system, providing anatomical reference for subsequent lesion relationship mapping and spatial relationship modeling.

[0029] In one possible implementation, the specific steps for determining the set of nodes in the lesion relationship graph and the node features corresponding to each node are as follows:

[0030] Each candidate lesion, structural abnormality region, and key anatomical region of the fundus, including the macular region, optic disc region, and major vascular junction region, obtained in step S2, are used as graph nodes in the lesion relationship graph, which together constitute the node set of the lesion relationship graph.

[0031] Among them, candidate lesions correspond to lesion nodes, structural abnormality areas correspond to structural abnormality nodes, and the macula, optic disc, and major vascular intersection areas correspond to anatomical structure nodes; the vascular trunk area obtained in step S2 is used to calculate the spatial relationship between the relevant map nodes and retinal vessels.

[0032] Based on the detection or localization results obtained in step S2 and the corresponding image regions, node features are constructed for different types of graph nodes. Different types of graph nodes are represented using features corresponding to their attributes.

[0033] For each lesion node, node features are constructed based on the lesion category, detection confidence, location coordinates, and bounding box information of the candidate lesions obtained in step S2. Within the candidate lesion region defined by the bounding box, region segmentation is performed based on the differences in color and edge response between the lesion region and surrounding tissue to obtain the lesion region contour. The lesion area, boundary perimeter, and roundness are calculated based on the lesion region contour to form the lesion shape features. Local texture features are extracted based on the image grayscale and color distribution information within the candidate lesion region. Based on the spatial positional relationship between the candidate lesion region and the main blood vessel region obtained in step S2, the spatial distance and overlap relationship between the candidate lesion region and the main blood vessel region are calculated. The adjacent blood vessel information corresponding to the lesion node is determined based on the spatial distance and overlap relationship.

[0034] The node features of the lesion nodes include lesion category, detection confidence level, spatial location information, lesion shape features, local texture features, and information about adjacent blood vessels.

[0035] For structurally anomalous nodes, node features are constructed based on the structural anomaly category, detection confidence, location coordinates, and region range obtained in step S2. The node features include structural anomaly category, detection confidence, spatial location information, and anomaly region range information.

[0036] For anatomical structure nodes, node features are constructed based on the location coordinates and region range of the macula, optic disc region, and major vascular intersection regions obtained in step S2. The node features include anatomical structure type, spatial location information, and anatomical region range information.

[0037] In one possible implementation, the extraction of statistical features of retinal thickness is specifically performed as follows: the retinal layer structure is segmented in the preprocessed optical coherence tomography (OCT) image to obtain the retinal layer boundary, and the retinal thickness is calculated based on the retinal layer boundary; based on the position of each graph node in a unified spatial coordinate system and the established spatial mapping relationship, the corresponding region of each graph node in the OCT image is determined, and the average, maximum, minimum, and standard deviation of the retinal thickness in the corresponding region are statistically analyzed, and the above statistics are combined to constitute the statistical features of retinal thickness.

[0038] In one possible implementation, the corresponding edge weights are calculated to form a lesion relationship graph for lesion relationship feature learning in the graph neural network, as detailed below:

[0039] The degree of association between nodes is determined based on the spatial and categorical relationships between different graph nodes, and the connection relationship between nodes is established based on the degree of association.

[0040] For graph nodes with established connections, the basic weights of the corresponding connecting edges are determined based on the degree of association between the connected graph nodes. When the connection involves lesion nodes or structural abnormality nodes, the basic weights are further adjusted based on the prior severity of the corresponding lesion nodes or structural abnormality nodes to obtain the corresponding edge weights; for the connection between anatomical structure nodes, the corresponding edge weights are determined based on the degree of association between the connected anatomical structure nodes.

[0041] The severity prior is used to characterize the importance of different lesions or structural abnormalities in the grading of diabetic retinopathy. For lesion nodes, a baseline severity is determined based on the lesion category, and this baseline severity is adjusted based on the lesion area, detection confidence level, and the spatial relationship between the lesion and the macular region to obtain the severity prior for that lesion node. For structural abnormality nodes, a baseline severity is determined based on the structural abnormality category, and this baseline severity is adjusted based on the abnormality area range and detection confidence level to obtain the severity prior for that structural abnormality node. The baseline severity corresponding to the lesion category and structural abnormality category is pre-set based on their correlation with the grading of diabetic retinopathy.

[0042] The node set, node features, connection relationships between nodes, and edge weights together constitute a lesion relationship graph.

[0043] In one possible implementation, S4 specifically operates as follows:

[0044] Visual features are extracted from the preprocessed color fundus image to obtain the corresponding visual feature map. Global feature aggregation and feature mapping are then performed on the visual feature map to obtain global visual features.

[0045] Based on the statistical characteristics of retinal thickness in the regions corresponding to each graph node, the local thickness change rate between spatially adjacent regions is calculated. The statistical characteristics of retinal thickness and the local thickness change rate together constitute the structural thickness parameter. The structural thickness parameter is feature-encoded to obtain structural thickness features used to characterize changes in retinal structure in optical coherence tomography images.

[0046] Information propagation and relationship feature aggregation between nodes are performed on the lesion relationship graph. Based on the node features, the connection relationship between nodes and the edge weights, lesion relationship features are extracted.

[0047] In the process of lesion relationship feature extraction, graph attention weights are calculated based on the similarity of node features and edge weights between adjacent graph nodes. These graph attention weights are used to control the contribution of adjacent nodes to the target node during information propagation and feature aggregation. The node features of adjacent graph nodes are then weighted and aggregated according to these graph attention weights to obtain updated graph node features. Finally, these updated graph node features are aggregated to obtain the lesion relationship features.

[0048] In one possible implementation, S5 specifically operates as follows:

[0049] First, feature mapping is performed on the global visual features, structural thickness features, and lesion relationship features respectively, transforming the three types of features into a feature space of a unified dimension. Cross-modal attention calculation is then performed on the mapped three types of features to obtain the interactive feature representations of each modality, including global visual feature representation, structural thickness feature representation, and lesion relationship feature representation.

[0050] Furthermore, based on the modal feature representations after interaction, the modal weights corresponding to each modality are calculated through gating fusion. The global visual feature representation, structural thickness feature representation, and lesion relationship feature representation after interaction are weighted according to the modal weights. The weighted modal feature representations are then concatenated according to feature dimensions, and feature mapping is performed on the concatenated features to obtain the fused feature representation.

[0051] In one possible implementation, S6 specifically operates as follows:

[0052] The predicted probability corresponding to each grading category is calculated based on the fusion feature representation, and the category with the highest predicted probability is determined as the grading result of diabetic retinopathy. The grading result of diabetic retinopathy includes five levels: No Diabetic Retinopathy (No DR), Mild Non-Proliferative Diabetic Retinopathy (Mild NPDR), Moderate Non-Proliferative Diabetic Retinopathy (Moderate NPDR), Severe Non-Proliferative Diabetic Retinopathy (Severe NPDR), and Proliferative Diabetic Retinopathy (PDR).

[0053] Simultaneously, the probability of macular edema risk is calculated based on the fusion feature representation, and the corresponding macular edema risk level is determined based on the probability of macular edema risk.

[0054] Furthermore, the prediction confidence level is determined based on the predicted probability distribution of each grade of diabetic retinopathy and the probability of macular edema risk. The prediction confidence level is used to characterize the degree of certainty of the model's prediction of the grade of diabetic retinopathy and the risk assessment of macular edema. When the prediction confidence level is lower than the preset confidence level threshold, a manual review prompt is generated.

[0055] The final output includes the diabetic retinopathy grading results, the predicted probability for each grading category, the probability of macular edema risk, the macular edema risk level, the prediction confidence level, and a manual review prompt.

[0056] In one possible implementation, S7 operates as follows:

[0057] Multimodal fundus image data with corresponding annotation information was used as training samples to train the lesion detection network, structural abnormality detection network, retinal layer structure segmentation network, and networks used for global visual feature extraction, structural thickness feature encoding, lesion relationship feature extraction, cross-modal fusion, diabetic retinopathy grading, and macular edema risk assessment involved in steps S2 to S6. During training, the corresponding task loss was determined based on the difference between the prediction results and the actual annotations for each training task, and the corresponding network parameters were optimized based on the task loss.

[0058] After training, the trained network is used to process the multimodal fundus image data to be analyzed according to steps S1 to S6, and outputs the diabetic retinopathy grading results, macular edema risk level, corresponding prediction probability, and manual review prompts.

[0059] Secondly, embodiments of this application provide a grading system for diabetic retinopathy based on multimodal fundus imaging, comprising the following modules:

[0060] Data acquisition and preprocessing module: used to acquire multimodal fundus image data of the subject and preprocess the multimodal fundus image data.

[0061] The multimodal fundus image data includes paired color fundus images and optical coherence tomography (OCT) images; during the acquisition of the OCT images, a fundus reference image is acquired simultaneously. The fundus reference image is used to characterize the spatial position and range of the scanning area corresponding to the OCT image in the fundus plane, and to establish the spatial correspondence between the color fundus images and the OCT images.

[0062] Lesion detection and structural region localization module: Based on preprocessed multimodal fundus image data, lesion candidate detection, structural abnormality detection and key fundus anatomical region localization are performed to obtain lesion candidate set, structural abnormality region and key fundus anatomical region.

[0063] Lesion Relationship Map Construction Module: Based on the obtained candidate lesion set, structurally abnormal regions, and key anatomical regions of the fundus, a lesion relationship map is constructed. The set of nodes in the lesion relationship map and the node features corresponding to each node are determined. The connection relationship between nodes is established based on the spatial position and category relationship between different nodes, and the corresponding edge weights are calculated, thereby forming a lesion relationship map for lesion relationship feature learning by graph neural network.

[0064] Specifically, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the established spatial mapping relationship, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node.

[0065] Multimodal feature extraction module: Extracts global visual features based on preprocessed color fundus images, extracts structural thickness features based on retinal thickness statistics, and extracts lesion relationship features based on lesion relationship map.

[0066] Cross-modal fusion module: used to perform cross-modal fusion of the global visual features, structural thickness features and lesion relationship features to obtain fused feature representation.

[0067] Graded Output and Risk Assessment Module: Based on fusion feature representation, this module performs graded classification of diabetic retinopathy and risk assessment of macular edema.

[0068] The model training module uses multimodal fundus image data with corresponding annotation information as training samples to train the networks involved in the lesion detection and structural region localization module, the lesion relationship map construction module, the multimodal feature extraction module, the cross-modal fusion module, and the hierarchical output and risk assessment module.

[0069] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0070] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect.

[0071] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0072] Compared with the prior art, the present invention has the following beneficial effects:

[0073] (1) This invention constructs a multimodal intelligent analysis framework that integrates color fundus images and optical coherence tomography images. By jointly utilizing fundus surface lesion information and retinal layer structure information, it achieves complementary expression of pathological features of diabetic retinopathy, thereby improving the accuracy of lesion grading and macular edema risk assessment.

[0074] (2) The present invention establishes a lesion relationship graph, which integrates lesion nodes, structural abnormality nodes, and anatomical structure nodes corresponding to the macula, optic disc, and major vascular intersection areas into the graph structure. Through a relation feature learning network based on graph attention mechanism, it learns the spatial and semantic relationships between lesions, between structural abnormality regions, and between lesions and key anatomical structures. Compared with traditional methods based on global image classification, it can more fully express the co-occurrence pattern of multiple lesions and spatial association information, and improve the interpretability of the model.

[0075] (3) The present invention employs a cross-modal attention fusion mechanism to deeply fuse color fundus visual features, retinal structural features and lesion relationship features, thereby improving the collaborative modeling ability between different modal information and thus enhancing the recognition performance of complex lesions.

[0076] (4) The present invention adopts a multi-task joint learning strategy, which simultaneously outputs the diabetic retinopathy grading results, macular edema risk level and corresponding prediction probability, and further calculates the prediction credibility based on the prediction probability. When the prediction credibility is lower than the preset threshold, a manual review prompt is generated, which is conducive to improving the safety, reliability and application value of the model in actual clinical auxiliary screening. Attached Figure Description

[0077] Figure 1 The overall flowchart of the intelligent grading method for diabetic retinopathy provided in the embodiments of the present invention is shown.

[0078] Figure 2 This is the experimental result of intelligent grading of diabetic retinopathy using multimodal fundus imaging, as described in an embodiment of the present invention. Detailed Implementation

[0079] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0080] like Figure 1 As shown, this embodiment provides a method for grading diabetic retinopathy based on multimodal fundus imaging, including the following steps:

[0081] S1. Image acquisition and preprocessing.

[0082] The study acquires paired color fundus images (Fundus), optical coherence tomography (OCT) images, and a fundus reference image generated synchronously during OCT acquisition for the same subject. The color fundus images characterize superficial fundus information such as microaneurysms, hemorrhages, hard exudates, cotton wool spots, and retinal vessel distribution. The OCT images characterize deep tissue structures such as retinal layer structure, local thickness changes, and cystic edema. The fundus reference image is a two-dimensional fundus localization image acquired synchronously during the OCT scan localization process, used to characterize the spatial location and extent of the scanned area corresponding to the OCT image in the fundus plane. The fundus reference image is not used for diabetic retinopathy grading prediction but is used to establish the spatial correspondence between the color fundus images and the OCT images.

[0083] First, the acquired color fundus images and OCT images are evaluated for image quality. In this embodiment, the image quality score is determined by both image sharpness score and occlusion score.

[0084] For image sharpness assessment: the image to be evaluated is converted to a grayscale image, and the Laplacian operator is used to calculate the edge response of the image. The variance of the Laplacian response characterizes the image sharpness, and its calculation formula is as follows:

[0085]

[0086] in, This represents the variance of the Laplace response. This represents the grayscale image to be evaluated. This represents the Laplacian operator. A larger Laplacian response variance indicates richer image edge and texture information, and higher image sharpness. Normalizing the Laplacian response variance yields an image sharpness score ranging from 0 to 1. .

[0087] For occlusion assessment: the occlusion area is determined based on the detected eyelashes, eyelids, and reflective areas, and the occlusion ratio is calculated based on the ratio between the occluded area and the effective imaging area.

[0088]

[0089] in, Indicates the occlusion ratio. Indicates the area of ​​the obstructed region. This represents the area of ​​the effective imaging region of the image. An occlusion degree score is calculated based on the occlusion ratio.

[0090]

[0091] in, The rating indicates the degree of occlusion; the greater the occlusion percentage, the lower the corresponding occlusion rating.

[0092] Calculate the image quality score based on the image sharpness score and the occlusion score:

[0093]

[0094] in, Indicates image quality score, and Let represent the weight coefficients corresponding to the image sharpness score and the occlusion score, respectively, and satisfy the following: .

[0095] In this embodiment, α=0.6, β=0.4, and the preset quality threshold is set to 0.6. When the image quality score Q<0.6, the corresponding image is determined not to meet the requirements for subsequent analysis, a re-acquisition prompt is generated, and the subsequent analysis of the corresponding image is terminated; when the image quality score Q≥0.6, subsequent preprocessing is performed on the corresponding image. The above weight coefficients and preset quality threshold can be adjusted according to the training data or validation data, and this embodiment does not limit this.

[0096] For color fundus images that meet preset quality requirements, the effective region image is determined. In this embodiment, a circular visual field mask is used to remove invalid black backgrounds from the color fundus image, and combined with the aforementioned occlusion detection results, eyelashes, eyelids, and reflective areas are removed or masked to obtain the effective region image of the color fundus. For OCT images that meet preset quality requirements, the imaging region containing effective retinal tissue structures is determined, and image edges and invalid regions that do not contain effective retinal structures are removed.

[0097] Subsequently, brightness normalization and noise suppression were performed on the effective area images of the color fundus and the effective imaging area of ​​the OCT. Specifically, the pixel intensity of the image was normalized, and Contrast Limited Adaptive Histogram Equalization (CLAHE) was applied to the color fundus image to enhance local contrast and reduce brightness differences caused by different acquisition devices and lighting conditions. Median filtering or Gaussian filtering was used to suppress random noise in the image.

[0098] Based on this, vascular enhancement processing is performed on the effective area image of the color fundus after brightness normalization and noise suppression. This embodiment adopts a multi-scale vascular enhancement method based on the Hessian matrix, which enhances the retinal vascular response according to the local second-order grayscale change characteristics corresponding to vascular structures at different scales, so as to improve the recognizability of major blood vessels, vascular bifurcation and vascular intersection structures, and provide a foundation for subsequent cross-modal spatial alignment and vascular structure analysis.

[0099] At this point, the effective region image of the color fundus after the determination of the effective imaging region, brightness normalization, noise suppression, and vascular enhancement processing is used as the preprocessed color fundus image.

[0100] Subsequently, the preprocessed color fundus image and the fundus reference image are aligned across modal spaces. Specifically, retinal vascular structures are extracted from the preprocessed color fundus image and the fundus reference image, and several corresponding vascular feature points are determined based on vascular orientation, bifurcation, and confluence characteristics. The vascular feature points are matched, and an abnormal matching point is removed using the Random Sample Consensus (RANSAC) algorithm.

[0101] The affine space transformation relationship between the color fundus image and the fundus reference image is estimated based on the selected matched vascular feature points. For coordinate point p in the color fundus image, its corresponding transformed coordinate p' can be expressed as:

[0102]

[0103] Where A represents the two-dimensional affine transformation matrix, and t represents the translation vector. Using the affine space transformation relationship, the coordinates in the fundus reference image are mapped to the color fundus image coordinate system. Combined with the OCT scan position recorded in the fundus reference image, the position and range of the corresponding scan area in the OCT image within the color fundus image coordinate system are determined.

[0104] This establishes a spatial mapping relationship between the coordinate system of color fundus images and the OCT scanning area, enabling the location of lesions in color fundus images to establish a spatial correspondence with the location of structural abnormalities and retinal thickness information within the corresponding scanning area of ​​the OCT image. This provides a unified spatial reference for subsequent lesion candidate detection, mapping of structural abnormality areas, extraction of retinal thickness information, and construction of lesion relationship maps.

[0105] After completing the above processing, the preprocessed color fundus image, the preprocessed OCT image, and the spatial mapping relationship between the color fundus image and the OCT image are obtained and used as input for step S2.

[0106] S2, lesion candidate detection, structural abnormality detection, and localization of key anatomical regions of the fundus.

[0107] Based on the preprocessed color fundus image, the preprocessed optical coherence tomography image, and the corresponding spatial mapping relationship obtained in step S1, this embodiment further performs lesion candidate detection, structural abnormality detection, and localization of key anatomical regions of the fundus.

[0108] First, the YOLOv8 target detection network is used to detect lesions in the preprocessed color fundus images. Detection targets include microaneurysms, hemorrhages, hard exudates, and cotton wool spots. The YOLOv8 target detection network outputs the lesion category, bounding box information, and detection confidence score for each detected target. For the i-th detected target, the output result is as follows:

[0109]

[0110] in, This represents the center coordinates of the i-th detected target. This indicates the width and height of the bounding box of the detected target. Indicates the type of lesion. This indicates the confidence level of the detection.

[0111] Based on the bounding box information of the detection target, the location coordinates and extent of the corresponding lesion region are determined, and the detection results are filtered according to a preset reliability threshold. When the following conditions are met: At that time, the corresponding detection target is identified as a candidate lesion, among which, This represents the preset confidence threshold for lesion detection. All candidate lesions that meet the criteria together constitute the lesion candidate set.

[0112] Subsequently, the preprocessed optical coherence tomography (OCT) image obtained in step S1 is processed using the U-Net structural anomaly segmentation network to detect structural anomaly regions, identifying cystic edema regions, reflective anomaly regions, and retinal thickening regions. The U-Net structural anomaly segmentation network outputs pixel-level segmentation results and detection confidence scores for each structural anomaly category, where the structural anomaly categories include cystic edema, reflective anomaly, and retinal thickening; the pixel-level segmentation results characterize the pixel distribution of each structural anomaly region in the OCT image. Based on the pixel-level segmentation results, the location coordinates and extent of each structural anomaly region are determined.

[0113] For the detected structural anomaly regions, they are filtered according to the corresponding detection confidence level, when the following conditions are met: When this occurs, the corresponding region is identified as a structural anomaly region, where, This indicates the confidence level of structural anomaly detection. This indicates the preset structural anomaly detection threshold.

[0114] Furthermore, to obtain the key anatomical regions of the fundus required for subsequent lesion relationship mapping, this embodiment performs anatomical structure localization on the preprocessed color fundus image. The Hough circle transform method is used to detect the optic disc region, and the coordinates of the optic disc center and the range of the optic disc region are obtained based on the detection results. The range of the macular region is determined based on the optic disc position constraints and the image features of the macular region. The U-Net vascular segmentation network is used to segment the vascular region of the color fundus image, obtaining a vascular region segmentation mask, and the main vascular trunk region and major vascular junction regions are determined based on vascular connectivity.

[0115] The results of vascular region segmentation are represented as follows:

[0116]

[0117] in, =1 indicates that the corresponding pixel belongs to the blood vessel region. =0 indicates that the corresponding pixel belongs to a non-vascular region.

[0118] Finally, based on the spatial mapping relationship between the color fundus image and the optical coherence tomography image obtained in step S1, the structural abnormality region is transformed to a unified spatial coordinate system, so that the lesion region, structural abnormality region, and key fundus anatomical region are under the same spatial reference system. After the above processing, the lesion candidate set, structural abnormality region, and key fundus anatomical region are obtained, and used as input for constructing the lesion relationship map in step S3.

[0119] S3, Construction of lesion relationship diagram.

[0120] Based on the candidate set of lesions, structurally abnormal regions, and key anatomical regions of the fundus obtained in step S2, this embodiment further constructs a lesion relationship map to describe the spatial and pathological relationships between different lesion regions, structurally abnormal regions, and key anatomical structures of the fundus.

[0121] Specifically, the lesion relationship diagram is represented as follows:

[0122]

[0123] Where V represents the set of graph nodes, W represents the connection relationship between graph nodes, and W represents the set of weights of the corresponding connection edges.

[0124] First, a set of graph nodes in the lesion relationship diagram is constructed based on the candidate lesion set, structural abnormality regions, and key anatomical regions of the fundus obtained in step S2. Each detected candidate lesion is used as a lesion node, the structural abnormality regions detected in the optical coherence tomography (OCT) images are used as structural abnormality nodes, and the macula, optic disc, and major vascular junctions are used as anatomical structure nodes.

[0125] Among them, candidate lesion nodes are used to characterize lesion areas such as microaneurysms, hemorrhages, hard exudates, and cotton wool spots detected in color fundus images; structural abnormality nodes are used to characterize cystic edema areas, abnormal reflection areas, and retinal thickening areas detected in optical coherence tomography images; and anatomical structure nodes are used to characterize key anatomical locations such as the macular region, optic disc region, and major vascular junction areas.

[0126] For the i-th graph node, its node information is represented as follows:

[0127]

[0128] in, This indicates the node type of the i-th graph node. Represents the spatial coordinates of the node. This indicates the range of the region corresponding to the node.

[0129] Furthermore, node features are constructed based on the detection results, spatial locations, and corresponding image regions of each graph node. These node features include node type features, spatial location features, and region features corresponding to the node type.

[0130] For each lesion node, a node feature vector is constructed based on the lesion category, detection confidence, location coordinates, and bounding box information of the candidate lesions obtained in step S2. To obtain the lesion shape features, within the candidate lesion region defined by the bounding box, the region is segmented based on the differences in color and edge response between the lesion region and the surrounding tissue to obtain the lesion region contour. The lesion area, boundary perimeter, and roundness are then calculated based on the lesion region contour to form the lesion shape features. The roundness calculation formula is:

[0131]

[0132] in, This represents the area of ​​the lesion region corresponding to the i-th lesion node. Indicates the perimeter of the lesion area. The lesion's roundness is indicated. The lesion area is used to characterize the lesion's size, while the boundary perimeter and roundness are used to characterize the lesion's morphology.

[0133] Local texture features are extracted based on the image grayscale and color distribution information within the candidate lesion area; the spatial distance and overlap relationship between the candidate lesion area and the main blood vessel area obtained in step S2 are calculated based on the spatial positional relationship between the candidate lesion area and the main blood vessel area, and the adjacent blood vessel information corresponding to the lesion node is determined based on the spatial distance and overlap relationship.

[0134] Therefore, the node feature vector corresponding to the i-th lesion node consists of the lesion category, detection confidence, spatial location information, lesion shape features, local texture features, and information about adjacent blood vessels:

[0135]

[0136] in, This represents the local texture features of the lesion area, extracted based on the grayscale or color distribution information within the lesion area. This indicates information about blood vessels near the lesion area.

[0137] For structurally anomalous nodes, a node feature vector is constructed based on the structural anomaly category, detection confidence, location coordinates, and region range obtained in step S2. The node feature vector includes the structural anomaly category, detection confidence, spatial location information, and anomaly region range information.

[0138] For anatomical structure nodes, node feature vectors are constructed based on the location coordinates and region ranges of the macula, optic disc, and major vascular intersection regions obtained in step S2. The node feature vectors include anatomical structure type, spatial location information, and anatomical region range information.

[0139] Furthermore, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the spatial mapping relationship obtained in step S1, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node.

[0140] Specifically, this embodiment uses the U-Net retinal layer structure segmentation network to segment the retinal layer structure of the preprocessed OCT image, obtaining the segmentation results corresponding to each retinal layer, and determining the boundaries of each retinal layer based on the segmentation results; the retinal thickness is calculated based on the distance between the corresponding retinal layer boundaries. Based on the position of the graph nodes in the unified spatial coordinate system and the spatial mapping relationship established in step S1, the corresponding region of the graph nodes in the OCT image is determined, and the average, maximum, minimum, and standard deviation of the retinal thickness within the corresponding region are statistically analyzed to form statistical features of retinal thickness:

[0141]

[0142] in, These represent the average, maximum, minimum, and standard deviation of retinal thickness within the corresponding region, respectively.

[0143] After obtaining the node features corresponding to each graph node, the connection relationship between the nodes is established based on the spatial position relationship and category relationship between different graph nodes.

[0144] For any two nodes and First, calculate the spatial distance between the center positions of the two nodes:

[0145]

[0146] in,( , )and( , ) represent the spatial coordinates of the two nodes respectively.

[0147] When the spatial distance between two nodes meets the preset association conditions, or when there is a preset category association relationship between the two nodes, a connection edge is established between the two nodes. The preset category association relationship is preset according to the pathological correlation between different lesion categories, structural abnormality categories, and anatomical structures. For example, the category association relationship between a lesion node and its corresponding spatially adjacent structural abnormality node, and between a lesion node and the macular region, can be set according to the corresponding category combination.

[0148]

[0149] Otherwise, no connection is established. For nodes with established connections, the degree of node association is determined based on the spatial distance and category association between the nodes. Specifically, the spatial distance between nodes is converted into a degree of spatial association, where the smaller the distance between nodes, the higher the degree of spatial association. The degree of spatial association between node i and node j is expressed as:

[0150]

[0151] in, Indicates spatial distance. This represents a scale parameter used to adjust the degree to which spatial distance affects the degree of association.

[0152] Furthermore, the category association coefficient is determined based on whether a pre-defined category association relationship exists between the two nodes. When there is a pre-defined category association relationship between the categories corresponding to node i and node j, a higher category association coefficient is assigned; when there is no pre-defined category association relationship, a lower category association coefficient is assigned.

[0153] Combining the spatial correlation degree and the category correlation coefficient, the correlation degree between node i and node j is obtained:

[0154]

[0155] in degree, Indicates the degree of spatial correlation. The weight coefficient represents the degree of spatial correlation, and 0 ≤ ≤1.

[0156] For graph nodes with established connections, the basic weight of the corresponding connecting edge is determined based on the degree of association between node i and node j.

[0157] For connections involving lesion nodes or structurally abnormal nodes, edge weights are determined based on the spatial correlation of nodes and the prior severity of the corresponding nodes. Specifically, the severity prior for lesion nodes is determined by the base severity corresponding to the lesion category, combined with the lesion area, detection confidence level, and the spatial relationship between the lesion and the macular region.

[0158] Specifically, the prior representation of the severity of lesion nodes is as follows:

[0159]

[0160] in, This represents the prior severity of the i-th lesion node. This indicates the baseline severity corresponding to the lesion type. Indicates the area of ​​the lesion. Indicates the confidence level of lesion detection. This indicates the degree of spatial association between the lesion and the macular region. , , and This represents the corresponding weighting coefficient.

[0161] For structurally anomalous nodes, the corresponding severity prior is determined based on the structural anomalous category, the extent of the anomalous region, and the detection confidence level. Specifically, the severity prior for the k-th structurally anomalous node can be expressed as:

[0162]

[0163] in, This represents the prior severity of the k-th structurally anomalous node. This indicates the basic severity corresponding to the structural anomaly category. Indicates the range of the abnormal area. This indicates the confidence level of structural anomaly detection. , and This represents the corresponding weighting coefficient.

[0164] After obtaining the prior severity of the lesion nodes and structural abnormality nodes, the corresponding edge weights are determined according to the node types with established connections.

[0165] For the connections between lesion nodes, between lesion nodes and structurally abnormal nodes, and between structurally abnormal nodes, since both connected nodes have corresponding severity priors, the final edge weights are calculated based on the degree of association between the nodes and the respective severity priors of the two nodes:

[0166]

[0167] in, This represents the edge weight between node i and node j. This indicates the degree of association between nodes, determined by their spatial location and category relationships. and These represent the prior severity levels corresponding to the two nodes. The adjustment coefficient represents the degree of influence of the prior severity on the edge weights.

[0168] For the connection relationship between lesion nodes or structural abnormality nodes and anatomical structure nodes, since anatomical structure nodes do not have prior knowledge of pathological severity, the final edge weights are calculated based on the degree of association between nodes and the prior severity of the lesion nodes or structural abnormality nodes. Let node i be a lesion node or structural abnormality node, and node j be an anatomical structure node, then the corresponding edge weights are expressed as follows:

[0169]

[0170] in, This represents the prior severity of the lesion node or structural abnormality node i, and the meanings of the other parameters are the same as those described above.

[0171] For the connectivity between anatomical structural nodes such as the macula, optic disc, and major vascular junctions, the corresponding edge weights are determined based on the spatial distance between the nodes:

[0172]

[0173] in, This represents the spatial distance between two anatomical structural nodes.

[0174] Through the above processing, a lesion relationship graph containing node set, node features, node connection relationship and edge weight information is obtained, which is used as the input for the graph neural network to learn lesion relationship features in step S4.

[0175] S4. Multimodal feature extraction.

[0176] Based on the preprocessed color fundus image obtained in step S1, the retinal thickness statistical features obtained in step S3, and the lesion relationship map, this embodiment further performs multimodal feature extraction to obtain the global visual features corresponding to the color fundus image, the structural thickness features corresponding to the optical coherence tomography image, and the lesion relationship features corresponding to the lesion relationship map, which are then used as inputs for subsequent cross-modal fusion.

[0177] First, visual features are extracted from the preprocessed color fundus image obtained in step S1. In this embodiment, a ResNet50 convolutional neural network is used as the feature extraction network for the color fundus image. The network extracts low-level texture information and high-level semantic information from the image layer by layer through convolutional layers to obtain the corresponding visual feature map.

[0178] Let the input color fundus image be The visual feature map extracted by the convolutional neural network is represented as:

[0179]

[0180] in, This represents the visual feature map corresponding to a color fundus image.

[0181] Furthermore, global average pooling is performed on the visual feature map to aggregate feature information in the spatial dimension, thereby obtaining global visual features:

[0182]

[0183] in, GAP represents global visual features and global average pooling operation.

[0184] To unify the representation dimensions of features from different modalities, the global visual features are further mapped using a fully connected layer:

[0185]

[0186] in, This represents the final obtained global visual features. and These represent the weight parameters and bias parameters corresponding to the fully connected layer, respectively.

[0187] Subsequently, the retinal thickness statistical features obtained in step S3 are encoded into structural features to obtain the structural thickness features corresponding to the optical coherence tomography image.

[0188] In this embodiment, step S3 has completed the segmentation of the retinal layer structure in the optical coherence tomography image and obtained the statistical characteristics of retinal thickness in the region corresponding to each map node. Based on the statistical characteristics of retinal thickness obtained in step S3, the local thickness change rate between the regions corresponding to spatially adjacent map nodes is further calculated to describe the structural changes between different regions.

[0189] For adjacent regions i and j, the rate of change of their local thickness is expressed as:

[0190]

[0191] in, To prevent tiny positive numbers with a denominator of zero.

[0192] The statistical features of retinal thickness obtained in step S3 and the calculated local thickness change rate together constitute the structural thickness parameter:

[0193]

[0194] Furthermore, in this embodiment, a multi-layer perceptron (MLP) is used to encode the structural thickness parameters to obtain the structural thickness features corresponding to the optical coherence tomography image:

[0195]

[0196] Subsequently, graph relationship features are extracted from the lesion relationship map obtained in step S3.

[0197] This embodiment employs a graph attention-based relational feature learning network to propagate and aggregate node information in the lesion relationship graph. Input information includes the node features, node connectivity relationships, and edge weights obtained in step S3.

[0198] For information propagation between graph nodes, the graph attention weights are determined based on the similarity of features of adjacent nodes and the edge weights calculated in step S3.

[0199] The node feature similarity is calculated using cosine similarity.

[0200]

[0201] in, This represents the node feature similarity between node i and node j. and Let i and j represent the node feature vectors corresponding to node i and node j, respectively.

[0202] Based on the edge weights obtained in step S3, the information propagation weights between nodes are adjusted:

[0203]

[0204] in, This represents the graph attention weight corresponding to node i receiving information from node j. This represents the edge weights obtained in step S3.

[0205] Based on the graph attention weights, the information of adjacent nodes is weighted and aggregated:

[0206]

[0207]

[0208] in, This represents the feature vector of node i after the (l+1)th layer update. Let i represent the set of adjacent nodes of node i. This represents the feature vector of node i in the l-th layer. Let represent the feature transformation matrix corresponding to the l-th layer, and σ represent the nonlinear activation function.

[0209] After multi-layer graph attention propagation, the features of all nodes are aggregated to obtain lesion relationship features:

[0210]

[0211] in, L represents the lesion relationship features, L represents the number of layers in the graph attention propagation, V represents the set of nodes in the lesion relationship graph, and Pool(⋅) represents the node feature aggregation operation.

[0212] Through the above processing, the global visual features corresponding to the color fundus images are obtained respectively. Structural thickness characteristics corresponding to optical coherence tomography images and the lesion relationship features corresponding to the lesion relationship diagram And serve as the input to the cross-modal fusion module in step S5.

[0213] S5, Multimodal Feature Fusion

[0214] Based on the global visual features, structural thickness features, and lesion relationship features obtained in step S4, this embodiment further performs cross-modal feature fusion to achieve information interaction between color fundus images, optical coherence tomography images, and lesion relationship maps, and obtain fused feature representations for grading diabetic retinopathy and assessing macular edema risk.

[0215] Specifically, the three types of features obtained in step S4 are used as inputs to the cross-modal attention mechanism, and information interaction between different modal features is performed through the multi-head cross-attention mechanism. For different modal input features, the attention relationship between different modalities is calculated based on their corresponding query vector, key vector, and value vector.

[0216] For any two modal features, the attention weights are calculated as follows:

[0217]

[0218] in, This represents the cross-modal attention weight of mode m to mode n. This represents the query vector corresponding to mode m. This represents the key vector corresponding to mode n. The feature dimension of the key vector is represented; where m and n represent the modalities corresponding to global visual features, structural thickness features, and lesion relationship features, respectively.

[0219] Based on the attention weights, other modal information is weighted and aggregated:

[0220]

[0221] in, This represents the feature representation of mode m obtained after cross-modal information interaction. This represents the value vector corresponding to mode n.

[0222] Through the aforementioned cross-modal attention calculation, global visual features can be associated with lesion relationship information, structural thickness features can be associated with lesion spatial relationship information, and lesion relationship features can be combined with visual representations in color fundus images and structural change information in OCT images to obtain interactive multimodal feature representations.

[0223] Furthermore, to adaptively fuse modal features based on their contribution levels in different cases, this embodiment employs a gated fusion method to calculate the weights corresponding to each modality. For the m-th modality, its modal weight is expressed as:

[0224]

[0225] in, This represents the modal weight corresponding to the m-th mode. This represents the m-th modal feature representation after cross-modal information interaction. and These represent the weight parameters and bias parameters corresponding to gating fusion, respectively, and the sum of the weights corresponding to the three modes is 1.

[0226] The modal features are weighted according to the aforementioned weighting coefficients:

[0227]

[0228] in, This represents the m-th modal feature after weight adjustment.

[0229] Finally, the weighted global visual features, structural thickness features, and lesion relationship features are concatenated according to feature dimensions, and feature mapping is performed through a fully connected layer to obtain the final fused feature representation:

[0230]

[0231] Where F represents the final fused feature representation, Concat(⋅) represents concatenation according to the feature dimension, and FC(⋅) represents the fully connected feature fusion operation; These represent the global visual features, structural thickness features, and lesion relationship features after weight adjustment, respectively.

[0232] Through the above processing, the obtained fusion feature representation integrates global visual information from color fundus images, structural thickness information from optical coherence tomography images, and lesion relationship information from lesion relationship maps, and serves as the input for step S6 to grade diabetic retinopathy and assess the risk of macular edema.

[0233] S6. Tiered Output and Risk Assessment

[0234] Based on the fusion feature representation obtained in step S5, this embodiment further performs diabetic retinopathy grading and macular edema risk assessment to obtain diabetic retinopathy grading results, predicted probabilities for each grading category, macular edema risk probabilities, macular edema risk levels, and prediction confidence.

[0235] Specifically, the fused feature representation F obtained in step S5 is input into the diabetic retinopathy grading prediction network. In this embodiment, the diabetic retinopathy grading prediction network uses a fully connected classification layer to classify and predict the fused feature representation, obtaining the prediction output corresponding to each grade category of diabetic retinopathy:

[0236]

[0237] in, This indicates the output of the diabetic retinopathy grading prediction. and These represent the weight parameters and bias parameters corresponding to the diabetic retinopathy grading prediction network, respectively.

[0238] Furthermore, the predicted output is normalized using the Softmax function to obtain the predicted probability corresponding to each hierarchical category:

[0239]

[0240] in, This represents the predicted probability corresponding to the k-th diabetic retinopathy grade. This represents the output value corresponding to the kth category in the graded prediction output, where K represents the total number of diabetic retinopathy graded categories. In this embodiment, K=5.

[0241] In this embodiment, the diabetic retinopathy grading categories include: No diabetic retinopathy (No DR), Mild non-proliferative diabetic retinopathy (Mild NPDR), Moderate non-proliferative diabetic retinopathy (Moderate NPDR), Severe non-proliferative diabetic retinopathy (Severe NPDR), and Proliferative diabetic retinopathy (PDR).

[0242] Based on the predicted probabilities corresponding to each level, the category with the highest probability is determined as the final diabetic retinopathy grading result:

[0243]

[0244] in, This indicates the final output of the diabetic retinopathy grading result.

[0245] Furthermore, the fusion feature representation F obtained in step S5 is input into the macular edema risk prediction network to predict the macular edema risk of the examinee.

[0246] In this embodiment, the macular edema risk prediction network uses a fully connected prediction layer to output macular edema risk prediction values:

[0247]

[0248] in, This indicates the output of macular edema risk prediction. and These represent the weight parameters and bias parameters corresponding to the risk prediction network, respectively.

[0249] The prediction results are then transformed using the Sigmoid function:

[0250]

[0251] The probability of macular edema risk is determined based on a preset risk threshold. At that time, the examinee was deemed to have a high risk of macular edema; when At that time, the risk of macular edema in the examinee was determined to be low.

[0252] in, This indicates a preset threshold for assessing the risk of macular edema, which can be adjusted based on training or validation data.

[0253] Furthermore, to improve the reliability of the prediction results, this embodiment evaluates the prediction credibility based on the predicted probability of diabetic retinopathy grade and the predicted probability of macular edema risk.

[0254] For the grading results of diabetic retinopathy, the maximum predictive probability is used as the grading confidence level:

[0255]

[0256] For the risk assessment results of macular edema, the distance between the predicted probability and the classification boundary is used as the confidence level of the risk prediction:

[0257]

[0258] in, The value represents the prediction confidence level corresponding to the macular edema risk assessment result, and its value ranges from [0,1]. The closer the macular edema risk probability is to 0 or 1, the higher the prediction confidence level; the closer the macular edema risk probability is to the classification boundary of 0.5, the lower the prediction confidence level.

[0259] Reliability of Diabetic Retinopathy Grading The confidence level is below the preset grading confidence threshold, or the confidence level of the macular edema risk prediction is below the threshold. When the risk level falls below a preset confidence threshold, a corresponding manual review prompt is generated to encourage doctors to make further judgments based on other examination results, thereby reducing the risk of misdiagnosis in borderline and complex cases.

[0260] Finally, this embodiment outputs the diabetic retinopathy grading results, the predicted probability for each grading category, the probability of macular edema risk, the macular edema risk level, the prediction confidence level, and a manual review prompt.

[0261] S7, Model Training and Application

[0262] Based on the multimodal intelligent grading model for diabetic retinopathy constructed using the above steps, this embodiment further trains the model and uses the trained model to grade diabetic retinopathy and assess the risk of macular edema.

[0263] Specifically, multimodal fundus image data with annotation information are collected as training samples. These training samples include paired color fundus images, optical coherence tomography (OCT) images, and corresponding diabetic retinopathy grading labels and macular edema risk labels. Furthermore, some training samples include lesion region annotation information, structural abnormality region annotation information, and retinal layer structure annotation information, used to assist in training the lesion detection network, structural abnormality detection network, and retinal layer structure segmentation network.

[0264] During training, different networks are trained according to their corresponding training tasks and annotation information. Specifically, a lesion detection network is trained based on lesion region annotation information, enabling it to identify lesion areas such as microaneurysms, hemorrhages, hard exudates, and cotton wool spots in color fundus images. A structural anomaly detection network is trained based on structural anomaly region annotation information, enabling it to identify cystic edema areas, abnormal reflection areas, and retinal thickening areas in optical coherence tomography (OCT) images. Finally, a retinal layer structure segmentation network is trained based on retinal layer structure annotation information, enabling it to obtain retinal layer boundary information and retinal thickness information.

[0265] Furthermore, the trained lesion detection network, structural abnormality detection network, and retinal layer structure segmentation network are connected with the global visual feature extraction network, structural thickness feature encoding network, and lesion relationship feature extraction network described in step S4, the cross-modal feature fusion network described in step S5, and the diabetic retinopathy grading prediction network and macular edema risk prediction network described in step S6 according to the processing flow described in steps S1 to S6 to form a multimodal intelligent grading model for diabetic retinopathy, and the model is jointly optimized and trained.

[0266] During the joint optimization training process, the corresponding task losses are determined based on the differences between the model output results of diabetic retinopathy grading, macular edema risk assessment, lesion detection, structural abnormality detection, and retinal layer structure segmentation and the corresponding real annotations. A joint loss function is constructed based on each task loss, and the corresponding network parameters are optimized through the backpropagation algorithm.

[0267] The joint loss function is expressed as:

[0268]

[0269] in, This represents the total loss during model training; Indicates the loss of diabetic retinopathy grading; This indicates the risk assessment loss related to macular edema. Indicates loss in lesion detection, Indicates structural anomaly detection loss, This indicates a loss of segmentation of the retinal layer structure; , , These represent the weighting coefficients for the corresponding task losses.

[0270] The grading loss for diabetic retinopathy is determined based on the difference between the predicted probability and the actual grading label for each grading category obtained in step S6. This embodiment uses the cross-entropy loss function to calculate the grading loss for diabetic retinopathy, expressed as:

[0271]

[0272] in, This represents the true label corresponding to the k-th diabetic retinopathy classification. This represents the predicted probability corresponding to the k-th diabetic retinopathy grade obtained in step S6, where K represents the total number of diabetic retinopathy grades. In this embodiment, diabetic retinopathy includes five grades: no diabetic retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, and proliferative diabetic retinopathy; therefore, K=5.

[0273] The macular edema risk assessment loss is determined based on the difference between the macular edema risk probability obtained in step S6 and the true risk label. This embodiment uses a binary cross-entropy loss function to calculate the macular edema risk assessment loss, expressed as:

[0274]

[0275] in, This indicates the true risk label for macular edema. This represents the probability of macular edema risk obtained in step S6.

[0276] For lesion detection loss, it is determined based on the difference between the lesion category prediction result and lesion location prediction result output by the lesion detection network and the corresponding lesion area annotation information; for structural abnormality detection loss, it is determined based on the difference between the structural abnormality category and structural abnormality area prediction result output by the structural abnormality detection network and the corresponding structural abnormality area annotation information; for retinal layer structure segmentation loss, it is determined based on the difference between the retinal layer structure segmentation result output by the retinal layer structure segmentation network and the corresponding retinal layer structure annotation information.

[0277] Furthermore, the lesion detection loss, structural abnormality detection loss, and retinal layer structure segmentation loss are determined using classification loss, localization loss, and segmentation loss, respectively, based on the output format of the corresponding networks. By setting weight coefficients corresponding to the losses of each task, the contribution of different training tasks to the model parameter optimization process is adjusted.

[0278] During training, data augmentation methods are used to improve the model's adaptability to samples with different acquisition devices, image qualities, and lesion severity. These data augmentation methods include random brightness adjustment, random rotation, scale variation, and noise perturbation.

[0279] Through the above-mentioned joint optimization training, the multimodal intelligent grading model for diabetic retinopathy can comprehensively learn global visual information and lesion manifestation information in color fundus images, retinal structural change information in optical coherence tomography images, and spatial and pathological association information between lesion nodes, structural abnormality nodes and anatomical structure nodes in the lesion relationship diagram, thereby realizing the grading of diabetic retinopathy and the risk assessment of macular edema.

[0280] Once the model training is complete, the color fundus images and optical coherence tomography (OCT) images of the subjects to be analyzed are input into the trained multimodal intelligent grading model for diabetic retinopathy. Steps S1 to S6 are performed sequentially for image preprocessing, lesion candidate detection, structural abnormality detection and localization of key anatomical regions of the fundus, lesion relationship map construction, multimodal feature extraction, cross-modal feature fusion, and grading output and risk assessment. Finally, the model outputs the grading results for diabetic retinopathy, the predicted probability for each grading category, the probability of macular edema risk, the risk level of macular edema, and prompts for manual review.

[0281] This application also provides a diabetic retinopathy grading system based on multimodal fundus imaging, including the following modules:

[0282] Data acquisition and preprocessing module: used to acquire multimodal fundus image data of the subject and preprocess the multimodal fundus image data.

[0283] The multimodal fundus image data includes paired color fundus images and optical coherence tomography (OCT) images; during the acquisition of the OCT images, a fundus reference image is acquired simultaneously. The fundus reference image is used to characterize the spatial position and range of the scanning area corresponding to the OCT image in the fundus plane, and to establish the spatial correspondence between the color fundus images and the OCT images.

[0284] Lesion detection and structural region localization module: Based on preprocessed multimodal fundus image data, lesion candidate detection, structural abnormality detection and key fundus anatomical region localization are performed to obtain lesion candidate set, structural abnormality region and key fundus anatomical region.

[0285] Lesion Relationship Map Construction Module: Based on the obtained candidate lesion set, structurally abnormal regions, and key anatomical regions of the fundus, a lesion relationship map is constructed. The set of nodes in the lesion relationship map and the node features corresponding to each node are determined. The connection relationship between nodes is established based on the spatial position and category relationship between different nodes, and the corresponding edge weights are calculated, thereby forming a lesion relationship map for lesion relationship feature learning by graph neural network.

[0286] Specifically, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the established spatial mapping relationship, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node.

[0287] Multimodal feature extraction module: Extracts global visual features based on preprocessed color fundus images, extracts structural thickness features based on retinal thickness statistics, and extracts lesion relationship features based on lesion relationship map.

[0288] Cross-modal fusion module: used to perform cross-modal fusion of the global visual features, structural thickness features and lesion relationship features to obtain fused feature representation.

[0289] Graded Output and Risk Assessment Module: Based on fusion feature representation, this module performs graded classification of diabetic retinopathy and risk assessment of macular edema.

[0290] The model training module uses multimodal fundus image data with corresponding annotation information as training samples to train the networks involved in the lesion detection and structural region localization module, the lesion relationship map construction module, the multimodal feature extraction module, the cross-modal fusion module, and the hierarchical output and risk assessment module.

[0291] The intelligent grading system for diabetic retinopathy provided by this invention can be deployed on hospital ophthalmology diagnosis and treatment platforms, physical examination and screening platforms, and primary-level eye disease screening terminals. It can also be combined with cloud computing platforms to realize multi-center data analysis, providing technical support for large-scale screening, auxiliary diagnosis and risk management of diabetic retinopathy.

[0292] To verify the effectiveness of the multimodal intelligent grading model for diabetic retinopathy described in this embodiment, the trained model was applied to test samples for grading diabetic retinopathy, assessing macular edema risk, and identifying structural anomalies in optical coherence tomography images. The experimental results are as follows: Figure 2As shown.

[0293] Figure 2 (a) shows the model's main evaluation metrics across different tasks. Specifically, the accuracy for the five-category classification of diabetic retinopathy was 0.472, with a quadratic weighted kappa (QWK) coefficient of 0.437; the area under the receiver operating characteristic (AUC) for identifying diabetic retinopathy requiring referral was 0.758; and the AUC for macular edema risk assessment was 0.772. Furthermore, when identifying structural abnormalities in optical coherence tomography (OCT) images, the classification accuracy was 0.811, with an AUC of 0.965.

[0294] Figure 2 Figure (b) shows the ROC curves for macular edema risk assessment and identification of cases requiring referral for diabetic retinopathy. The AUC for macular edema risk assessment was 0.772, and the AUC for identifying cases requiring referral for diabetic retinopathy was 0.758, indicating that the model can distinguish between macular edema risk and cases of diabetic retinopathy with higher lesion severity based on multimodal features.

[0295] Figure 2 (c) shows the confusion matrix corresponding to the five-level classification results of diabetic retinopathy. The five-level classification includes no diabetic retinopathy (No DR), mild non-proliferative diabetic retinopathy (Mild NPDR), moderate non-proliferative diabetic retinopathy (Moderate NPDR), severe non-proliferative diabetic retinopathy (Severe NPDR), and proliferative diabetic retinopathy (PDR). Test results show that the model can classify diabetic retinopathy at different degrees, with relatively good recognition performance for samples without diabetic retinopathy, while there is some confusion between adjacent lesion levels.

[0296] Figure 2 Figure (d) shows the confusion matrix corresponding to the structural anomaly identification results of optical coherence tomography (OCT) images. The test categories shown include four types: choroidal neovascularization (CNV), diabetic macular edema (DME), drusen, and normal. The classification accuracy for this task reached 0.811, and the AUC reached 0.965, indicating that the structural information contained in OCT images can provide effective discriminative information for the identification of retinal structural anomalies.

[0297] Based on the above experimental results, the method described in this embodiment can utilize visual and lesion information in color fundus images, retinal structural information in optical coherence tomography images, and spatial correlation information in lesion relationship diagrams to achieve diabetic retinopathy grading and macular edema risk assessment, and can output corresponding predicted probabilities, providing a reference for subsequent risk judgment and manual review.

[0298] For those skilled in the art, without departing from the concept of the technical solution of this invention, adjustments or replacements can be made to the lesion category setting method, graph node construction method, feature extraction network structure, cross-modal fusion method, and hierarchical prediction method according to actual application needs. All such equivalent transformations should be included within the protection scope of this invention.

Claims

1. A method for grading diabetic retinopathy based on multimodal fundus imaging, characterized in that, Includes the following steps: S1. Acquire multimodal fundus image data of the subject and preprocess the multimodal fundus image data; The multimodal fundus image data includes paired color fundus images and optical coherence tomography (OCT) images; during the acquisition of the OCT images, a fundus reference image is acquired simultaneously. The fundus reference image is used to characterize the spatial position and range of the scanning area corresponding to the OCT image in the fundus plane, and to establish the spatial correspondence between the color fundus images and the OCT images. S2. Based on the preprocessed multimodal fundus image data, perform lesion candidate detection, structural abnormality detection, and localization of key fundus anatomical regions to obtain the lesion candidate set, structural abnormality regions, and key fundus anatomical regions. S3. Based on the obtained candidate lesion set, structural abnormality area and key anatomical area of ​​fundus, construct a lesion relationship graph, determine the node set in the lesion relationship graph and the node features corresponding to each graph node, and establish the connection relationship between nodes according to the spatial position relationship and category relationship between different graph nodes, calculate the corresponding edge weight, and thus form a lesion relationship graph for graph neural network to learn lesion relationship features. Specifically, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the established spatial mapping relationship, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node. S4. Extract global visual features based on preprocessed color fundus images, extract structural thickness features based on retinal thickness statistical features, and extract lesion relationship features based on lesion relationship diagram; S5. Perform cross-modal fusion on the global visual features, structural thickness features, and lesion relationship features to obtain a fused feature representation; S6. Tiered output and risk assessment; Diabetic retinopathy grading and macular edema risk assessment based on fusion feature representation; S7. Conduct network training and use the trained network to classify diabetic retinopathy and assess the risk of macular edema.

2. The method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The preprocessing includes image quality assessment, effective imaging region determination, brightness normalization, noise suppression, vessel enhancement, and cross-modal spatial alignment. First, image quality assessment is performed on the acquired color fundus images and optical coherence tomography (OCT) images. This assessment includes sharpness assessment and occlusion detection. Sharpness assessment is used to obtain an image sharpness score; occlusion detection is used to identify eyelashes, eyelids, and reflective areas, and to obtain the corresponding occlusion areas and occlusion ratios. An occlusion degree score is determined based on the occlusion ratio, and then an image quality score is calculated based on the image sharpness score and the occlusion degree score. When the image quality score is lower than a preset quality threshold, the corresponding image is determined not to meet the subsequent intelligent grading requirements; when the image quality score is not lower than the preset quality threshold, the corresponding image is subjected to subsequent preprocessing. For a color fundus image that meets a preset quality threshold, the effective imaging region is determined, and invalid background and occluded areas are removed during the determination process to obtain a color fundus effective region image. For an optical coherence tomography (OCT) image that meets the preset quality threshold, its effective imaging region is determined. Subsequently, brightness normalization and noise suppression are performed sequentially on the color fundus effective region image and the OCT image with the effective imaging region determined. Based on this, vascular enhancement is performed on the color fundus effective region image with brightness normalization and noise suppression. Finally, cross-modal spatial alignment is performed on the color fundus image and the OCT image with the above preprocessing to establish a unified spatial coordinate system and a spatial mapping relationship between the two modal images. The preprocessed color fundus image, the preprocessed OCT image, and the spatial mapping relationship are obtained.

3. The method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 2, characterized in that, The lesion candidate set is obtained by detecting the preprocessed color fundus image through a lesion detection network, including the lesion category, detection confidence, location coordinates and bounding box information of each candidate lesion; The structural abnormality regions are obtained by detecting the preprocessed optical coherence tomography images through a structural abnormality detection network. These regions include cystic edema areas, abnormal reflection areas, and retinal thickening areas. Each structural abnormality region corresponds to a structural abnormality category, detection confidence level, location coordinates, and region range. The location of key anatomical regions of the fundus refers to the identification of key anatomical structures in the preprocessed color fundus image and the determination of the location coordinates and range of the corresponding anatomical regions; wherein, the key anatomical regions of the fundus include the macular region, the optic disc region, the main vascular trunk region, and the main vascular junction region; The preprocessed color fundus images are used to locate and classify lesions using a lesion detection network, detecting microaneurysms, hemorrhages, hard exudates, and cotton wool spots. The network outputs the lesion category, detection confidence level, and bounding box information for each detected target. Based on the bounding box information, the location coordinates of the detected target and the corresponding candidate lesion region are determined. Detected targets with a detection confidence level not lower than a preset confidence threshold are identified as candidate lesions, and all candidate lesions together form the lesion candidate set. The preprocessed optical coherence tomography (OCT) images are used to detect structural aberration regions, identifying cystic edema areas, abnormal reflection areas, and retinal thickening areas. The structural aberration category, detection confidence level, location coordinates, and region range of each detection result are output. Detection results with a detection confidence level not lower than a preset confidence threshold are identified as structural aberration regions. Simultaneously, the preprocessed color fundus image is analyzed using a macular region localization method, an optic disc detection model, and a vascular segmentation model. The macular region localization method is used to determine the range of the macular region; the optic disc detection model is used to determine the coordinates of the optic disc center and the range of the optic disc region; the vascular segmentation model is used to obtain a vascular region segmentation mask, and based on the vascular region segmentation mask, to determine the main vascular trunk region and the main vascular intersection region; using spatial mapping relationships, the structurally abnormal regions are mapped to the color fundus image coordinate system, so that the structurally abnormal regions and the key anatomical regions of the fundus are under a unified spatial reference system.

4. The method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 3, characterized in that, The specific steps for determining the set of nodes in the lesion relationship diagram and the node characteristics corresponding to each node are as follows: Each candidate lesion, structural abnormality region, and key anatomical region of the fundus, such as the macular region, optic disc region, and major vascular junction region, obtained in step S2 are used as graph nodes in the lesion relationship diagram, which together constitute the node set of the lesion relationship diagram. Among them, candidate lesions correspond to lesion nodes, structural abnormality regions correspond to structural abnormality nodes, and the macula region, optic disc region, and major vascular intersection regions correspond to anatomical structure nodes; the vascular trunk region obtained in step S2 is used to calculate the spatial relationship between the relevant map nodes and retinal vessels. Based on the detection or localization results obtained in step S2 and the corresponding image regions, node features are constructed for different types of graph nodes; different types of graph nodes are represented by features corresponding to their attributes. For each lesion node, node features are constructed based on the lesion category, detection confidence, location coordinates, and bounding box information of the candidate lesions obtained in step S2. Within the candidate lesion area defined by the bounding box, region segmentation is performed based on the differences in color and edge response between the lesion area and surrounding tissue to obtain the lesion area contour. The lesion area, boundary perimeter, and roundness are calculated based on the lesion area contour to form lesion shape features. Local texture features are extracted based on the image grayscale and color distribution information within the candidate lesion area. Based on the spatial positional relationship between the candidate lesion area and the main blood vessel area obtained in step S2, the spatial distance and overlap relationship between the candidate lesion area and the main blood vessel area are calculated. The adjacent blood vessel information corresponding to the lesion node is determined based on the spatial distance and overlap relationship. The node features of the lesion nodes include lesion category, detection confidence level, spatial location information, lesion shape features, local texture features, and adjacent blood vessel information; For structurally anomalous nodes, node features are constructed based on the structural anomalous category, detection confidence, location coordinates, and region range obtained in step S2. The node features include structural anomalous category, detection confidence, spatial location information, and anomalous region range information. For anatomical structure nodes, node features are constructed based on the location coordinates and region range of the macula region, optic disc region, and major vascular intersection region obtained in step S2. The node features include anatomical structure type, spatial location information, and anatomical region range information. The extraction of statistical features of retinal thickness is carried out as follows: the retinal layer structure is segmented in the preprocessed optical coherence tomography (OCT) image to obtain the retinal layer boundary, and the retinal thickness is calculated based on the retinal layer boundary; according to the position of each graph node in a unified spatial coordinate system and the established spatial mapping relationship, the corresponding region of each graph node in the OCT image is determined, and the average, maximum, minimum and standard deviation of the retinal thickness in the corresponding region are statistically analyzed, and the above statistics are combined to form the statistical features of retinal thickness.

5. The method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The corresponding edge weights are calculated to form a lesion relationship graph for lesion relationship feature learning in graph neural networks. The specific operation is as follows: The degree of association between nodes is determined based on the spatial and categorical relationships between different graph nodes, and the connection relationships between nodes are established based on the degree of association. For graph nodes with established connections, the basic weight of the corresponding connecting edge is determined based on the degree of association between the connected graph nodes; when the connection involves lesion nodes or structural abnormality nodes, the basic weight is further adjusted based on the prior severity of the corresponding lesion nodes or structural abnormality nodes to obtain the corresponding edge weight; for the connection between anatomical structure nodes, the corresponding edge weight is determined based on the degree of association between the connected anatomical structure nodes. The severity prior is used to characterize the importance of different lesions or structural abnormalities in the grading of diabetic retinopathy. For each lesion node, the baseline severity is determined based on the lesion type, and the baseline severity is adjusted by combining the lesion area, detection confidence, and spatial relationship between the lesion and the macular region to obtain the severity prior corresponding to the lesion node. For structurally anomalous nodes, the corresponding baseline severity is determined based on the structural anomalous category, and the baseline severity is adjusted in conjunction with the anomalous region range and detection confidence to obtain the severity prior corresponding to the structurally anomalous node; the baseline severity corresponding to the lesion category and the structural anomalous category is preset according to their correlation with the diabetic retinopathy grading. The node set, node features, connection relationships between nodes, and edge weights together constitute a lesion relationship graph.

6. The method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The specific operation of S4 is as follows: Visual features are extracted from the preprocessed color fundus image to obtain the visual feature map corresponding to the color fundus image. Global feature aggregation and feature mapping are then performed on the visual feature map to obtain global visual features. Based on the statistical characteristics of retinal thickness in the regions corresponding to each graph node, the local thickness change rate between spatially adjacent regions is calculated. The statistical characteristics of retinal thickness and the local thickness change rate together constitute the structural thickness parameter. The structural thickness parameter is feature-encoded to obtain structural thickness features used to characterize changes in retinal structure in optical coherence tomography images. The lesion relationship graph is subjected to information propagation and relationship feature aggregation between graph nodes. Based on the node features, the connection relationship between nodes and the edge weights, the lesion relationship features are extracted. In the process of extracting lesion relationship features, graph attention weights are calculated based on the similarity of node features between adjacent graph nodes and edge weights; the node features of adjacent graph nodes are weighted and aggregated according to the graph attention weights to obtain updated graph node features, and the updated graph node features are aggregated to obtain lesion relationship features.

7. A method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The specific operation of S5 is as follows: First, feature mapping is performed on the global visual features, structural thickness features, and lesion relationship features respectively, transforming the three types of features into a feature space of a unified dimension; cross-modal attention calculation is performed on the three types of mapped features to obtain the feature representations of each modality after interaction, including global visual feature representation, structural thickness feature representation, and lesion relationship feature representation; Furthermore, based on the modal feature representations after interaction, the modal weights corresponding to each modality are calculated through gating fusion. The global visual feature representation, structural thickness feature representation, and lesion relationship feature representation after interaction are weighted according to the modal weights. The weighted modal feature representations are then concatenated according to feature dimensions, and feature mapping is performed on the concatenated features to obtain the fused feature representation.

8. A method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The specific operation of S6 is as follows: The predicted probability corresponding to each grading category is calculated based on the fusion feature representation, and the category with the highest predicted probability is determined as the grading result of diabetic retinopathy. The grading result of diabetic retinopathy includes five levels: no diabetic retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, and proliferative diabetic retinopathy. Simultaneously, the probability of macular edema risk is calculated based on the fusion feature representation, and the corresponding macular edema risk level is determined based on the probability of macular edema risk. Furthermore, the prediction confidence level is determined based on the predicted probability distribution of each grade of diabetic retinopathy and the probability of macular edema risk; when the prediction confidence level is lower than the preset confidence level threshold, a manual review prompt is generated. The final output includes the diabetic retinopathy grading results, the predicted probability for each grading category, the probability of macular edema risk, the macular edema risk level, the prediction confidence level, and a manual review prompt.

9. A method for grading diabetic retinopathy based on multimodal fundus imaging according to claim 1, characterized in that, The specific operation of S7 is as follows: Multimodal fundus image data with corresponding annotation information were used as training samples to train the lesion detection network, structural abnormality detection network, retinal layer structure segmentation network, and networks used for global visual feature extraction, structural thickness feature encoding, lesion relationship feature extraction, cross-modal fusion, diabetic retinopathy grading, and macular edema risk assessment involved in steps S2 to S6. During training, the corresponding task loss is determined based on the difference between the prediction results and the true annotations for each training task, and the corresponding network parameters are optimized based on the task loss. After training, the trained network is used to process the multimodal fundus image data to be analyzed according to steps S1 to S6, and outputs the diabetic retinopathy grading results, macular edema risk level, corresponding prediction probability, and manual review prompts.

10. A grading system for diabetic retinopathy based on multimodal fundus imaging, characterized in that, Includes the following modules: Data acquisition and preprocessing module: used to acquire multimodal fundus image data of the subject and preprocess the multimodal fundus image data; The multimodal fundus image data includes paired color fundus images and optical coherence tomography (OCT) images; during the acquisition of the OCT images, a fundus reference image is acquired simultaneously. The fundus reference image is used to characterize the spatial position and range of the scanning area corresponding to the OCT image in the fundus plane, and to establish the spatial correspondence between the color fundus images and the OCT images. Lesion detection and structural region localization module: Based on preprocessed multimodal fundus image data, lesion candidate detection, structural abnormality detection and key fundus anatomical region localization are performed to obtain lesion candidate set, structural abnormality region and key fundus anatomical region; Lesion Relationship Map Construction Module: Based on the obtained lesion candidate set, structural abnormality areas and key anatomical areas of the fundus, a lesion relationship map is constructed. The set of nodes in the lesion relationship map and the node features corresponding to each node are determined. The connection relationship between nodes is established according to the spatial position and category relationship between different nodes, and the corresponding edge weights are calculated, thereby forming a lesion relationship map for lesion relationship feature learning by graph neural network. Specifically, for graph nodes whose spatial location falls within the coverage area of ​​the OCT scan, the corresponding region of the graph node in the OCT image is determined according to the established spatial mapping relationship, and the retinal thickness statistical features of the corresponding region are extracted and supplemented into the node features of the corresponding graph node. Multimodal feature extraction module: Extracts global visual features based on preprocessed color fundus images, extracts structural thickness features based on retinal thickness statistics, and extracts lesion relationship features based on lesion relationship map; Cross-modal fusion module: used to perform cross-modal fusion of the global visual features, structural thickness features and lesion relationship features to obtain fused feature representation; The graded output and risk assessment module: performs grading of diabetic retinopathy and risk assessment of macular edema based on fusion feature representation; The model training module uses multimodal fundus image data with corresponding annotation information as training samples to train the networks involved in the lesion detection and structural region localization module, the lesion relationship map construction module, the multimodal feature extraction module, the cross-modal fusion module, and the hierarchical output and risk assessment module.