Diabetic retinopathy ultra-widefield fundus image grading method

Abstract

The present disclosure relates to the technical field of medical image processing, and provides a diabetic retinopathy ultra-wide-angle fundus image grading method. A double-branch feature enhancement and fusion network based on complementary global features and local features is constructed as a diabetic retinopathy image automatic grading network. The diabetic retinopathy image automatic grading network is trained using an ultra-wide-angle fundus image dataset, and a trained diabetic retinopathy image automatic grading network is obtained. The grading effect of the trained diabetic retinopathy image automatic grading network is tested using a test set. The trained diabetic retinopathy image automatic grading network is comprehensively tested and evaluated using multi-dimensional evaluation indexes for automatic grading and recognition effects when applied to diabetic retinopathy grading of ultra-wide-angle fundus images. The present disclosure solves the problem of large differences between the large field of view of the ultra-wide-angle fundus image and the small lesion scale, and the difficulty in balancing global and local features, thereby improving the accuracy and robustness of diabetic retinopathy image automatic grading.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, and in particular to a method for grading ultra-wide-angle fundus images of diabetic retinopathy. Background Technology

[0002] Intelligent analysis and automatic grading of medical images is a crucial area of ​​smart healthcare. In the grading of diabetic retinopathy (DR), accurately capturing and identifying minute lesions on the retina (such as microaneurysms, hemorrhages, and hard exudates) is essential. Traditional fundus examinations primarily rely on standard color fundus photography (CFP), which typically only covers a 45° area in the central retina. However, approximately 30%-40% of DR features are mainly distributed in the peripheral retinal region. A single posterior pole image often fails to provide comprehensive pathological details, easily leading to missed diagnoses or underestimation of the severity of the condition. In recent years, the emergence of ultra-wide-angle fundus imaging (UWF) technology has overcome this limitation. This technology can cover approximately 200° of the retina in a single image. The imaging principle of the retinal image is based on laser scanning ophthalmoscopy, using red and green lasers for imaging, allowing separate examination of the choroid and retina, facilitating their differentiation. The core of ultra-wide-angle fundus imaging technology lies in using red and green dual-wavelength lasers and an ultra-wide-angle optical system to achieve fast, non-mydriatic, and high-resolution fundus imaging. However, under ultra-wide field of view, there is a huge scale difference between high-resolution images and tiny lesions. In addition, there are complex background interferences such as severe geometric distortion at the image edges, eyelash artifacts, and uneven illumination. Single-view feature extraction networks usually cannot take into account both global distribution and local details.

[0003] Ultra-wide-angle fundus images are complex and multi-scale, but existing classification methods based on single architectures or traditional strategies lack comprehensive consideration of global context and fine local features, thus affecting classification accuracy. For example, mainstream deep learning methods based on standard fundus color images using convolutional neural networks (CNNs) such as Inception and ResNet perform well in the central visual field, but their performance drops sharply when faced with ultra-wide-angle images due to a lack of peripheral feature perception. To address the scarcity of ultra-wide-angle data, some existing technologies have proposed domain-adaptive transfer learning frameworks that use prior knowledge from standard fundus color images to assist in ultra-wide-angle image analysis. However, while reducing inter-domain differences, these frameworks often fail to fully preserve semantic information closely related to minute lesions, resulting in insufficient classification stability in the target domain. In solutions directly targeting ultra-wide-angle images, due to the excessively high image resolution, some existing technologies employ direct downsampling strategies to input convolutional neural networks, inevitably leading to severe loss of features of subtle early lesions such as microaneurysms. Existing technologies have also proposed a graph-based multi-level feature fusion network in an attempt to model global long-distance dependencies. However, its complex graph node aggregation process can easily lead to excessive smoothing of subtle textures in low-contrast edge regions, affecting the grading precision.

[0004] With the evolution of deep learning architectures, dual-branch feature extraction and fusion methods combining the advantages of Transformer and CNN have gradually become a research hotspot in medical image analysis. By combining the advantages of Transformer in capturing global spatial context and long-range dependencies with the strength of CNN in extracting local micro-focal textures, richer pathological feature information can be captured. However, how to dynamically integrate these two heterogeneous features in a large field of view and suppress complex artifact noise remains a pressing technical challenge. In addition, existing automatic grading methods for diabetic retinopathy images based on ultra-wide-angle fundus images face three major challenges when moving towards practical clinical applications: First, the issue of accuracy, which requires the grading method to accurately identify microaneurysms and hemorrhages at extremely small scales within a 200° ultra-wide field of view, while overcoming severe stretching distortion in the peripheral areas of the image and the semantic gap when fusing heterogeneous features from CNN and Transformer; second, the issue of robustness, which requires the grading method to effectively resist interference from complex artifacts such as eyelash occlusion, eyelid shadows, and uneven illumination, avoiding misclassifying background noise as lesions; and finally, the issue of universality and sensitivity, as real clinical data often exhibits severe class imbalance, requiring the grading method to have the ability to adaptively adjust the attention of global background and local lesions to improve the sensitivity to early or borderline mild diabetic retinopathy features. Summary of the Invention

[0005] This disclosure aims to address at least one of the problems existing in the prior art by providing a method for grading diabetic retinopathy using ultra-wide-angle fundus images. This method combines feature extraction and dynamic fusion of global spatial context and local minute pathological details within an ultra-wide field of view, suppressing complex artifact interference and adaptively balancing data categories, thereby improving the accuracy and robustness of automatic grading of diabetic retinopathy images. By extracting global and local features through parallel dual-branch extraction and introducing bidirectional attention interaction and feature modulation mechanisms, this method fully utilizes the advantages of ultra-wide-angle images and overcomes their inherent optical and artifact defects, significantly improving the grading effect of diabetic retinopathy. It has extremely high application value in fields such as smart ophthalmology, large-scale diabetic retinopathy screening, and clinical decision support.

[0006] This disclosure provides a method for grading ultra-wide-angle fundus images of diabetic retinopathy, including: A dataset of ultra-wide-angle fundus images is obtained, the dataset is preprocessed, and randomly divided into a training set, a validation set, and a test set according to a preset ratio. A dual-branch feature enhancement and fusion network based on complementary global and local features is constructed as an automatic grading network for diabetic retinopathy lesion images. The automatic grading network for diabetic retinopathy lesion images includes a dual-stream hybrid feature extraction backbone network and a cascaded classification module. The dual-stream hybrid feature extraction backbone network introduces a bidirectional attention interaction module and a feature modulation module. The cascaded classification module includes the feature modulation module and an adaptive category attention module. The ultra-wide-angle fundus images and their corresponding grading labels in the training set are fed into the automatic grading network for diabetic retinopathy images for training. The loss function of the automatic grading network for diabetic retinopathy images is calculated, and the model weights of the automatic grading network for diabetic retinopathy images are iteratively updated by combining a dynamic learning rate strategy until a preset number of iterations is reached. After each iteration, the performance of the automatic grading network for diabetic retinopathy images is evaluated using the validation set, and the model weights that perform best on the validation set are saved to obtain the trained automatic grading network for diabetic retinopathy images. The grading performance of the trained automatic grading network for diabetic retinopathy images was tested using the test set. The trained automatic grading network for diabetic retinopathy images was applied to grade diabetic retinopathy in ultra-wide-angle fundus images. Multi-dimensional evaluation indicators were used to comprehensively test and evaluate the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images.

[0007] Optionally, the preprocessing of the ultra-wide-angle fundus image dataset includes: The ultra-wide-angle fundus images in the ultra-wide-angle fundus image dataset are cropped and then scaled to a preset standard resolution size; The contrast-limited adaptive histogram equalization algorithm is applied to the ultra-wide-angle fundus image; The ultra-wide-angle fundus images are normalized based on the channel mean and standard deviation calculated from the ultra-wide-angle fundus image dataset.

[0008] Optionally, before normalizing the ultra-wide-angle fundus image, the preprocessing of the ultra-wide-angle fundus image dataset further includes: Data augmentation is performed on the training set, and Gaussian blur and Gaussian noise are randomly applied to a portion of the ultra-wide-angle fundus images in the training set.

[0009] Optionally, when randomly dividing the training set, validation set, and test set according to a preset ratio, ensure that all ultra-wide-angle fundus images of the same person appear only in any one of the training set, validation set, and test set.

[0010] Optionally, the dual-stream hybrid feature extraction backbone network includes a Transformer branch and a CNN branch; wherein, the Transformer branch adopts a Swing Transformer structure to capture long-range dependencies and global spatial context of ultra-wide-angle fundus images through moving window attention; the CNN branch adopts a ConvNeXt-style convolutional neural network to extract local fine-grained texture features through continuous convolutional blocks. The dual-stream hybrid feature extraction backbone network also establishes lateral connections through the bidirectional attention interaction module in multiple stages of feature extraction in the Transformer branch and the CNN branch. The bidirectional attention interaction module is used to inject the global semantic information extracted by the Transformer branch into the CNN branch through an attention gating mechanism, and at the same time feed back the high-frequency local detail features extracted by the CNN branch to the Transformer branch. The dual-stream hybrid feature extraction backbone network embeds the feature modulation module in the deep network of the CNN branch. The feature modulation module is used to combine channel attention and spatial attention based on dilated convolution to expand the receptive field and suppress background noise without reducing the feature map resolution, thereby enhancing the network's feature response to key lesion areas. The dual-stream hybrid feature extraction backbone network further integrates the high-level features output by the Transformer branch and the CNN branch through the bidirectional attention interaction module, and inputs them into the cascaded classification module.

[0011] Optionally, the bidirectional attention interaction module is used for: The CNN branch feature X output by the CNN branch and the Transformer branch feature Y output by the Transformer branch are added element by element to obtain the fusion basis F_in; The fusion basis F_in is input into parallel local and global perceptual paths for processing. The outputs of the local and global perceptual paths are added together and then activated by Sigmoid to generate fusion weights W. The local perceptual path uses a bottleneck structure consisting of pointwise convolution, channelwise depthwise convolution, and pointwise convolution to extract edge textures. The global perceptual path uses global average pooling and global max pooling in parallel, and the results are added after processing by a multilayer perceptron to obtain a global semantic description. The fusion weights W are used to perform weighted aggregation on the CNN branch feature X and the Transformer branch feature Y to obtain the fusion feature F_fuse; The fusion feature F_fuse is used to generate corresponding gating coefficients through two independent convolutions. After activation by Sigmoid, the update directions of the corresponding CNN branch feature X and Transformer branch feature Y are adaptively adjusted. The fusion feature F_fuse is then used to update the CNN branch feature X and Transformer branch feature Y according to the corresponding update directions, resulting in the corresponding CNN branch enhancement feature Z1 and Transformer branch enhancement feature Z2.

[0012] Optionally, the feature modulation module is used to adaptively modulate the input features using cascaded channel attention branches and spatial attention branches; The channel attention branch is used to aggregate spatial information by global average pooling of the fusion base F_in, learn the channel dependencies through layer normalization and convolution, generate channel weights by Sigmoid activation, and multiply the channel weights with the fusion base F_in to obtain the channel-modulated feature F′. The spatial attention branch is used to perform average pooling and max pooling on the channel-modulated feature F′ in the channel dimension and then concatenate the features. Convolution and dilated convolution are then performed in parallel. The convolution result and the dilated convolution result are added together and activated by Sigmoid to generate a spatial attention map Ms. The spatial attention map Ms is then multiplied element-wise with the channel-modulated feature F′ to obtain the spatially modulated feature. The feature modulation module is also used to add the spatially modulated features, weighted by a learnable scaling parameter, to the fusion basis F_in to obtain the final output.

[0013] Optionally, the adaptive category attention module is used for: Convolution is used to expand the number of channels C of the input features to k×C, where k is the number of rank categories; After random deactivation, the input features are divided into k subgroups, each subgroup corresponding to a potential class. Parallel dilated convolution is performed on each subgroup to obtain the enhanced feature group F_group. Global average pooling and global max pooling are performed on the enhanced feature group F_group. The contributions of global average pooling and global max pooling are dynamically balanced by learnable parameters. The results of global average pooling and global max pooling are concatenated in the channel dimension and then weighted and fused. After convolution and sigmoid activation, channel attention weights are obtained. The channel attention weights are used to reweight the feature concatenation results of the enhanced feature group F_group in the channel dimension to obtain reweighted features. Convolution is used to compress the number of channels of the reweighted features back to the number of channels C, generating an attention map Matt; The learnable scaling parameters are added to the input features through residual connections to obtain the final output.

[0014] Optionally, the loss function is the cross-entropy loss function.

[0015] Optionally, using multi-dimensional evaluation metrics, the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images is comprehensively tested and evaluated, including: The overall consistency of the trained automatic grading network for diabetic retinopathy images in multi-classification tasks is evaluated by calculating the Kappa coefficient. The multi-classification problem is transformed into an independent binary classification problem, and the accuracy, sensitivity, specificity, and F1 score are calculated to deeply analyze the ability of the trained automatic grading network for diabetic retinopathy images to identify and classify each specific level of diabetic retinopathy.

[0016] Compared with existing technologies, this disclosure solves the problems of huge differences in scale between large field of view and small lesions in ultra-wide-angle fundus images and the difficulty in taking into account both global and local features. By using a dual-branch feature enhancement and fusion network, it can automatically and accurately capture global context and local small pathological details to accurately classify the severity of diabetic retinopathy. It also effectively suppresses interference from artifacts such as eyelashes and improves the classification sensitivity of early critical lesions. It can be used as a reference and service in various fields such as ophthalmic clinical decision support, large-scale diabetic retinopathy screening, and smart healthcare. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with the corresponding pictures in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart of a method for grading ultra-wide-angle fundus images of diabetic retinopathy provided in one embodiment of this disclosure; Figure 2 A schematic diagram of the structure of an automatic grading network for diabetic retinopathy images provided in another embodiment of this disclosure; Figure 3 A schematic diagram of the structure of a bidirectional attention interaction module provided in another embodiment of this disclosure; Figure 4 A schematic diagram of a feature modulation module provided for another embodiment of this disclosure; Figure 5 This is a schematic diagram of an adaptive category attention module provided for another embodiment of this disclosure. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.

[0020] One embodiment of this disclosure relates to a method for grading ultra-wide-angle fundus images of diabetic retinopathy, the process of which is as follows: Figure 1 As shown, it includes steps S110 to S150.

[0021] Step S110: Obtain the ultra-wide-angle fundus image dataset, preprocess the ultra-wide-angle fundus image dataset, and randomly divide it into training set, validation set and test set according to a preset ratio.

[0022] Specifically, the ultra-wide-angle fundus image dataset may include several ultra-wide-angle fundus images carrying corresponding grading labels. To ensure the authority and accuracy of the data annotation, all ultra-wide-angle fundus images are rigorously annotated by professional ophthalmologists with extensive clinical experience according to the International Clinical Grading Standard for Diabetic Retinopathy Severity, and are classified and labeled according to five levels: no obvious retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, and proliferative diabetic retinopathy, thus obtaining the corresponding grading label for each ultra-wide-angle fundus image.

[0023] Considering that ultra-wide-angle fundus images typically exhibit uneven illumination, fine lesions, and an imbalance in the number of samples from different categories, a unified preprocessing operation is first performed on them. For example, the preprocessing of the ultra-wide-angle fundus image dataset includes: cropping the ultra-wide-angle fundus images in the dataset and then scaling them to a preset standard resolution; applying a contrast-limited adaptive histogram equalization algorithm to the ultra-wide-angle fundus images; and normalizing the ultra-wide-angle fundus images based on the channel mean and standard deviation calculated from the dataset.

[0024] Specifically, considering the extremely large field of view of ultra-wide-angle fundus images and the presence of meaningless background areas and device edges, the ultra-wide-angle fundus images are first cropped to divide them into image blocks during preprocessing. Then, each image block is uniformly scaled to the standard resolution size required by the network, such as 512×512 pixels, to meet the network input size requirements and reduce computational complexity. After cropping, the ultra-wide-angle fundus image dataset, composed of image blocks, is divided into training, validation, and test sets according to a preset ratio, such as 7:1:2. In particular, when randomly dividing the training, validation, and test sets according to the preset ratio, it is ensured that all ultra-wide-angle fundus images of the same person appear only in one of the training, validation, or test sets, thereby avoiding data leakage issues caused by duplicate ownership of ultra-wide-angle fundus images.

[0025] Subsequently, the Limiting Contrast Adaptive Histogram Equalization (CLAHE) algorithm was applied to the ultra-wide-angle fundus images to enhance local image contrast and highlight subtle lesion features such as microaneurysms, hemorrhages, and exudations. Next, the ultra-wide-angle fundus images were normalized based on the channel mean and standard deviation calculated from the ultra-wide-angle fundus image dataset to accelerate the convergence speed during model training.

[0026] For example, in order to further alleviate model overfitting and enhance the model's adaptability to changes in the real clinical imaging environment, the ultra-wide-angle fundus image dataset is preprocessed before normalization of the ultra-wide-angle fundus images. This preprocessing also includes: performing data augmentation on the training set, and randomly applying Gaussian blur and Gaussian noise to some ultra-wide-angle fundus images in the training set.

[0027] Specifically, considering the potential imbalance in the number of samples across different categories in clinical data, data augmentation techniques such as random horizontal flipping, random vertical flipping, random rotation, and random color jitter can be used to increase the diversity of the training set. This simulates image changes under different shooting angles and lighting conditions, thereby improving the model's generalization ability. Simultaneously, Gaussian blur and Gaussian noise are randomly applied to some ultra-wide-angle fundus images in the training set to enhance the model's robustness to low-quality or noisy images.

[0028] Through the above data preprocessing and data augmentation operations, the data diversity can be effectively improved without destroying the key lesion information in the image, thereby providing a stable and reliable data foundation for the subsequent training of the dual-branch feature enhancement and fusion network.

[0029] Step S120: Construct a dual-branch feature enhancement and fusion network based on complementary global and local features as an automatic grading network for diabetic retinopathy lesion images; the automatic grading network for diabetic retinopathy lesion images includes a dual-stream hybrid feature extraction backbone network and a cascaded classification module, wherein the dual-stream hybrid feature extraction backbone network introduces a bidirectional attention interaction module and a feature modulation module, and the cascaded classification module includes a feature modulation module and an adaptive category attention module.

[0030] Specifically, considering the huge scale difference between the ultra-large field of view and the small pathological features in ultra-wide-angle fundus images, a single network is difficult to take into account both global context and local details, thus affecting the accuracy of classification. Therefore, a dual-stream hybrid feature extraction backbone network based on the complementarity of global and local features is adopted to extract features from the same ultra-wide-angle fundus image.

[0031] For example, combined Figure 2The dual-stream hybrid feature extraction backbone network employs a parallel dual-branch architecture, including a Transformer branch and a CNN branch. The Transformer branch uses a Swing Transformer structure as its Swing Transformer module, which captures long-range dependencies and global spatial context in ultra-wide-angle fundus images through moving window attention, effectively addressing geometric distortions caused by peripheral field stretching. The CNN branch uses a ConvNeXt-style convolutional neural network as its Convnext module, which focuses on extracting fine-grained local texture features of small lesions such as microaneurysms and hemorrhages through successive convolutional blocks. Specifically, the CNN branch may also include a Stem layer for preliminary local feature extraction and downsampling, improving computational efficiency.

[0032] To address the feature differences between different modalities and architectures, and to fully utilize the salient and complementary features of both global and local images for more accurate lesion grading, the dual-stream hybrid feature extraction backbone network establishes lateral connections through a bidirectional attention interaction module (BAIM) at multiple stages of feature extraction in the Transformer and CNN branches. The BAIM, a bidirectional adaptive interaction module designed based on the attention mechanism, injects global semantic information extracted by the Transformer branch into the CNN branch through an attention gating mechanism, providing macroscopic positional priors for local convolutions. Simultaneously, it feeds back high-frequency local detail features extracted by the CNN branch to the Transformer branch to compensate for the limitations of the self-attention mechanism in capturing microscopic textures. Through dynamic adaptive computation, the BAIM solves the problem of feature semantic misalignment between heterogeneous branches, achieving multi-level cross-branch information interaction and fusion.

[0033] Considering that ultra-wide-angle fundus images are often accompanied by complex background interference at the edges, the dual-stream hybrid feature extraction backbone network embeds a feature modulation module (FMM) in the deep network of the CNN branch. The feature modulation module is used to combine channel attention and spatial attention based on dilated convolution to expand the receptive field and suppress background noise such as eyelashes and eyelid shadows without reducing the feature map resolution, thereby enhancing the network's feature response to key lesion areas.

[0034] To enable the network to better combine the extracted multi-scale features and output accurate classification results, the dual-stream hybrid feature extraction backbone network also deeply fuses the high-level features output by the Transformer branch and the CNN branch through the bidirectional attention interaction module at the end, and inputs them into the cascaded classification module.

[0035] Combined Figure 2The cascaded classification module comprises a Feature Modulation Module (FMM) and an Adaptive Category Attention Module (ACAM). The ACAM, also known as the Category Attention Module, further extracts multi-scale features for specific categories using grouped dilated convolutions. By introducing learnable scaling parameters, it adaptively adjusts the fusion weights of global max pooling and global average pooling. Global max pooling focuses on capturing local, sudden lesion responses, while global average pooling focuses on aggregating global background context. This dynamic adjustment effectively improves the automatic grading network for diabetic retinopathy images, enabling the identification of early, mild, or borderline difficult-to-classify samples in ultra-wide-angle fundus image imbalance scenarios. Finally, the fused feature vector output by the ACAM is processed through fully connected layers to obtain accurate diabetic retinopathy lesion grading results.

[0036] For example, combined Figure 3 The bidirectional attention interaction module is used to: element-wise add the CNN branch features X and Transformer branch features Y as input to obtain the fusion basis F_in; input the fusion basis F_in into the parallel local and global perception paths for processing, add the outputs of the local and global perception paths, and then apply the Sigmoid activation function to generate the fusion weights W; the local perception path uses a bottleneck structure composed of pointwise convolution, channelwise depthwise convolution, and pointwise convolution to extract edge textures such as the edge textures of small lesions; the global perception path uses global average pooling (GAP) and global max pooling (GMP) in parallel, and after processing by a multilayer perceptron, they are added to obtain a global semantic description containing background and saliency information; the fusion weights W are used to weight and aggregate the CNN branch features X and Transformer branch features Y to obtain the fusion feature F_fuse. The dimensions of both the CNN branch features X and the Transformer branch features Y are C×H×W, where C represents the number of channels, H represents the height, and W represents the width.

[0037] To achieve bidirectional information flow, the bidirectional attention interaction module further introduces a gating update mechanism. This mechanism utilizes the fused feature F_fuse to generate corresponding gating coefficients through two independent convolutional layers (e.g., 1×1 convolutional layers). These coefficients are then activated by the Sigmoid activation function, adaptively adjusting the update directions of their respective CNN branch features X and Transformer branch features Y. Combined with the fused feature F_fuse, these features are updated according to their corresponding update directions, resulting in the corresponding CNN branch enhancement features Z1 and Transformer branch enhancement features Z2. This gating update mechanism injects lesion-related information from the joint feature (i.e., the fused feature F_fuse) into the corresponding branches with stronger weights, while suppressing irrelevant noise, thereby achieving effective complementarity and preservation of global semantics and local details.

[0038] For example, combined Figure 4 To enhance the network's ability to extract minute lesions and suppress background noise, the feature modulation module is used to adaptively modulate the input features using cascaded channel attention branches and spatial attention branches.

[0039] The channel attention branch is used to aggregate spatial information by global average pooling of the fusion base F_in. It learns the channel dependencies through layer normalization and convolution such as a 1×1 convolutional layer. The channel weights are generated by Sigmoid activation function. The channel weights are multiplied by the fusion base F_in to obtain the channel-modulated feature F′.

[0040] The spatial attention branch is used to perform average pooling and max pooling on the channel-modulated features F′ in the channel dimension and then concatenate the features. Convolution processing is then performed in parallel, such as using a standard 3×3 convolutional layer and dilated convolution processing, such as using a 3×3 dilated convolution with a dilation rate of d=3. The convolution processing result and the dilated convolution processing result are added together and then activated by the Sigmoid activation function to generate a spatial attention map Ms. The spatial attention map Ms is then multiplied element-wise with the channel-modulated features F′ to obtain the spatially modulated features, thereby achieving accurate localization of lesion areas and background filtering.

[0041] The feature modulation module is also used to add the spatially modulated features, weighted by the learnable scaling parameter γ, to the fusion basis F_in to obtain the final output.

[0042] For example, to address the training bias caused by the uneven distribution of categories in the grading of diabetic retinopathy, the adaptive category attention module dynamically recalibrates the feature response by aggregating channel grouping with adaptive statistical features.

[0043] Combined Figure 5 The adaptive category attention module is used to: expand the number of channels C of the input features with dimensions H×W×C to k×C using convolutional layers such as 1×1 convolutional layers, where k is the number of rank categories; after random deactivation, divide the input features with dimensions H×W×kL into k subgroups, each subgroup corresponding to a latent rank category, with each subgroup having a feature dimension of H×W×L; perform parallel dilated convolution processing on each subgroup, such as using 1×1 dilated convolution with a dilation rate d=2, to capture contextual information under different receptive fields, resulting in an enhanced feature group F_group; introduce an adaptive feature aggregation mechanism to generate channel weights: perform global average pooling and global max pooling on the enhanced feature group F_group to obtain global average pooling and global max pooling results with feature dimensions of 1×1×L. By dynamically balancing the contributions of global average pooling and global max pooling using a learnable parameter α, the results of global average pooling and global max pooling are concatenated along the channel dimension and then weighted and fused. After activation by convolutional layers such as 1×1 and the Sigmoid activation function, channel attention weights are obtained. The enhanced feature group F_group is then reweighted along the channel dimension using these channel attention weights to obtain reweighted features, which suppress irrelevant information and enhance salient features. The number of channels of the reweighted features is compressed back to the number of channels C using convolutional layers such as 1×1, generating an attention map Matt. A learnable scaling parameter γ, initialized to 0, is introduced and added to the input features through residual connections to obtain the final output, ensuring stable feature propagation and focusing on hard-to-classify samples.

[0044] Step S130: The ultra-wide-angle fundus images and their corresponding grading labels in the training set are fed into the automatic grading network for diabetic retinopathy images for training. The loss function of the automatic grading network for diabetic retinopathy images is calculated, and the model weights of the automatic grading network for diabetic retinopathy images are iteratively updated by combining a dynamic learning rate strategy until the preset number of iterations is reached. After each iteration, the performance of the automatic grading network for diabetic retinopathy images is evaluated using a validation set, and the model weights that perform best on the validation set are saved to obtain the trained automatic grading network for diabetic retinopathy images.

[0045] For example, the loss function used is the cross-entropy loss function, which is used to accurately measure the difference between the grade category probability distribution output by the automatic grading network for diabetic retinopathy images and the true grading labels of the ultra-wide-angle fundus images, so that the automatic grading network for diabetic retinopathy images gradually reaches its optimum during training.

[0046] Specifically, before training the automatic grading network for diabetic retinopathy images, to address the data imbalance problem in ultra-wide-angle fundus images from real-world clinical scenarios, initial weights were set for each target grade category based on its proportion. Target grade categories with lower proportions (such as severe non-proliferative and proliferative lesions) received higher attention. Simultaneously, during the iterative update of model weights, AdamW was used as the model optimizer, and a dynamic adjustment method combining warmup and cosine annealing was employed for the learning rate update. This helps prevent the automatic grading network from getting trapped in local optima during training and avoids the impact of sudden changes in the learning rate on the training process.

[0047] Step S140: Use the test set to test the grading effect of the trained automatic grading network for diabetic retinopathy images.

[0048] Specifically, ultra-wide-angle fundus images from the test set are input into a pre-trained automatic grading network for diabetic retinopathy images. The grading effectiveness of the network is determined based on its output grading results. The ultra-wide-angle fundus images in the test set undergo the same preprocessing as those in the training set.

[0049] Step S150: Apply the trained automatic grading network for diabetic retinopathy images to grade diabetic retinopathy in ultra-wide-angle fundus images. Use multi-dimensional evaluation indicators to comprehensively test and evaluate the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images.

[0050] For example, in step S150, multi-dimensional evaluation metrics are used to comprehensively test and evaluate the automatic classification and recognition performance of the trained automatic classification network for diabetic retinopathy images. This includes: evaluating the overall consistency of the trained automatic classification network for diabetic retinopathy images in multi-classification tasks by calculating the Kappa coefficient; transforming the multi-classification problem into an independent binary classification problem; and calculating accuracy, sensitivity, specificity, and F1 score to deeply analyze the recognition and classification capabilities of the trained automatic classification network for each specific diabetic retinopathy level. This allows for a deeper analysis of the true performance and shortcomings of the trained automatic classification network for diabetic retinopathy images from different dimensions. The F1 score is essentially the harmonic mean of precision and recall.

[0051] To enable those skilled in the art to better understand the above embodiments, a specific example is provided below for illustration.

[0052] A method for grading diabetic retinopathy using ultra-wide-angle fundus images is proposed. This method, based on a bi-branch feature enhancement and fusion network, is applied to automated grading tasks for diabetic retinopathy in the field of medical image processing. The method includes: first, acquiring an ultra-wide-angle fundus image dataset; preprocessing the ultra-wide-angle fundus images; and then randomly dividing the dataset into training, validation, and test sets according to a preset ratio. Subsequently, a method is constructed as follows: Figure 2 The dual-branch feature enhancement and fusion network based on complementary global and local features is used as the automatic grading network for diabetic retinopathy images. Next, ultra-wide-angle fundus images from the training set and their corresponding grading labels (labels indicating lesion levels) are input into the constructed automatic grading network for training. The network weights are optimized to learn the mapping relationship between image features and lesion levels. After training reaches a preset number of iterations and the performance of the automatic grading network stabilizes, the optimal network weights—the model weights that perform best on the validation set—are saved. Then, ultra-wide-angle fundus images from the test set are input into the trained automatic grading network to obtain the corresponding grading results for diabetic retinopathy, completing the automatic grading task for diabetic retinopathy images under ultra-wide-angle fundus images. Finally, multi-dimensional evaluation metrics are used to comprehensively test and evaluate the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images. The following steps (steps one through five) provide a detailed explanation of this method for grading ultra-wide-angle fundus images of diabetic retinopathy.

[0053] Step 1: Data Preprocessing: 11,018 ultra-wide-angle fundus images were selected to form an ultra-wide-angle fundus image dataset. All images had a uniform resolution of 3900×3072 pixels. To ensure the authority and accuracy of the data annotation, all images were strictly annotated by experienced ophthalmologists according to the International Clinical Grading Standard for Diabetic Retinopathy Severity. The images were then classified and labeled according to five diabetic retinopathy levels: no obvious retinopathy, mild non-proliferative diabetic retinopathy, moderate non-proliferative diabetic retinopathy, severe non-proliferative diabetic retinopathy, and proliferative diabetic retinopathy, thus obtaining the corresponding grading labels for each ultra-wide-angle fundus image.

[0054] To ensure the reliability of model training and evaluation, and to effectively assess the model's generalization ability on unknown data, all 11,018 ultra-wide-angle fundus images were randomly divided into training, validation, and test sets in a 7:1:2 ratio. During the data partitioning process, it was ensured that all images of the same patient appeared in only one subset, thereby avoiding data leakage issues caused by duplicate patients.

[0055] Considering the characteristics of ultra-wide-angle fundus images, such as uneven illumination, fine lesions, and uneven sample sizes across different categories, a unified preprocessing operation was first performed on the data. All 11,018 ultra-wide-angle fundus images were uniformly scaled to 512×512 pixels to meet the network input size requirements and reduce computational complexity. Subsequently, a contrast-limited adaptive histogram equalization algorithm was applied to each ultra-wide-angle fundus image to enhance local image contrast and highlight subtle lesion features such as microaneurysms, hemorrhages, and exudations. Finally, the channel mean and standard deviation calculated from the ultra-wide-angle fundus image dataset were used to normalize each ultra-wide-angle fundus image to accelerate the convergence speed during model training.

[0056] To further alleviate model overfitting and enhance its adaptability to changes in real clinical imaging environments, data augmentation operations were performed on the training set before normalization. These operations included random horizontal flipping, random vertical flipping, random rotation, and random color jittering to simulate image changes under different shooting angles and lighting conditions. Simultaneously, Gaussian blur and Gaussian noise were randomly applied to some ultra-wide-angle fundus images in the training set to enhance the model's robustness to low-quality or noisy images. Through these data preprocessing and augmentation operations, data diversity can be effectively improved while ensuring that key lesion information in the images is not destroyed, thus providing a stable and reliable data foundation for the subsequent training of the dual-branch feature enhancement and fusion network.

[0057] Step 2: Model Building: Construct a dual-branch feature enhancement and fusion network based on complementary global and local features as the automatic grading network for diabetic retinopathy lesion images. The automatic grading network for diabetic retinopathy lesion images includes a dual-stream hybrid feature extraction backbone network and a cascaded classification module. The dual-stream hybrid feature extraction backbone network incorporates a bidirectional attention interaction module and a feature modulation module. The cascaded classification module includes a feature modulation module and an adaptive category attention module. The overall network structure of the automatic grading network for diabetic retinopathy lesion images is as follows: Figure 2 As shown.

[0058] An automatic grading network for diabetic retinopathy images employs a dual-stream hybrid feature extraction backbone network to perform parallel feature extraction from the same ultra-wide-angle fundus image. The dual-stream hybrid feature extraction backbone network consists of two branches. One branch is a Transformer branch, which uses a Swing Transformer structure and a moving window self-attention mechanism to perform hierarchical feature modeling, thereby acquiring long-range dependencies and global spatial context information in the ultra-wide-angle fundus image and adapting to the field-of-view stretching and geometric distortion caused by ultra-wide-angle imaging. The other branch is a CNN branch, which uses a ConvNeXt-style convolutional neural network structure to extract local texture information and fine-grained structural features from the image through continuous convolutional modules, including lesion features such as microaneurysms, hemorrhages, and hard exudates. The dual-branch structure of the dual-stream hybrid feature extraction backbone network simultaneously models global semantic information and local detail information, resulting in a more comprehensive feature representation.

[0059] The dual-stream hybrid feature extraction backbone network also embeds structures such as... between multiple stages of the Transformer and CNN branches. Figure 3 The diagram shows a bidirectional attention interaction module. This module constructs a cross-branch feature interaction structure based on an attention mechanism. Global semantic information extracted by the Transformer branch is injected into the CNN branch through an attention gating mechanism, thereby introducing global positional prior information during convolutional feature learning. High-frequency local detail features extracted by the CNN branch are fed back to the Transformer branch through reverse attention interaction, enabling the self-attention mechanism to obtain richer local texture representations. This bidirectional interaction structure achieves feature semantic alignment between heterogeneous networks and completes cross-branch information fusion at multiple scales.

[0060] The dual-stream hybrid feature extraction backbone network also embeds features such as... into the deep networks of CNN branches. Figure 4 The feature modulation module is shown. This module combines channel attention and dilated convolution-based spatial attention to adaptively modulate the feature map. The dilated convolution structure expands the network's receptive field while maintaining the feature map resolution, allowing features to cover a wider range of image context information. The channel attention mechanism weights the feature channels, enhancing feature responses related to lesion areas while suppressing interference from eyelashes, eyelid shadows, and other background noise.

[0061] The high-level features output from the dual-stream hybrid feature extraction backbone network are deeply fused through a bidirectional attention interaction module at the terminal, and the deeply fused features are then input into a cascaded classification module for lesion grade determination. The cascaded classification module includes a feature modulation module and, for example, a feature modulation module and, if necessary, a feature enhancement module. Figure 5The adaptive category attention module is shown. This module extracts category-related features at different scales through grouped dilated convolutions and combines global max pooling and global average pooling to aggregate global information from these features. Learnable parameters are introduced to adaptively weight and fuse the two pooling methods. Global max pooling emphasizes highlighting lesion regions with strong local responses, while global average pooling focuses on integrating global background semantic information. Through a dynamic weight adjustment mechanism, the network can adjust the feature contribution based on the importance of different categories of features, thereby improving the ability to identify early, mild lesions and borderline samples. The fused feature vector outputs predictions for five diabetic retinopathy grade categories through a fully connected layer, achieving automatic grading of ultra-wide-angle fundus images.

[0062] Step 3: Model Training: Training was performed on a workstation with an i7-7700 CPU, an NVIDIA GeForce RTX 3090 GPU, and 16GB of RAM. The programming language was Python 3.11, and the deep learning framework was PyTorch. All modules of the automatic grading network for diabetic retinopathy images were loaded with weights pre-trained on the ImageNet dataset as initialization parameters to accelerate convergence and improve generalization ability. During training, the cross-entropy loss function was used to measure the difference between the probability distribution of the grade categories predicted by the automatic grading network for diabetic retinopathy images and the true grade labels of the ultra-wide-angle fundus images. The optimizer was AdamW, with an initial learning rate of 0.0001, a momentum parameter of 0.9, and a weight decay of 0.05. The learning rate update strategy adopted a dynamic adjustment method combining warmup and cosine annealing, and the learning rate was updated after each step, rather than after each epoch, to help the automatic grading network for diabetic retinopathy images escape local optima and avoid the impact of sudden changes in the learning rate on the training process. The preset number of iterations was set to 100, and the training was carried out for a total of 100 epochs. After each epoch, the performance of the automatic grading network for diabetic reticulum lesions was evaluated on the validation set, and the model weights of the automatic grading network for diabetic reticulum lesions that performed best on the validation set were saved.

[0063] Step 4: Model Testing and Evaluation: To verify the performance of the trained automatic diabetic retinopathy image grading network, it was tested on a test set. The ultra-wide-angle fundus images in the test set underwent the same preprocessing as those in the training set. These images were then fed into the automatic diabetic retinopathy image grading network loaded with the model weights that performed best on the validation set. The network output predicted diabetic retinopathy grading results. Based on these grading results, the automatic diabetic retinopathy image grading effect of the trained network was determined. The Kappa coefficient was calculated to evaluate the overall consistency of the multi-classification task, transforming the multi-classification problem into an independent binary classification problem. Accuracy, sensitivity, specificity, and F1 score were calculated to further analyze the network's ability to identify and classify each specific diabetic retinopathy grade. Testing and evaluation of the trained automatic grading network for diabetic retinopathy images revealed that, compared to traditional single-branch networks or classification algorithms without feature enhancement, this network can more effectively capture lesion information, achieve better grading results, and has higher accuracy.

[0064] Compared with the prior art, the embodiments disclosed herein solve the problems of huge differences in scale between large field of view and small lesions in ultra-wide-angle fundus images and the difficulty in taking into account both global and local features. By using a dual-branch feature enhancement and fusion network, it can automatically and accurately capture global context and local small pathological details to accurately classify the severity of diabetic retinopathy. It also effectively suppresses interference from artifacts such as eyelashes and improves the classification sensitivity of early critical lesions. It can be used as a reference and service in various fields such as ophthalmic clinical decision support, large-scale diabetic retinopathy screening, and smart healthcare.

[0065] Those skilled in the art will understand that the above embodiments are specific implementations of this disclosure, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this disclosure.

Claims

1. A method for grading ultra-wide-angle fundus images of diabetic retinopathy, characterized in that, include: A dataset of ultra-wide-angle fundus images is obtained, the dataset is preprocessed, and randomly divided into a training set, a validation set, and a test set according to a preset ratio. A dual-branch feature enhancement and fusion network based on complementary global and local features is constructed as an automatic grading network for diabetic retinopathy lesion images. The automatic grading network for diabetic retinopathy lesion images includes a dual-stream hybrid feature extraction backbone network and a cascaded classification module. The dual-stream hybrid feature extraction backbone network introduces a bidirectional attention interaction module and a feature modulation module. The cascaded classification module includes the feature modulation module and an adaptive category attention module. The ultra-wide-angle fundus images and their corresponding grading labels in the training set are fed into the automatic grading network for diabetic retinopathy images for training. The loss function of the automatic grading network for diabetic retinopathy images is calculated, and the model weights of the automatic grading network for diabetic retinopathy images are iteratively updated by combining a dynamic learning rate strategy until a preset number of iterations is reached. After each iteration, the performance of the automatic grading network for diabetic retinopathy images is evaluated using the validation set, and the model weights that perform best on the validation set are saved to obtain the trained automatic grading network for diabetic retinopathy images. The grading performance of the trained automatic grading network for diabetic retinopathy images was tested using the test set. The trained automatic grading network for diabetic retinopathy images was applied to grade diabetic retinopathy in ultra-wide-angle fundus images. Multi-dimensional evaluation indicators were used to comprehensively test and evaluate the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images. The dual-stream hybrid feature extraction backbone network includes a Transformer branch and a CNN branch; wherein, the Transformer branch adopts a Swing Transformer structure to capture long-range dependencies and global spatial context of ultra-wide-angle fundus images through moving window attention; the CNN branch adopts a ConvNeXt-style convolutional neural network to extract local fine-grained texture features through continuous convolutional blocks. The dual-stream hybrid feature extraction backbone network also establishes lateral connections through the bidirectional attention interaction module in multiple stages of feature extraction in the Transformer branch and the CNN branch. The bidirectional attention interaction module is used to inject the global semantic information extracted by the Transformer branch into the CNN branch through an attention gating mechanism, and at the same time feed back the high-frequency local detail features extracted by the CNN branch to the Transformer branch. The dual-stream hybrid feature extraction backbone network embeds the feature modulation module in the deep network of the CNN branch. The feature modulation module is used to combine channel attention and spatial attention based on dilated convolution to expand the receptive field and suppress background noise without reducing the feature map resolution, thereby enhancing the network's feature response to key lesion areas. The dual-stream hybrid feature extraction backbone network further integrates the high-level features output by the Transformer branch and the CNN branch through the bidirectional attention interaction module, and inputs them into the cascaded classification module.

2. The method for grading diabetic retinopathy ultra-wide-angle fundus images according to claim 1, characterized in that, The preprocessing of the ultra-wide-angle fundus image dataset includes: The ultra-wide-angle fundus images in the ultra-wide-angle fundus image dataset are cropped and then scaled to a preset standard resolution size; The contrast-limited adaptive histogram equalization algorithm is applied to the ultra-wide-angle fundus image; The ultra-wide-angle fundus images are normalized based on the channel mean and standard deviation calculated from the ultra-wide-angle fundus image dataset.

3. The method for grading diabetic retinopathy ultra-wide-angle fundus images according to claim 2, characterized in that, Before normalizing the ultra-wide-angle fundus image, the preprocessing of the ultra-wide-angle fundus image dataset further includes: Data augmentation is performed on the training set, and Gaussian blur and Gaussian noise are randomly applied to a portion of the ultra-wide-angle fundus images in the training set.

4. The method for grading diabetic retinopathy ultrawide-angle fundus images according to any one of claims 1 to 3, characterized in that, When randomly dividing the training set, validation set, and test set according to a preset ratio, ensure that all ultra-wide-angle fundus images of the same person appear only in any one of the training set, validation set, and test set.

5. The method for grading diabetic retinopathy ultra-wide-angle fundus images according to claim 1, characterized in that, The bidirectional attention interaction module is used for: The CNN branch feature X output by the CNN branch and the Transformer branch feature Y output by the Transformer branch are added element by element to obtain the fusion basis F_in; The fusion basis F_in is input into parallel local and global perceptual paths for processing. The outputs of the local and global perceptual paths are added together and then activated by Sigmoid to generate fusion weights W. The local perceptual path uses a bottleneck structure consisting of pointwise convolution, channelwise depthwise convolution, and pointwise convolution to extract edge textures. The global perceptual path uses global average pooling and global max pooling in parallel, and the results are added after processing by a multilayer perceptron to obtain a global semantic description. The fusion weights W are used to perform weighted aggregation on the CNN branch feature X and the Transformer branch feature Y to obtain the fusion feature F_fuse; The fusion feature F_fuse is used to generate corresponding gating coefficients through two independent convolutions. After activation by Sigmoid, the update directions of the corresponding CNN branch feature X and Transformer branch feature Y are adaptively adjusted. The fusion feature F_fuse is then used to update the CNN branch feature X and Transformer branch feature Y according to the corresponding update directions, resulting in the corresponding CNN branch enhancement feature Z1 and Transformer branch enhancement feature Z2.

6. The method for grading diabetic retinopathy ultra-wide-angle fundus images according to claim 5, characterized in that, The feature modulation module is used to adaptively modulate the input features using cascaded channel attention branches and spatial attention branches; The channel attention branch is used to aggregate spatial information by global average pooling of the fusion base F_in, learn the channel dependencies through layer normalization and convolution, generate channel weights by Sigmoid activation, and multiply the channel weights with the fusion base F_in to obtain the channel-modulated feature F′. The spatial attention branch is used to perform average pooling and max pooling on the channel-modulated feature F′ in the channel dimension and then concatenate the features. Convolution and dilated convolution are then performed in parallel. The convolution result and the dilated convolution result are added together and activated by Sigmoid to generate a spatial attention map Ms. The spatial attention map Ms is then multiplied element-wise with the channel-modulated feature F′ to obtain the spatially modulated feature. The feature modulation module is also used to add the spatially modulated features, weighted by a learnable scaling parameter, to the fusion basis F_in to obtain the final output.

7. The method for grading diabetic retinopathy ultra-wide-angle fundus images according to claim 6, characterized in that, The adaptive category attention module is used for: Convolution is used to expand the number of channels C of the input features to k×C, where k is the number of rank categories; After random deactivation, the input features are divided into k subgroups, each subgroup corresponding to a potential class. Parallel dilated convolution is performed on each subgroup to obtain the enhanced feature group F_group. Global average pooling and global max pooling are performed on the enhanced feature group F_group. The contributions of global average pooling and global max pooling are dynamically balanced by learnable parameters. The results of global average pooling and global max pooling are concatenated in the channel dimension and then weighted and fused. After convolution and sigmoid activation, channel attention weights are obtained. The channel attention weights are used to reweight the feature concatenation results of the enhanced feature group F_group in the channel dimension to obtain reweighted features. Convolution is used to compress the number of channels of the reweighted features back to the number of channels C, generating an attention map Matt; The learnable scaling parameters are added to the input features through residual connections to obtain the final output.

8. The method for grading diabetic retinopathy ultrawide-angle fundus images according to any one of claims 5 to 7, characterized in that, The loss function used is the cross-entropy loss function.

9. The method for grading diabetic retinopathy ultrawide-angle fundus images according to claim 1, characterized in that, Using multi-dimensional evaluation metrics, the automatic grading and recognition performance of the trained automatic grading network for diabetic retinopathy images was comprehensively tested and evaluated, including: The overall consistency of the trained automatic grading network for diabetic retinopathy images in multi-classification tasks is evaluated by calculating the Kappa coefficient. The multi-classification problem is transformed into an independent binary classification problem, and the accuracy, sensitivity, specificity, and F1 score are calculated to deeply analyze the ability of the trained automatic grading network for diabetic retinopathy images to identify and classify each specific level of diabetic retinopathy.

Images