Heterogeneous dual-stream fusion method and system for diabetic retinopathy grading

Abstract

This invention discloses a heterogeneous two-stream fusion method and system for grading diabetic retinopathy (DR), comprising obtaining the DR grading output using a heterogeneous two-stream architecture: processing the input fundus image into images of different resolutions; extracting global contextual features from the low-resolution image using a lightweight visual Transformer model distilled from composite knowledge, and extracting local lesion features from the high-resolution image using a convolutional neural network model; interactively fusing the global contextual features and local lesion features of the two-branch architecture through a symmetrical bidirectional cross-attention fusion module to obtain an enhanced fused feature representation; and finally inputting the fused features into a classifier to output the DR severity grading result. This invention aims to improve the accuracy and robustness of grading diagnosis through in-depth analysis of global information and local details, and can be applied to medical fields such as clinical computer-aided diagnosis and ocular image analysis.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a heterogeneous dual-stream fusion method and system for grading diabetic retinopathy. Background Technology

[0002] Diabetic retinopathy (DR) is a fundus complication caused by long-term hyperglycemia that progressively damages the blood vessels in the retina and is one of the leading causes of blindness in working-age adults. The progression of DR can be divided into different stages, such as non-proliferative DR (NPDR) and proliferative DR (PDR). Early-stage lesions, such as microaneurysms and hemorrhages, may present with mild or even no symptoms. If these lesions are not detected and intervened in time, they will damage vision and, in severe cases, even lead to blindness.

[0003] In practice, the screening and severity grading of diabetic retinopathy (DR) largely rely on ophthalmologists manually reviewing images. As the number of DR patients gradually increases, manual case analysis consumes significant manpower and time. Furthermore, the subtle differences between different degrees of DR lesions, the increasing complexity of the lesions, and the susceptibility of diagnostic results to the influence of physicians' subjective experience pose challenges to the consistency and accuracy of diagnoses. Therefore, developing computer-aided diagnostic (CAD) tools capable of large-scale, high-efficiency screening has become an urgent medical need.

[0004] In recent years, existing research has proposed various deep learning-based automatic DR (Retinal Diagnosis) grading schemes, which can be mainly summarized into the following categories: Methods using Convolutional Neural Networks (CNNs) as the core architecture. These methods leverage the inherent local inductive bias of CNNs, demonstrating excellent performance in extracting local lesion features (such as microaneurysms and exudates) from fundus images. However, the CNN architecture is inherently limited by the local receptive field of convolution operations, making it difficult to effectively establish dependencies between distant pixels in an image, such as understanding the overall direction of retinal vessels or the distribution of lesions. Methods employing Visual Transformer (ViT) architectures, especially those utilizing large-scale base models pre-trained on massive amounts of image data, such as the RETFound model. These methods, with their self-attention mechanism, can capture global dependencies and contextual information in images, exhibiting powerful performance in image processing. However, the large number of parameters leads to high computational costs and a high barrier to clinical deployment. Furthermore, due to the lack of inductive bias in CNNs, their accuracy is sometimes lower than that of CNN models when analyzing and identifying fine local lesion textures that are small in pixel proportion but important. Current research focuses on hybrid architectures that integrate CNNs and ViT. Existing fusion methods typically employ a serial structure, while for parallel structures, feature fusion is often achieved through simple concatenation or element-wise addition. These simple static fusion strategies have shortcomings, easily ignoring the heterogeneity between local spatial hierarchical features and global contextual features, failing to achieve deep fusion of heterogeneous features, and thus limiting the improvement of model performance.

[0005] Currently, there is a lack of an automatic grading method for the severity of diabetic retinopathy that can simultaneously capture global contextual information and local lesion details, achieve a good balance between model performance and computational efficiency, and enable deep adaptive fusion of heterogeneous features. Summary of the Invention

[0006] The purpose of this invention is to provide a heterogeneous two-stream fusion method for grading diabetic retinopathy. This addresses the problems of existing diabetic retinopathy grading methods, such as limited representational capabilities of single models, high computational costs of large models, high barriers to clinical deployment, and insufficient fusion of heterogeneous features in hybrid architectures.

[0007] To achieve the above objectives, the present invention provides the following solution:

[0008] This invention provides a heterogeneous dual-stream fusion method for grading diabetic retinopathy, comprising the following steps:

[0009] S1: Global Feature Extraction Module: This module uses composite knowledge distillation technology to transfer the knowledge of a large basic model to a lightweight ViT encoder to process low-resolution 224×224 images, efficiently capturing global contextual information while significantly reducing computational cost and parameter count to meet the needs of efficient and lightweight deployment.

[0010] S2: This branch employs a CNN encoder (EfficientNet-B5) pre-trained on ImageNet. It is used to process high-resolution images of 456×456 and utilizes the inherent local inductive bias of CNNs to extract fine local texture and spatial hierarchy features of small lesions, effectively compensating for the shortcomings of ViT in analyzing subtle local lesions.

[0011] S3: Symmetrical Bidirectional Cross-Attention Module: This module is designed to achieve adaptive, symmetrical bidirectional cross-attention depth interaction and enhancement between two heterogeneous branches (global ViT features and local CNN features). Through this dynamic fusion mechanism, the method ensures that it can comprehensively consider both the microscopic details of DR lesions and the macroscopic structure of retinal images, avoiding information conflicts caused by simple static fusion.

[0012] S4: To map the fused high-dimensional feature vector to the final class prediction, cross-entropy loss is used for model optimization. The fused features are then fed into the multilayer perceptron classification head or the DR (Digital Regression) classification diagnostic results.

[0013] Furthermore, in S1, a composite knowledge distillation scheme is designed, which includes soft target distillation, feature distillation, and contrastive distillation to comprehensively transfer knowledge from the teacher model to the lightweight student model from multiple dimensions (including prediction distribution, feature hierarchy, and data relationships). In the experiments, the RETFound encoder serves as the teacher model, and a ViT-Base model with reduced depth and width serves as the student model. With the teacher model parameters frozen, a composite loss function is used... To jointly optimize the student model, the function specifically includes:

[0014] S1.1, Soft Target Distillation Technology:

[0015] Simply using hard labels for supervision ignores the implicit knowledge inherent in class probability prediction, namely, the similarity information between classes. This invention uses labels with a temperature coefficient. The Softmax function is used to soften the logits output of the teacher model. By minimizing the KL divergence between the soft objectives of the student and teacher models, the student model is forced to fit the fine distribution of the class probabilities of the teacher model. This part of the loss function is defined as:

[0016]

[0017] In the above formula, and These are the logits outputs of the student and teacher models, respectively. It is the Softmax function. It is the temperature coefficient. When When the value is greater than 1, the output distribution becomes flatter, which helps the student model learn richer inter-category structural information. The factor is used to scale the gradient during backpropagation, ensuring that it is applied at different temperatures. Under these conditions, the contribution magnitudes of the loss terms for soft and hard targets remain relatively stable.

[0018] S1.2, Characteristic distillation technology:

[0019] To guide student models in simulating more robust intermediate feature representations of teacher models, this invention introduces feature distillation. This method encourages student models to utilize pre-defined intermediate layers... The feature map of the student model should be matched with the feature map of the teacher model. Considering that the differences in network structure between the student and teacher models may lead to inconsistent feature map dimensions, this invention introduces an adaptation layer φ to align dimensions. This loss term is defined as the square of the L2 norm between the adapted student features and teacher features at a specified intermediate layer, as follows:

[0020]

[0021] In the above formula, It is the selected set of intermediate layers. and The student and teacher models represent the second and third classes, respectively. Layer feature representation, It is for the first Learnable adaptation layers of layer feature maps. Minimize this loss by directly supervising the intermediate representations of the student model in the feature space.

[0022] S1.3, Comparative learning of distillation techniques:

[0023] Higher-level knowledge is embedded in the structural relationships between data samples. To transfer the relational knowledge learned by the teacher model, this study constructs a contrastive learning distillation loss. Its core idea is that if two samples are similar (positive sample pairs) in the teacher model's representation space, then they should also be brought closer in the student model's representation space; otherwise, they should be pushed apart. This loss can be formalized as:

[0024]

[0025] In the above formula, This refers to the batch size. It is the student model for anchor point samples Feature representation, It is the feature representation of the teacher model for the same positive sample, while This represents the teacher feature representation of N negative samples. The vector inner product is used to calculate similarity. It is a temperature hyperparameter used to adjust the sharpness of the similarity distribution.

[0026] The above three distillation losses are compared with the standard supervised learning loss (i.e., the standard cross-entropy loss). The weighted combinations are then used to form the final distillation objective function:

[0027]

[0028] in, It is the cross-entropy loss between the student model and the real label, used to ensure the model's basic prediction performance on the real label. These are hyperparameters used to balance the relative importance of the three losses: soft target distillation, characteristic distillation, and comparative distillation. The composite objective function is jointly optimized. This invention successfully reduced the number of parameters in the student model from 3.7 billion to 327 million, while minimizing performance loss in key downstream tasks. The resulting ViT architecture includes image patch embedding, location embedding, class labeling, and a Transformer encoder layer, successfully providing a powerful and efficient global feature encoder for hybrid models, while inheriting the representation capabilities of the RETFound base model.

[0029] S2 selects EfficientNet-B5 as the local feature CNN branch, focusing on extracting fine local texture and spatial hierarchy features from high-resolution images:

[0030] Among the various variants of EfficientNet, the B5 variant offers an ideal balance between parameter count and performance, and its powerful texture processing capabilities are well-suited for medical image analysis. This invention uses weights pre-trained on the large natural image dataset ImageNet to... Perform initialization, As an initial feature extractor, it can capture local spatial patterns and hierarchical features in images. The main motivation for transfer learning lies in the limited number of high-resolution fundus images available for model training. Utilizing knowledge from diverse datasets such as ImageNet enhances the model's generalization ability and maintains good performance even with a limited number of training images. In practical processing, this branch receives... High-resolution preprocessed image (e.g., 456×456×3) As input, the image is encoded through convolutional layers of EfficientNet-B5, and after a series of MBConv blocks, hierarchical features from low to high levels are extracted step by step. This invention takes the final feature map before the global average pooling layer and flattens it to obtain a... Local eigenvectors of dimension .

[0031] Furthermore, in step S3, this invention designs a symmetrical bidirectional cross-attention fusion module. The core mechanism of this module is: to promote... and Deep interaction allows two features to act as queries, actively seeking the desired information from each other's features (as keys and values). The specific method is as follows:

[0032] Global-to-local information augmentation: The module uses three independent learnable linear projection layers (i.e., fully connected layers, with weight matrices respectively) , , This is used to generate the query, key, and value matrix required for the attention mechanism. Global features. As a query, from local features The process extracts relevant local details to enrich the global view with fine texture. The attention output generated in this process is added to the original global features via residual connections, then stabilized through a normalization layer, ultimately resulting in an enhanced global feature that absorbs local details. This process can be formally represented as:

[0033]

[0034] in, Depend on Generate, and and Depend on Linear projection generation. Output. This is the enhanced global feature.

[0035] Local-to-global information enhancement: Symmetrically, local features As a query, from global features The process involves identifying the context within the lesion to understand its importance from a global perspective. This process also includes residual connections and layer normalization to obtain an enhanced local feature that incorporates global context awareness. This process can be formally represented as:

[0036]

[0037] in, Depend on Generate, and and Depend on generate.

[0038] After the attention output from the two branches described above, a unified residual fusion function and layer normalization strategy are applied after each branch. The extracted fusion information is then fused with the original features again through residual connections (i.e., element-wise addition). The fused result is then input into a layer normalization unit for processing to avoid missing key information. This process ultimately generates features that absorb key local details and fuse global information, ensuring the model's training stability. Finally, these two feature vectors, which have undergone bidirectional interaction and enhancement, are... and By splicing together, a fusion feature is formed. The data is then fed into a classifier for decision-making. This process can be formally represented as:

[0039]

[0040] The pre-trained, powerful EfficientNet-B5 branch inherits the precise ability of CNNs to capture local lesions (such as microaneurysms and hemorrhages), while the lightweight ViT branch, after knowledge distillation, constructs global context dependencies, fusing the features of both to generate more comprehensive features. Through this design, HDNet overcomes the limitation of traditional static fusion, which tends to ignore deep heterogeneous relationships between features, achieving a dynamic, context-aware feature enhancement that enables the model to simultaneously perceive the microscopic details of DR lesions and the macroscopic structure of retinal images.

[0041] To fuse the high-dimensional feature vector To map to the final class prediction, a classification head is designed, and the model is optimized using cross-entropy loss. The data is fed into a classification head consisting of a multilayer perceptron (MLP) containing two hidden layers with ReLU activation (dimensions 512 and 256, respectively), designed for learning. A non-linear mapping between the score and the category label. Outputs an M-dimensional original score vector (logits). The Softmax function is used to transform z into a class probability distribution. The entire model is trained in an end-to-end manner, and its optimization objective is to minimize the predicted probability distribution. With real labels Classification cross-entropy loss For those containing The dataset of a sample, the total loss on the entire dataset It is the mean of the losses of all samples, and the total loss is defined as:

[0042]

[0043] in It is a one-hot encoded real tag. The model predicts the sample. Category The probability of this is determined. Finally, the classification head outputs the diagnostic result of the diabetic lesion grading based on the input fusion enhancement features.

[0044] Furthermore, the present invention also provides a heterogeneous dual-stream fusion system for grading diabetic retinopathy, comprising a processor and a memory interconnected thereon, wherein the memory stores computer-executable instructions, and when the computer-executable instructions are executed by the at least one processor, the microprocessor is programmed or configured to any of the aforementioned heterogeneous dual-stream fusion methods for grading diabetic retinopathy.

[0045] Furthermore, the present invention also provides a computer-readable storage medium having program code loaded thereon, the program code comprising a set of instructions which, when executed on a computing device, cause the computing device to perform the steps defined in any of the aforementioned heterogeneous dual-stream fusion methods for grading diabetic retinopathy.

[0046] Furthermore, the present invention also provides a computer program product comprising program instructions stored on a computer-readable medium, which, when loaded and executed by a processor, are intended to implement any of the aforementioned heterogeneous dual-stream fusion methods for grading diabetic retinopathy.

[0047] Compared with existing technologies, this invention has the following main advantages: it can automatically obtain accurate grading results for diabetic lesions by comprehensively analyzing the input fundus images. Specifically, this invention first processes the fundus images input by the device into different resolutions and inputs them into two parallel feature extraction branches: the global feature extraction branch designs a composite distillation scheme, using a lightweight visual Transformer model that is significantly compressed using techniques such as feature distillation and contrastive learning distillation, reducing the number of parameters of the feature extractor from 3.7G to 327MB, which is only 8.8% of its original size. This efficiently captures global contextual information such as retinal vessel trends and lesion distribution on low-resolution images with low computational cost; the local lesion feature extraction branch uses the convolutional neural network EfficientNet-b5 to accurately focus on key local lesion details such as microaneurysms and hemorrhages on high-resolution images, capturing local spatial patterns and hierarchical features in the image. Traditional hybrid models use simple feature splicing or element-wise addition fusion methods, which easily ignore deep heterogeneous relationships between features. In contrast, this method introduces a symmetrical bidirectional cross-attention fusion module. The symmetrical bidirectional design of this module ensures adaptive deep interaction between the global features extracted by ViT and the local features extracted by CNN. This allows global features to actively "query" and absorb the most relevant lesion details from local features, while also enabling local lesion features to "perceive" their context within the global retinal structure. This overcomes the performance bottleneck of simple addition or splicing, generating highly complementary and representative fusion features that balance global contextual information with high-resolution local information. Therefore, this invention effectively solves the technical challenges of single-model representation being one-sided (e.g., CNN lacking a global perspective), the high clinical deployment threshold of large models (e.g., the excessive computational cost of large ViT models), and the insufficient handling of heterogeneous features by existing fusion methods. While ensuring or even surpassing the accuracy of large models in grading diabetic retinopathy, this invention significantly reduces the number of model parameters and computational complexity, making it easily deployable in resource-constrained medical devices or primary healthcare institutions to improve the accuracy of diabetic retinopathy diagnosis. It can be applied to fields such as ocular image analysis and precision medicine. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention;

[0049] Figure 2 This is an algorithm framework diagram of a heterogeneous two-stream fusion method for grading diabetic retinopathy provided in an embodiment of the present invention;

[0050] Figure 3 This is a framework diagram of the composite knowledge distillation scheme in an embodiment of the present invention;

[0051] Figure 4 This is a diagram showing the internal structure of the symmetrical bidirectional cross-attention fusion module in an embodiment of the present invention.

[0052] Figure 5 This is a comparison of model performance on the MESSIDOR-2 dataset in the embodiments of the present invention;

[0053] Figure 6 This is a comparison of model performance on the IDRID dataset in the embodiments of the present invention;

[0054] Figure 7 This is the ROC curve for multi-class classification on the MESSIDOR-2 dataset in an embodiment of the present invention;

[0055] Figure 8 This is the ROC curve for multi-class classification on the IDRID dataset in an embodiment of the present invention;

[0056] Figure 9 This is a diagram illustrating the heatmap effect in an actual sample according to an embodiment of the present invention.

[0057] Figure 10 This is a heatmap illustration of the ViT branch in an actual sample in an embodiment of the present invention;

[0058] Figure 11 This is a heatmap illustration of the CNN branch effect in actual samples in an embodiment of the present invention. Detailed Implementation

[0059] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. The described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0060] The core of this invention is to provide a heterogeneous dual-stream fusion method for grading diabetic retinopathy, in order to solve the problems existing in the prior art.

[0061] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 This is a basic flowchart illustrating a heterogeneous two-stream fusion method for grading diabetic retinopathy provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes the following steps:

[0063] S1: Receives the input image and performs parallel resizing to generate images at different resolutions. Specifically, the 224×224 pixel image is fed into the ViT branch, while the 456×456 pixel image is fed into the CNN branch.

[0064] S2: Two branches work in parallel to extract features from different dimensions. The ViT branch extracts feature vectors representing global contextual information from its input. The CNN branch (EfficientNet) extracts feature vectors representing local texture information from its input.

[0065] S3: The feature vectors of ViT and CNN are fed into the vector cross-attention module. This module implements bidirectional information exchange: ViT features act as queries to extract information from CNN features, and CNN features act as queries to extract information from ViT features. Through a standard multi-head attention mechanism and residual connections, deeply fused and enhanced fused features are generated.

[0066] S4: Integrate the two enhanced feature vectors output by the cross-attention module. These two vectors absorb each other's advantageous information from the ViT and CNN features. By directly concatenating them along the feature dimension, a final fused feature with higher information density and stronger expressive power is formed;

[0067] S5: Input the fused feature vector into a multilayer perceptron (MLP) classification head. This classification head maps the high-dimensional features to the final category space through fully connected layers, normalization layers, and activation functions, completing the classification of image content and outputting the final prediction result.

[0068] Figure 2 This invention provides an algorithmic framework diagram for a heterogeneous two-stream fusion method for grading diabetic retinopathy (DR). To effectively address the challenges in DR grading (i.e., subtle lesion identification, global context understanding and multi-scale integration, model efficiency and deployment limitations, and the difficulty of heterogeneous feature fusion), this invention proposes a heterogeneous two-stream fusion method called HDNet. Its core design is its innovative parallel dual-branch heterogeneous architecture. By processing input images at different scales, the heterogeneous branches leverage their efficient feature extraction capabilities, and finally, through deep fusion, achieve the goal of DR grading. This design can solve the problems of accurate identification of subtle lesions, comprehensive understanding of global context, model efficiency and deployability, and dynamic and effective fusion of heterogeneous features, ultimately achieving a more accurate and robust grading of DR severity. The overall framework of HDNet consists of three core components:

[0069] Global Feature Extraction Module: This module uses composite knowledge distillation technology to transfer the knowledge of a large base model to a lightweight ViT encoder for processing low-resolution 224×224 images. It efficiently captures global contextual information while significantly reducing computational costs and the number of parameters to meet the needs of efficient and lightweight deployment.

[0070] Lesion Focusing Module: This branch employs a CNN encoder (EfficientNet-B5) pre-trained on ImageNet. It is used to process high-resolution 456×456 images, leveraging the inherent local inductive bias of CNNs to extract fine local textures and spatial hierarchy features of minute lesions, effectively compensating for the shortcomings of ViT in local detail analysis.

[0071] Symmetrical Bidirectional Cross-Attention Module (BCAF): This module is designed to achieve adaptive, symmetrical bidirectional cross-attention deep interaction and enhancement between two heterogeneous branches (global ViT features and local CNN features). Through this dynamic fusion mechanism, HDNet ensures that it can comprehensively consider both the microscopic details of DR lesions and the macroscopic structure of retinal images, avoiding information conflicts caused by simple static fusion.

[0072] like Figure 3 As shown, this invention designs a composite distillation scheme. It comprehensively transfers knowledge from the teacher model to a lightweight student model from multiple dimensions, including prediction distribution, feature hierarchy, and data relationships. In the experiment, the RETFound encoder serves as the teacher model, and a ViT-Base model with reduced depth and width serves as the student model. With the teacher model parameters frozen, a composite loss function is used to jointly optimize the student model, specifically including soft target distillation, feature distillation, and contrastive learning distillation. Simply using hard labels for supervision ignores the hidden knowledge inherent in class probability prediction, i.e., the similarity information between classes. This invention uses a temperature coefficient... The Softmax function is used to soften the logits output of the teacher model. By minimizing the KL divergence between the soft objectives of the student and teacher models, the student model is forced to fit the finer distribution of the class probabilities of the teacher model. To guide the student model to simulate a more robust intermediate feature representation of the teacher model, this invention introduces feature distillation. This method prompts the student model to perform feature distillation at a predefined intermediate layer. The feature map of the student model should be matched with the feature map of the teacher model. Considering that the differences in network structure between the student and teacher models may lead to inconsistent feature map dimensions, this invention introduces an adaptation layer φ to align dimensions. Higher-level knowledge is implied in the structural relationships between data samples. To transfer the relational knowledge learned by the teacher model, this invention constructs a contrastive learning distillation loss. Its core idea is that if two samples are similar (positive sample pairs) in the representation space of the teacher model, then they should also be brought closer in the representation space of the student model; otherwise, they should be pushed apart.

[0073] like Figure 4 As shown, this invention designs a symmetrical bidirectional cross-attention fusion module. This module performs cross-attention calculations in two directions in parallel: global-to-local information enhancement and global feature fusion. As a query, from local features The process extracts relevant local details to enrich the global view with fine texture. The attention output generated in this process is added to the original global features via residual connections, then stabilized through a normalization layer, ultimately resulting in an enhanced global feature that absorbs local details. This process can be formally represented as:

[0074] .

[0075] in, Depend on Generate, and and Depend on Linear projection generation;

[0076] Local-to-global information enhancement: Symmetrically, local features As a query, from global features The process involves identifying the context within the lesion to understand its importance from a global perspective. This process also includes residual connections and layer normalization to obtain an enhanced local feature that incorporates global context awareness. This process can be formally represented as:

[0077] .

[0078] in, Depend on Generate, and and Depend on generate;

[0079] After the attention output from the bi-branch approach described above, a unified residual fusion function and layer normalization strategy are applied after each branch. The extracted fusion information is then fused again with the original features through residual connections (i.e., element-wise addition). The fused result is then input into a layer normalization unit for further processing to avoid missing key information. This process ultimately generates features that absorb key local details and fuse global information, ensuring the model's training stability. These two feature vectors, after bi-directional interaction and enhancement, are then... and By splicing together, a fusion feature is formed. The data is then fed into a classifier for decision-making. This process can be formally represented as:

[0080] .

[0081] See Figure 5 and Figure 6 It can be seen that the heterogeneous dual-stream fusion method for grading diabetic retinopathy in this invention is superior to existing diabetic retinopathy diagnostic methods in all aspects.

[0082] To further illustrate the effectiveness of the heterogeneous two-stream fusion method for grading diabetic retinopathy in the examples of this invention, Figure 7 and Figure 8 The ROC curves further demonstrate the model's discriminative ability. Different colored curves represent the performance of different models, while the diagonal dashed line represents the random guessing baseline (AUC=0.5), indicating the model has no classification ability. The visualization shows that the ROC curves representing all categories significantly deviate from the diagonal dashed line and strongly bulge towards the upper left corner of the graph. This clearly indicates that the invention achieves a very high true positive rate while maintaining an extremely low false positive rate when identifying each category. On the MESSIDOR2 dataset, its micro-average ROC area reaches 95%, demonstrating excellent performance across all categories. The macro-average ROC area also reaches 92%. The results demonstrate the model's powerful ability in multi-class classification tasks on the MESSIDOR-2 dataset. On the IDRiD dataset, the micro-average ROC area reaches 88%, demonstrating excellent performance across all categories. The macro-average ROC area also reaches 88%.

[0083] Furthermore, to illustrate the effectiveness of the heterogeneous two-stream fusion method for grading diabetic retinopathy according to the embodiments of the present invention, Figure 9 , Figure 10 , Figure 11Heatmaps are used to explain and interpret the model's decision-making mechanism. By overlaying a color layer onto the original input image, it visually demonstrates which regions in the image the model focuses on during classification decisions. The internal working principle of the proposed dual-branch parallel model is analyzed in depth. Figure 10 and Figure 11 The activation heatmaps of the ViT branch and the CNN branch in the model are shown separately when they operate independently. The figures reveal that the decision-making processes of the two independent branches each have their own emphasis but also significant limitations: the ViT branch ( Figure 10 CNNs, with their self-attention mechanism, demonstrate the ability to capture global context and long-range dependencies. Their activation regions are typically more macroscopic and complete, leaning more towards exploring global information. (CNN branch) Figure 11 The focus is more on the local texture and edge details of the image, and it can accurately capture some subtle features, such as the distribution of retinal blood vessels and local information of the eyeball.

[0084] Figure 9 The heatmap effect of the method shown in the right image illustrates that after integrating the information from both methods, it absorbs the precise localization capability of CNN for key local details and integrates the macroscopic grasp of overall structure and context of ViT. Ultimately, the hotspot region is accurately located on the most distinctive core target, making the identification of lesion areas more targeted. This intuitive comparison powerfully demonstrates that the fusion architecture proposed in this invention is not a simple feature concatenation, but rather, through effective bidirectional information interaction, it enables the model to learn how to collaboratively utilize information from different dimensions to form a more comprehensive and accurate decision.

[0085] In summary, the heterogeneous two-stream fusion method for grading diabetic retinopathy in this invention processes the input fundus image using a heterogeneous two-stream parallel architecture to obtain accurate grading results for diabetic retinopathy. First, the input fundus image is acquired and processed into images of different resolutions. Then, a lightweight visual Transformer model is used in parallel to extract global contextual features from the low-resolution image, and a convolutional neural network, EfficientNet-B5, is used to extract local lesion features from the high-resolution image. A symmetrical bidirectional cross-attention fusion module deeply fuses the global contextual features and local lesion features of the two-branch architecture to obtain an enhanced fused feature representation. Finally, this fused feature representation is input into a classifier to output the severity grading result of diabetic retinopathy. This invention aims to improve the accuracy and robustness of grading diagnosis through deep collaborative analysis of global information and local details, and can be applied to fields such as clinical computer-aided diagnosis.

[0086] Furthermore, the present invention also provides a heterogeneous two-stream fusion method for grading diabetic retinopathy, comprising a processor and a memory interconnected thereon, wherein the memory stores computer-executable instructions, and when the computer-executable instructions are executed by the at least one processor, the microprocessor is programmed or configured to any of the aforementioned heterogeneous two-stream fusion methods for grading diabetic retinopathy.

[0087] Furthermore, the present invention also provides a computer-readable storage medium having program code loaded thereon, the program code comprising a set of instructions which, when executed on a computing device, cause the computing device to perform the steps defined in any of the aforementioned heterogeneous dual-stream fusion methods for grading diabetic retinopathy.

[0088] Furthermore, the present invention also provides a computer program product comprising program instructions stored on a computer-readable medium, which, when loaded and executed by a processor, are intended to implement any of the aforementioned heterogeneous dual-stream fusion methods for grading diabetic retinopathy.

[0089] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus (systems), or computer program products. Therefore, the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of computer program products embodied on one or more computer-readable storage media (including but not limited to disk storage, optical storage, etc.) containing computer-executable program code. The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each flowchart illustration and / or block, and combinations of flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation for the process. Figure 1 One or more processes and / or frames Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or frames Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or frames Figure 1 The steps of the function specified in one or more boxes.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A heterogeneous dual-stream fusion method for grading diabetic retinopathy, characterized in that, This involves using a pre-trained neural network model and a lightweight feature extractor obtained through a composite knowledge distillation scheme to process a large amount of eye image information. A symmetrical bidirectional cross-attention fusion module then fuses the outputs of these two different model architectures to obtain the diagnostic results for retinal disease grading. The heterogeneous dual-stream fusion model generates retinal disease grading diagnostic results through the following steps: S1: Constructing a heterogeneous dual-stream feature extraction model: The model includes a parallel CNN branch and a ViT branch; wherein, the CNN branch is configured to receive a high-resolution eye image of 456×456 pixels and extract small local lesion features of diabetic retinopathy using the local inductive bias of CNN; the ViT branch is configured to receive a low-resolution eye image of 224×224 pixels and extract global contextual features of the retinal image using a self-attention mechanism; S2: Model Training Based on Composite Knowledge Distillation: The feature extractor of the ViT branch is obtained through training using a composite knowledge distillation scheme. The specific training process includes: using the large-scale base model RETFound as the teacher model, and jointly optimizing the student model through a composite loss function while freezing the teacher model parameters; the composite loss function includes: a soft-target distillation loss constructed using a Softmax function with a temperature coefficient, used to fit the distribution of the teacher model's class probabilities and capture the hidden knowledge between classes; a feature distillation loss with dimension alignment introduced into the adaptation layer, so that the feature map of the student model matches the teacher model in the preset intermediate layer; a supervised contrastive learning distillation loss constructed based on positive and negative sample pairs, used to transfer the teacher model's understanding of the structural relationships between data samples, where similar samples are brought closer in the student model's representation space, and dissimilar samples are pushed away; finally, the above soft-target distillation loss, feature distillation loss, supervised contrastive learning distillation loss, and standard cross-entropy loss are weighted and combined to optimize the model; S3: Feature fusion based on symmetric bidirectional cross-attention: Utilizing the BCFA module to fuse the global features output by the ViT branch. Local features of CNN branch output Perform deep fusion; before performing cross-attention calculation, first combine local features. and global features Projected onto the same common dimension; The fusion process specifically includes two enhancement paths executed in parallel: (a) Global to Local Enhancement Path: Projecting local features from CNN branches onto the same common dimension; As a query, the global features of the ViT branch are used. The attention output is computed using the key and value; this attention output is then compared with the original local features. Element-wise summation is performed through residual connections, and the summed result is then processed by a LayerNorm to obtain enhanced local features. (b) Local-to-Global Enhancement Path: Symmetrically, the global features of the ViT branch are enhanced. As a query, local features of the CNN branch are used. The attention output is computed as both key and value; this attention output is then compared with the original global features. Element-wise summation is performed through residual connections, and the summed result is then processed by a LayerNorm to obtain enhanced global features. ; S4: Classification Output: The enhanced local features are then... and enhance global features The features are spliced ​​together to form a fused feature, which is then fed into a classification head. The classification head maps the high-dimensional features to the category space through a multilayer perceptron and outputs the classification result of diabetic retinopathy.

2. The heterogeneous dual-stream fusion method for grading diabetic retinopathy according to claim 1, characterized in that, In step S1: The CNN branch uses an EfficientNet-B5 encoder pre-trained on ImageNet; after the image is encoded by the convolutional layer of EfficientNet-B5 and processed by the MBConv block, the final feature map before the global average pooling layer is taken and flattened to obtain the local feature vector.

3. The heterogeneous dual-stream fusion method for grading diabetic retinopathy according to claim 1, characterized in that, In step S4: the multilayer perceptron classification head contains two ReLU activated hidden layers with dimensions of 512 and 256 respectively; its function is to learn... Non-linear mapping between category labels; Output an M-dimensional original score vector. The Softmax function is used to transform z into a class probability distribution. The entire model is trained in an end-to-end manner, and its optimization objective is to minimize the predicted probability distribution. With real labels Classification cross-entropy loss ; The final diagnosis of diabetic retinopathy is obtained by converting the data into a class probability distribution using the Softmax function.

4. A heterogeneous dual-stream fusion system for grading diabetic retinopathy, comprising an interconnected microprocessor and a memory, characterized in that, The microprocessor is programmed or configured to perform the heterogeneous dual-stream fusion method for grading diabetic retinopathy as described in any one of claims 1 to 3.

5. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the heterogeneous dual-stream fusion method for grading diabetic retinopathy as described in any one of claims 1 to 3.

6. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute, via a processor, the heterogeneous dual-stream fusion method for grading diabetic retinopathy as described in any one of claims 1 to 3.

Images