A multi-disease classification method for color fundus images
By combining data preprocessing with collaborative training of a three-branch deep learning network, the problems of class imbalance, multiple lesion features, and image scale in color fundus image classification were solved, achieving high accuracy and robust multi-disease classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QUZHOU PEOPLES HOSPITAL (QUZHOU CENT HOSPITAL)
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-29
AI Technical Summary
Existing color fundus image classification technologies suffer from problems such as class imbalance, coexistence of multiple lesion features, image scale diversity, and insufficient generalization ability, resulting in low classification accuracy and significant performance fluctuations when applied across domains.
We employ data preprocessing techniques such as sample equalization, image enhancement, and black border removal to construct a three-branch deep learning network. Through a multi-head bidirectional cross-attention mechanism and a multi-scale fusion module, we combine classification loss, consistency loss, and contrastive learning loss for collaborative training, and introduce learnable dynamic weights for weighted fusion.
It significantly improves the classification accuracy and robustness of color fundus images, effectively handles multiple lesion features and image scale differences, and enhances the model's adaptability to different devices and scenarios.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical image analysis, artificial intelligence, and machine vision, and provides a method for classifying multiple diseases using color fundus images. Background Technology
[0002] The fundus is the only part of the body where arteries, veins, and capillaries can be directly observed non-invasively, reflecting many diseases such as diabetic retinopathy, glaucoma, hypertension, and pathological myopia. Therefore, fundus examination can accurately diagnose fundus diseases and certain systemic diseases, making it one of the most basic and convenient medical examination methods. Color fundus photography is the primary and indispensable examination tool in the diagnosis and treatment of fundus diseases, serving as a crucial basis for doctors' diagnosis and differential diagnosis. It can visually display fundus signs such as hemorrhage, exudation, pigmentation, proliferation, atrophy, degeneration, and eye tears. Based on abnormal fundus changes and medical history, doctors can make a comprehensive judgment, arrive at a preliminary or final diagnosis, and propose a treatment plan or further examinations.
[0003] In recent years, with the rapid development of deep neural network technology, deep learning-based color fundus image classification has made significant progress in the field of assisted diagnosis. Typical methods often employ convolutional neural networks (CNNs) to extract global and local features, and further improve classification performance through attention mechanisms or multimodal fusion strategies. However, existing technologies still face multiple challenges in practical applications.
[0004] 1. Class imbalance: In most public and clinical fundus image datasets, the number of normal retina or common lesion samples far exceeds that of rare lesions (such as choroidal neovascularization). If no balancing is performed, the model will easily overfit the majority class and ignore the minority class.
[0005] 2. Multiple lesion images: Multiple lesion features often exist simultaneously in a single fundus image. Traditional single-label classification models are difficult to fully cover all lesions, which can easily lead to missed detections or misdiagnosis.
[0006] 3. Differences in image quality and scale: Due to differences in objective factors such as acquisition equipment, lighting and patient condition, images vary significantly in clarity, contrast and field of view; lesions of different diseases vary in size and shape, and convolution operations with a single receptive field cannot take into account both global and detailed information.
[0007] 4. Insufficient generalization ability: Clinical data comes from diverse sources, and the distribution of images collected by different hospitals or devices varies greatly, which leads to significant performance fluctuations of existing models when applied across domains, limiting their practical value.
[0008] Given the importance of color fundus images in ophthalmology, and the current technical challenges and clinical needs in the field of color fundus image classification, this invention aims to provide an innovative and efficient solution. Utilizing advanced deep learning technology, this invention can automatically and accurately perform color fundus image classification tasks, providing strong technical support for the early diagnosis, progression monitoring, and treatment efficacy evaluation of fundus diseases. This method not only significantly reduces the workload of professionals but also improves segmentation accuracy and processing speed, thereby enhancing the efficiency and quality of overall medical services. Summary of the Invention
[0009] The purpose of this invention is to solve the technical problems of low classification accuracy and weak generalization ability caused by the imbalance of categories in color fundus images, the coexistence of multiple lesion features, and the diversity of image scales.
[0010] To achieve the above objectives, the present invention employs the following technical means:
[0011] This invention provides a method for classifying multiple diseases using color fundus images, comprising the following steps:
[0012] Step 1: Data preprocessing: Perform sample equalization on the original color fundus image, remove the black borders around the image, enhance the image using contrast-limited adaptive histogram equalization, and finally scale the image to a preset size.
[0013] Step 2, Data Augmentation: For each image obtained after Step 1, perform data augmentation twice randomly to expand the dataset and generate data for augmented branch channel A and augmented branch channel B.
[0014] Step 3, Image Classification: Set up three branch channels: Enhanced Branch A channel, Enhanced Branch B channel, and Fusion Branch channel. The data generated in Step 2 is fed into the shared backbone network ResNet101 in parallel. Enhanced Branch A and Enhanced Branch B obtain multi-level features in the preset layers of the backbone network. Through the multi-head bidirectional cross-attention mechanism of the feature interaction fusion module, the two features of Enhanced Branch A and Enhanced Branch B are deeply interacted to obtain the interacted features. In Enhanced Branch A / B channels, the interacted features are used as the input of the next level. In the Fusion Branch channel, the interacted features are concatenated, and the dimensionality is reduced by 1×1 convolution to output the fused features.
[0015] The three branch channels fuse multi-scale features at preset levels through a multi-scale fusion module, each with its own classifier, and output preliminary classification logits.
[0016] The parameters of the entire network are trained in a combined manner using classification loss, consistency loss, and contrastive learning loss.
[0017] Step 4, Model Fine-tuning and Dynamic Fusion: Freeze the backbone network, feature interaction fusion module, and multi-scale fusion module, fine-tune only the classification head, introduce learnable dynamic weights, perform weighted fusion of the preliminary classification logits output by the three branches, and finally output the probability of each disease category, and give hints for high confidence samples.
[0018] In the above scheme, step 1, sample equalization of the original color fundus image includes the following steps: a random undersampling strategy is adopted for the majority class samples, and the ratio of the number of majority class samples to the total number of minority class samples is 1±δ, where δ∈[0,0.3]; in order to alleviate the overfitting caused by undersampling and improve the generalization ability of the model, random undersampling is re-executed at the beginning of each training iteration (epoch) to dynamically generate different training subsets.
[0019] In the above solution, step 1, removing the black borders around the image, includes the following steps:
[0020] The original color fundus image is converted into a grayscale image. Based on a preset grayscale threshold, pixels with grayscale values higher than the threshold are identified as valid retinal regions. The first and last valid pixel boundaries are detected along the vertical and horizontal directions on the grayscale image. The original image is then cropped based on these boundaries, and the surrounding black occlusion areas are removed.
[0021] In the above scheme, step 3 involves using a dual-channel structure including enhanced branch A channel and enhanced branch B channel to extract features from the data output in step 2, employing a feature interaction fusion module to perform feature interaction, and fusing the interacted features, including the following steps:
[0022] A dual-channel structure is introduced into the backbone of the network, which shares the input parameters of two different enhanced versions of the original image in parallel to extract multi-level features.
[0023] At multiple preset levels, a multi-head bidirectional cross-attention mechanism is applied to the feature maps corresponding to two channels to achieve deep semantic interaction between enhanced features. In the enhancement branch A / B channels, the interactive features are used as input to the next level; in the fusion branch channels, the interactive features are concatenated and dimensionality is reduced through a 1×1 convolution operation to obtain the fused features. The specific calculation formula for the multi-head bidirectional cross-attention is as follows:
[0024]
[0025]
[0026] Represents the number of attention heads, This represents the total dimension of the input features, and A and B represent the channels. Indicates the first The calculation results of each attention head, This represents the output projection matrix, used to map the concatenated multi-head attention results to the target dimension. The query matrix for channel A is in the th... The projections in each attention head are used to calculate the similarity with the key matrix. The key matrix of channel B represents the first channel. The projection in each attention head is used to calculate the dot product with the query matrix. The value matrix representing channel B is in the th... The projection in each attention head stores the actual feature information. Indicates transpose. Indicates the scaling factor. This represents the dimension of the key vector.
[0027] In the above scheme, step 3 involves fusing multi-scale features through a multi-scale fusion module and then feeding them into the classifier. This includes the following steps:
[0028] For the interactive feature maps output by layers 2, 3, and 4 of the dual-channel backbone network ResNet101, and the fused features obtained by fusing branch channels, the feature size is... ,in Each feature map after interaction at each scale is projected onto a unified d=256-dimensional channel space using 1×1 convolution, and global average pooling is performed to obtain d-dimensional feature vectors. The three branch channels are concatenated by a multi-scale fusion module, and then further reduced to N dimensions by 1×1 convolution. Each channel is set with a classifier to output preliminary classification logits, where N represents the number of disease types.
[0029] In the above scheme, step 3 involves setting up a classification head on each of the three branch channels, and then collaboratively training the model using classification loss, consistency loss, and contrastive learning loss. The specific formula is as follows:
[0030]
[0031] in, The logits represent the output of different branch categories. It is a real label , It is the result after Sigmoid. , For branches A and B, BCE is the multi-label binary cross-entropy loss, and JS is the Jensen-Shannon divergence. For NT-Xent loss, , These are the weighting coefficients.
[0032] In the above scheme, step 4 involves fine-tuning the network and weighted fusion of the classifiers, including:
[0033]
[0034]
[0035] In this system, GAP is a global pooling operation, MLP is a two-layer fully connected network that learns and merges weights, and finally outputs a weighted result. To integrate the characteristics of the branch channel output, Enhance the dynamic weighting coefficient of branch A channel. Enhance the dynamic weighting coefficient of branch B channel. This represents the dynamic weighting coefficient of the fusion branch.
[0036] Because the present invention employs the above-mentioned technical means, it has the following beneficial effects:
[0037] 1. This invention addresses the technical problems of model overfitting to the majority class due to class imbalance and loss of lesion information caused by black borders around the image by employing a sample balancing strategy (using random undersampling and dynamically generating training subsets), a black border removal algorithm (based on grayscale thresholding to identify effective retinal regions), and contrast-limited adaptive histogram equalization image enhancement techniques implemented in step 1. These techniques significantly improve the utilization rate of effective information and the identifiability of subtle lesions in the original color fundus image by ensuring uniform distribution of samples across disease categories, removing invalid background interference, and enhancing image contrast. Specifically, the sample balancing strategy dynamically adjusts the number of majority class samples to a proportion comparable to that of the minority class (1±δ, δ∈[0,0.3]) in each training iteration, effectively mitigating the risk of overfitting caused by undersampling; the black border removal algorithm accurately locates the retinal region boundary through grayscale image thresholding, avoiding invalid background interference caused by device acquisition; and CLAHE enhancement optimizes the image contrast distribution, enabling more significant visual representation of subtle lesion features (such as exudates and hemorrhages) during the preprocessing stage. The synergistic effect of the above-mentioned techniques lays a high-quality input foundation for subsequent deep feature extraction, ensuring that the network can effectively capture key pathological information.
[0038] 2. This invention addresses the technical problems of insufficient generalization ability and performance fluctuations caused by prediction conflicts between different feature streams under cross-domain data distribution differences by employing the three-branch channel collaborative training mechanism (including a multi-task optimization strategy of classification loss, consistency loss, and contrastive learning loss) constructed in step 3 and the weighted fusion technique using learnable dynamic weight coefficients introduced in step 4. This technique effectively suppresses the model's sensitivity to noise and distribution drift by collaboratively optimizing classification consistency between branches, enhancing feature discriminative power, and achieving adaptive weight allocation based on sample confidence. Specifically, the three branches (enhanced branches A / B and the fusion branch) share a backbone network and achieve deep semantic interaction between dual-channel enhanced data through a multi-head bidirectional cross-attention mechanism, enabling different lesion feature streams to complement and learn from each other. The classification loss ensures basic prediction accuracy, the consistency loss forces the output distribution between branches to converge to eliminate conflicts, and the contrastive learning loss strengthens the discriminative boundary of the feature space. During the fine-tuning phase, the dynamic weight coefficients are adaptively adjusted by the MLP network based on the confidence of each branch to achieve the fusion ratio, thus avoiding the sensitivity of fixed weighting strategies to high-noise samples or cross-domain data. This mechanism not only improves the robustness of the model to differences in image scale and changes in acquisition equipment, but also ensures that the final classification results have reliable consistency and interpretability in clinical scenarios. Attached Figure Description
[0039] Figure 1 This is a basic flowchart of the present invention;
[0040] Figure 2 This is a flowchart of the preprocessing of color fundus images involved in this invention;
[0041] Figure 3 This is a multi-disease classification network for color fundus images involved in this invention;
[0042] Figure 4 This is a structural diagram of the feature interaction fusion module of the classification network involved in this invention;
[0043] Figure 5 This is a structural diagram of the multi-scale fusion module of the classification network involved in this invention;
[0044] Figure 6 This is a schematic diagram illustrating the results of the multi-disease classification method using color fundus images involved in this invention; Detailed Implementation
[0045] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.
[0046] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.
[0047] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0048] The hardware environment used for algorithm testing is: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz GPU NVIDIA TITAN XP, and the runtime environment is Python 3.6 and related extension packages PyTorch.
[0049] The core algorithms of this invention include deep convolutional neural networks, image enhancement algorithms (CLAHE and homomorphic filtering), and feature interaction modules.
[0050] like Figure 1 As shown, the present invention provides a method for multi-disease classification of color fundus images, comprising the following steps:
[0051] Step 1: Perform sample equalization on the original color fundus images to ensure uniform distribution of disease categories; remove black borders around the images; perform image enhancement using contrast-limited adaptive histogram equalization; finally, scale the images to a uniform 224*224 pixels. The processing result is as follows: Figure 2 As shown.
[0052] Sample balancing method: A random undersampling strategy is adopted for the majority class samples, retaining only a portion that is comparable in number to the minority class samples; to alleviate overfitting caused by undersampling and improve the model's generalization ability, random undersampling is re-executed at the beginning of each training iteration (epoch) to dynamically generate different training subsets.
[0053] Black border removal algorithm: Convert the original color fundus image into a grayscale image, and identify the effective retinal areas as pixels with grayscale values higher than the preset grayscale threshold (value=5). On the grayscale image, detect the first and last effective pixel boundaries in the vertical and horizontal directions respectively. Based on the above boundaries, crop the original image and remove the black occlusion areas around it.
[0054] Step 2: For each image obtained after processing in Step 1, perform data augmentation twice randomly to expand the dataset, generating two augmented channels. The data augmentation methods include random rotation and photometric transformation, etc. The processing results are as follows: Figure 2 As shown;
[0055] Step 3: The data from channels A and B are fed in parallel into a shared backbone network, ResNet101, to extract multi-level features. At multiple preset layers (including layer 2, layer 3, and layer 4), a multi-head bidirectional cross-attention mechanism in the feature interaction fusion module is applied to the feature maps corresponding to the two channels to enhance deep semantic interaction between features. Subsequently, the interacted features are concatenated, dimensionality is reduced through 1×1 convolution, and fused features are output to construct a fusion branch. The three branches fuse multi-scale features through a multi-scale fusion module, each setting its own classifier to output preliminary classification logits. The network structure is as follows: Figure 3-5 As shown. The loss function includes classification loss, consistency loss, and contrastive learning loss.
[0056] Multi-head cross-attention mechanism: The specific formula is as follows:
[0057]
[0058]
[0059] Represents the number of attention heads, This represents the total dimension of the input features, and A and B represent the channels. Indicates the first The calculation results of each attention head, This represents the output projection matrix, used to map the concatenated multi-head attention results to the target dimension. The query matrix for channel A is in the th... The projections in each attention head are used to calculate the similarity with the key matrix. The key matrix of channel B represents the first channel. The projection in each attention head is used to calculate the dot product with the query matrix. The value matrix representing channel B is in the th... The projection in each attention head stores the actual feature information. Indicates transpose. Indicates the scaling factor. Indicates the dimension of the key vector;
[0060] Multi-scale fusion and classification: This involves considering the feature maps output by layers 2, 3, and 4 of the dual-channel backbone network ResNet101, as well as the fused features obtained from the fusion branch. The feature size is... Where i∈{2,3,4}. A 1×1 convolution is applied to the feature maps at each scale, projecting them onto a unified d=256-dimensional channel space, and global average pooling is performed to obtain d-dimensional feature vectors. Each branch concatenates the multi-scale vectors to obtain a multi-scale fused feature map, which is then further reduced to N dimensions using a 1×1 convolution, where N represents the number of disease types.
[0061] The loss function, specifically the formula, is as follows:
[0062]
[0063] in, Represents the logits of different branch classification outputs in a multi-branch channel. It is a real label , It is the result after Sigmoid. , To enhance the features of branches A and B, BCE is the multi-label binary cross-entropy loss (classification loss), and JS is the Jensen-Shannon divergence (consistency loss). This is the NT-Xent loss (contrastive learning loss). , These are the weighting coefficients.
[0064] Step 5, Model Fine-tuning and Dynamic Fusion: Freeze the backbone network, feature interaction fusion module and multi-scale fusion module, fine-tune only the classification head, introduce learnable dynamic weights, perform weighted fusion of the three logits, and finally output the probability of each disease category, and give hints for high confidence samples.
[0065] The specific implementation algorithm is as follows:
[0066]
[0067]
[0068] In this system, GAP is a global pooling operation, MLP is a two-layer fully connected network that learns and merges weights, and finally outputs a weighted result. To integrate the characteristics of the branch channel output, Enhance the dynamic weighting coefficient of branch A channel. Enhance the dynamic weighting coefficient of branch B channel. This represents the dynamic weighting coefficient of the fusion branch.
[0069] Example 1
[0070] This invention provides a method for classifying multiple diseases using color fundus images, comprising the following steps:
[0071] Step 1: Data preprocessing: Perform sample equalization on the original color fundus image, remove the black borders around the image, enhance the image using contrast-limited adaptive histogram equalization, and finally scale the image to a preset size.
[0072] Step 2, Data Augmentation: For each image obtained after Step 1, perform data augmentation twice randomly to expand the dataset and generate data for augmented branch channel A and augmented branch channel B.
[0073] Step 3, Image Classification: Set up three branch channels: Enhanced Branch A channel, Enhanced Branch B channel, and Fusion Branch channel. Feed the data generated in Step 2 into the shared backbone network ResNet101 in parallel. Enhanced Branch A and Enhanced Branch B obtain multi-level features in the preset layers of the backbone network. Through the multi-head bidirectional cross-attention mechanism of the feature interaction fusion module, the two features of Enhanced Branch A and Enhanced Branch B are deeply interacted to obtain the interacted features. In the Fusion Branch channel, the interacted features are concatenated, and the dimensionality is reduced by 1×1 convolution to output the fused features.
[0074] The three branch channels fuse multi-scale features at preset levels through a multi-scale fusion module, each with its own classifier, and output preliminary classification logits.
[0075] The parameters of the entire network are trained in a combined manner using classification loss, consistency loss, and contrastive learning loss.
[0076] Step 4, Model Fine-tuning and Dynamic Fusion: Freeze the backbone network, feature interaction fusion module, and multi-scale fusion module, fine-tune only the classification head, introduce learnable dynamic weights, perform weighted fusion of the preliminary classification logits output by the three branches, and finally output the probability of each disease category, and give hints for high confidence samples.
[0077] In the above scheme, step 1, sample equalization of the original color fundus image includes the following steps: a random undersampling strategy is adopted for the majority class samples, and the ratio of the number of majority class samples to the total number of minority class samples is 1±δ, where δ∈[0,0.3]; in order to alleviate the overfitting caused by undersampling and improve the generalization ability of the model, random undersampling is re-executed at the beginning of each training iteration (epoch) to dynamically generate different training subsets.
[0078] In the above solution, step 1, removing the black borders around the image, includes the following steps:
[0079] The original color fundus image is converted into a grayscale image. Based on a preset grayscale threshold, pixels with grayscale values higher than the threshold are identified as valid retinal regions. The first and last valid pixel boundaries are detected along the vertical and horizontal directions on the grayscale image. The original image is then cropped based on these boundaries, and the surrounding black occlusion areas are removed.
[0080] In the above scheme, step 3 involves using a dual-channel structure, including enhanced branch A channel and enhanced branch B channel, to perform feature interaction on the data output from step 2, and then fusing the interacted features. This includes the following steps:
[0081] A dual-channel structure is introduced into the backbone of the network, which shares the input parameters of two different enhanced versions of the original image in parallel to extract multi-level features.
[0082] At multiple preset levels, a multi-head bidirectional cross-attention mechanism is applied to the feature maps corresponding to two channels to enhance deep semantic interaction between features. Subsequently, the interacted features are concatenated and dimensionality reduced through a 1×1 convolution operation to obtain fused features. The specific calculation formula for the multi-head bidirectional cross-attention is as follows:
[0083]
[0084]
[0085] Represents the number of attention heads, This represents the total dimension of the input features, and A and B represent the channels. Indicates the first The calculation results of each attention head, This represents the output projection matrix, used to map the concatenated multi-head attention results to the target dimension. The query matrix for channel A is in the th... The projections in each attention head are used to calculate the similarity with the key matrix. The key matrix of channel B represents the first channel. The projection in each attention head is used to calculate the dot product with the query matrix. The value matrix representing channel B is in the th... The projection in each attention head stores the actual feature information. Indicates transpose. Indicates the scaling factor. This represents the dimension of the key vector.
[0086] In the above scheme, step 3 involves fusing multi-scale features through a multi-scale fusion module and then feeding them into the classifier. This includes the following steps:
[0087] For the interactive feature maps output by layers 2, 3, and 4 of the dual-channel backbone network ResNet101, and the fused features obtained by fusing branch channels, the feature size is... ,in Each feature map after interaction at each scale is projected onto a unified d=256-dimensional channel space using 1×1 convolution, and global average pooling is performed to obtain d-dimensional feature vectors. The three branch channels are concatenated by a multi-scale fusion module, and then further reduced to N dimensions by 1×1 convolution. Each channel is set with a classifier to output preliminary classification logits, where N represents the number of disease types.
[0088] In the above scheme, step 3 involves setting up a classification head on each of the three branch channels, and then collaboratively training the model using classification loss, consistency loss, and contrastive learning loss. The specific formula is as follows:
[0089]
[0090] in, The logits represent the output of different branch categories. It is a real label , It is the result after Sigmoid. , For branches A and B, BCE is the multi-label binary cross-entropy loss, and JS is the Jensen-Shannon divergence. For NT-Xent loss, , These are the weighting coefficients.
[0091] In the above scheme, step 4 involves fine-tuning the network and weighted fusion of the classifiers, including:
[0092]
[0093]
[0094] In this system, GAP is a global pooling operation, MLP is a two-layer fully connected network that learns and merges weights, and finally outputs a weighted result. To integrate the characteristics of the branch channel output, Enhance the dynamic weighting coefficient of branch A channel. Enhance the dynamic weighting coefficient of branch B channel. This represents the dynamic weighting coefficient of the fusion branch.
[0095] In summary, the present invention has the following characteristics:
[0096] 1. In terms of image preprocessing, this invention employs sample equalization, black border removal, and CLAHE, which significantly improves the effective information utilization and contrast of the original color fundus image, making subtle lesions easier for the network to capture.
[0097] 2. In terms of feature extraction, this invention uses dual-channel and multi-head bidirectional cross-attention to achieve deep interaction between different enhancement streams, and enhances the model's adaptability to lesions of different sizes through multi-scale feature fusion.
[0098] 3. In terms of robustness improvement, this invention adopts multi-task collaborative training, sets independent classifiers in the two enhancement branches and the fusion branch, and introduces collaborative optimization of classification loss, consistency loss and contrastive learning loss, which effectively suppresses inter-branch conflicts and improves feature discrimination power, while enhancing the robustness of the model to noise and distribution drift.
[0099] 4. A multi-disease classification method for color fundus images is proposed. In clinical use, it can quickly obtain the probability of each disease category based on color fundus images, providing technical support for ophthalmic diagnosis and detection.
Claims
1. A method for classifying multiple diseases using color fundus images, characterized in that, Includes the following steps: Step 1: Data preprocessing: Perform sample equalization on the original color fundus image, remove the black borders around the image, enhance the image using contrast-limited adaptive histogram equalization, and finally scale the image to a preset size. Step 2, Data Augmentation: For each image obtained after Step 1, perform data augmentation twice randomly to expand the dataset and generate data for augmented branch channel A and augmented branch channel B. Step 3, Image Classification: Set up three branch channels: Enhanced Branch A channel, Enhanced Branch B channel, and Fusion Branch channel. The data generated in Step 2 is fed into the shared backbone network ResNet101 in parallel. Enhanced Branch A and Enhanced Branch B obtain multi-level features in the preset layers of the backbone network. Through the multi-head bidirectional cross-attention mechanism of the feature interaction fusion module, the two features of Enhanced Branch A and Enhanced Branch B are deeply interacted to obtain the interacted features. In Enhanced Branch A / B channels, the interacted features are used as the input of the next level. In the Fusion Branch channel, the interacted features are concatenated, and the dimensionality is reduced by 1×1 convolution to output the fused features. The three branch channels fuse multi-scale features at preset levels through a multi-scale fusion module, each with its own classifier, and output preliminary classification logits. The parameters of the entire network are trained in a combined manner using classification loss, consistency loss, and contrastive learning loss. Step 4, Model Fine-tuning and Dynamic Fusion: Freeze the backbone network, feature interaction fusion module, and multi-scale fusion module, fine-tune only the classification head, introduce learnable dynamic weights, perform weighted fusion of the preliminary classification logits output by the three branches, and finally output the probability of each disease category, and give hints for high confidence samples.
2. The method according to claim 1, characterized in that, In step 1, sample equalization is performed on the original color fundus images, including the following steps: a random undersampling strategy is adopted for the majority class samples, and the ratio of the number of majority class samples to the total number of minority class samples is 1±δ, where δ∈[0,0.3]; in order to alleviate the overfitting caused by undersampling and improve the generalization ability of the model, random undersampling is re-executed at the beginning of each training iteration (epoch) to dynamically generate different training subsets.
3. The method according to claim 1, characterized in that, Step 1, removing the black borders around the image, includes the following steps: The original color fundus image is converted into a grayscale image. Based on a preset grayscale threshold, pixels with grayscale values higher than the threshold are identified as valid retinal regions. The first and last valid pixel boundaries are detected along the vertical and horizontal directions on the grayscale image. The original image is then cropped based on these boundaries, and the surrounding black occlusion areas are removed.
4. The method according to claim 1, characterized in that, In step 3, a dual-channel structure including enhanced branch A channel and enhanced branch B channel is used to extract features from the data output in step 2. A feature interaction fusion module is then used to perform feature interaction and fuse the interacted features. This includes the following steps: A dual-channel structure is introduced into the backbone of the network, which shares the input parameters of two different enhanced versions of the original image in parallel to extract multi-level features. At multiple preset levels, a multi-head bidirectional cross-attention mechanism is applied to the feature maps corresponding to the two channels to achieve deep semantic interaction between the enhanced features. The interactive features are used as the input to the next level in the enhanced branch A / B channels. In the fusion branch channel, the interacted features are concatenated and then dimensionality-reduced through a 1×1 convolution operation to obtain the fused features. The specific calculation formula for the multi-head bidirectional cross attention is as follows: Represents the number of attention heads, This represents the total dimension of the input features, and A and B represent the channels. Indicates the first The calculation results of each attention head, This represents the output projection matrix, used to map the concatenated multi-head attention results to the target dimension. The query matrix for channel A is in the th... The projections in each attention head are used to calculate the similarity with the key matrix. The key matrix of channel B represents the first channel. The projection in each attention head is used to calculate the dot product with the query matrix. The value matrix representing channel B is in the th... The projection in each attention head stores the actual feature information. Indicates transpose. Indicates the scaling factor. This represents the dimension of the key vector.
5. The method according to claim 1, characterized in that, In step 3, multi-scale features are fused using a multi-scale fusion module and then fed into the classifier, including the following steps: For the interactive feature maps output by layers 2, 3, and 4 of the dual-channel backbone network ResNet101, and the fused features obtained by fusing branch channels, the feature size is... ,in Each feature map after interaction at each scale is projected onto a unified d=256-dimensional channel space using 1×1 convolution, and global average pooling is performed to obtain d-dimensional feature vectors. The three branch channels are concatenated by a multi-scale fusion module, and then further reduced to N dimensions by 1×1 convolution. Each channel is set with a classifier to output preliminary classification logits, where N represents the number of disease types.
6. The method according to claim 1, characterized in that, In step 3, a classification head is set on each of the three branch channels, and the model is trained collaboratively using classification loss, consistency loss, and contrastive learning loss. The specific formula is as follows: in, The logits represent the output of different branch categories. It is a real label , It is the result after Sigmoid. , For branches A and B, BCE is the multi-label binary cross-entropy loss, and JS is the Jensen-Shannon divergence. For NT-Xent loss, , These are the weighting coefficients.
7. The method according to claim 1, characterized in that, In step 4, the network is fine-tuned, and the classifiers are weighted and fused, including: In this system, GAP is a global pooling operation, MLP is a two-layer fully connected network that learns and merges weights, and finally outputs a weighted result. To integrate the characteristics of the branch channel output, Enhance the dynamic weighting coefficient of branch A channel. Enhance the dynamic weighting coefficient of branch B channel. This represents the dynamic weighting coefficient of the fusion branch.