Eye disease classification method based on multi-mode collaborative comparison fusion network

Abstract

The invention belongs to the technical field of accurate recognition of eye medical images, and particularly provides an eye disease classification method based on a multi-modal collaborative comparison fusion network, and the method comprises the following steps: S1, obtaining fundus color photograph images and OCT image data from a Kaggle public data set, and obtaining marked disease type labels of the fundus color photograph images and the OCT image data; s2, preprocessing the fundus color photo image and OCT image data, and randomly dividing a processed data set into a training set, a test set and a verification set according to a proportion of 7: 2: 1; and S3, carrying out data enhancement on the data in the divided training set by adopting modes of random deformation, overturning, rotation, immediate zooming and random color jitter, and expanding the data set. According to the method, in a modal specific feature extraction part, Swin Transformer V2 and ConvNeXt V2 which are improved according to eye medical image features are adopted as a backbone network, the capturing capacity of fundus color photo and OCT image features is enhanced, an axial attention mechanism is fused in the ConvNeXt V2, and the characterization capacity of an OCT layered structure is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image accurate identification, and particularly relates to an eye disease classification method based on a multi-modal collaborative comparison fusion network. BACKGROUND

[0002] Eye diseases, such as diabetic retinopathy (DR), age-related macular degeneration (AMD), glaucoma, retinal vein occlusion, etc., are the main causes of visual impairment and blindness. Early and accurate diagnosis and treatment are crucial for protecting patients' vision. Fundus color photography and OCT imaging are two commonly used non-invasive examination methods in ophthalmology clinics, which provide complementary eye information. Fundus color photography mainly shows the planar information of the retinal surface, blood vessels, optic disc and macular area, which is convenient for observing the location and morphology of lesions; while OCT provides cross-sectional or three-dimensional layered information of the retina and anterior segment structure, which can clearly show the micro changes and pathological features of each layer structure.

[0003] Currently, ophthalmologists often need to combine these two image information and other clinical data for comprehensive diagnosis. However, relying on manual interpretation is low in efficiency, and is easily affected by the experience and fatigue level of doctors, with the risk of missed or misdiagnosis.

[0004] With the development of deep learning technology, using convolutional neural networks (CNN) and visual Transformers (ViT) and other models to automatically analyze medical images has become a research hotspot. Although single-modal (such as using fundus color photography or using OCT only) deep learning models have improved the diagnosis efficiency to some extent, due to the limitations of single-modal information, it is difficult to fully capture complex pathological features, and its diagnosis accuracy still has room for improvement. Multi-modal fusion methods aim to combine the advantages of different modalities to provide more comprehensive information for diagnosis. Existing multi-modal fusion methods include early fusion (simple concatenation at pixel or feature level), late fusion (result merging at decision level), etc. These methods often have difficulty in effectively handling the differences and potential misalignment problems in different modal feature spaces, especially for image modalities with large structural and representational differences (such as the two-dimensional planar view of fundus color photography and the cross-sectional or three-dimensional view of OCT), simple fusion strategies may not be able to fully exploit the complementary information between modalities. Therefore, how to design a deep learning model that can effectively align and integrate different modal features to achieve accurate identification of eye diseases is a challenge currently faced. SUMMARY

[0005] (I) Technical problems solved

[0006] In view of the problems that the existing multi-modal fusion method of eye medical images is difficult to effectively process the inherent differences between modalities, difficult to capture deep collaborative information and realize effective alignment of features, the application provides an eye disease classification method based on a multi-modal collaborative comparison fusion network, aiming to fully mine and utilize the complementary information of fundus color images and OCT images, and significantly improve the diagnosis accuracy of eye diseases.

[0007] (Two) Technical solutions

[0008] In order to achieve the above purpose, the application specifically adopts the following technical solutions:

[0009] An eye disease classification method based on a multi-modal collaborative comparison fusion network comprises the following steps:

[0010] S1. Obtain fundus color image and OCT image data from Kaggle public data set, and obtain the labeled disease type label;

[0011] S2. Preprocess the fundus color image and OCT image data, and divide the processed data set into training set, test set and validation set according to the ratio of 7:2:1;

[0012] S3. Data augmentation is performed on the data in the divided training set by random deformation, flipping, rotation, random scaling and random color jittering, so as to expand the data set;

[0013] S4. Establish an eye disease classification model based on a multi-modal collaborative comparison fusion network;

[0014] S5. Input the preprocessed and data-augmented eye medical image training set and validation set into the eye disease classification model based on the multi-modal collaborative comparison fusion network, and calculate the model loss through the loss function;

[0015] S6. Train the model in S5 and analyze the final data results, and respectively adopt weighted fusion precision, recall rate, cross-modal consistency analysis, cross-modal feature consistency index and disease category embedding discriminative score as evaluation indexes.

[0016] Further, the specific steps of preprocessing in S2 are as follows:

[0017] S2-1. Adopt uniform size adjustment and pixel value normalization to eliminate the influence of imaging equipment, acquisition parameters, patient individual differences and other factors;

[0018] S2-2. Apply CLAHE method to enhance the contrast of OCT image, and highlight the boundary of each layer of retina and lesion details.

[0019] Further, the specific steps of data augmentation in S3 to expand the data set are as follows:

[0020] S3-1. Random horizontal flip, random vertical flip, random plane rotation in the range of -10° to +10°, random scaling in the range of [0.8, 1.2], random translation in the range of ±20 pixels;

[0021] S3-2. Random color jittering of fundus color photos, including fine tuning of brightness, contrast, saturation and hue.

[0022] Further, the specific steps of establishing an eye disease classification model based on a multi-modal collaborative contrast fusion network in S4 are as follows:

[0023] S4-1. Constructing a fundus color photo feature encoder, this path includes multiple stages, each stage containing a Modified SwinTransformer V2 Block layer and a Patch Merging layer, through feature extraction and processing at different levels, gradually extracting image features, this model can more effectively capture local texture and subtle structure information in fundus images;

[0024] S4-2. Constructing an OCT image feature encoder, this path includes multiple stages, each stage containing a Modified ConvNext layer and an axial attention layer, through spatial downsampling of feature maps and expansion of channel dimensions between different stages, forming hierarchical multi-scale feature representation;

[0025] S4-3. Constructing a multi-scale collaborative fusion module, this path includes cross-modal interaction, disease-guided contrast learning and multi-scale feature aggregation, through collaborative fusion process to generate high-dimensional feature representation for disease classification.

[0026] Further, S6 at the end of prediction also includes accurate evaluation indicators to illustrate the advantages of the eye disease classification method based on the multi-modal collaborative contrast fusion network, including precision, overall diagnostic accuracy, disease category embedding discriminability score and cross-modal consistency analysis as evaluation indicators.

[0027] (Three) beneficial effects

[0028] Compared with the prior art, the present application provides an eye disease classification method based on a multi-modal collaborative contrast fusion network, which has the following beneficial effects:

[0029] In the modality-specific feature extraction part, this invention uses SwinTransformer V2 and ConvNeXt V2, which are improved for the characteristics of ocular medical images, as backbone networks to enhance the ability to capture features of fundus color photos and OCT images. An axial attention mechanism is incorporated into ConvNeXt V2 to improve the ability to represent OCT layered structures.

[0030] In the multimodal collaborative fusion part, this invention introduces a multi-scale cross-modal attention mechanism to achieve information exchange and spatial alignment between features of different modalities. More importantly, it designs a disease-guided contrastive learning strategy, which effectively overcomes modal differences and captures clinically significant deep collaborative information by forcing the learning of feature representations that are consistent across modalities and discriminative for different disease categories.

[0031] This invention utilizes an innovative ocular disease classification method based on a multimodal collaborative contrast fusion network. This method fully leverages the complementary advantages of fundus color photography and OCT images to achieve accurate classification of ocular diseases, potentially reducing the probability of missed diagnoses and misdiagnoses, and providing doctors with more reliable diagnostic support. Attached Figure Description

[0032] Figure 1 This is a flowchart of an ocular disease classification method based on a multimodal collaborative contrast fusion network according to the present invention.

[0033] Figure 2 This is a structural diagram of the fundus color photograph feature encoder of the present invention;

[0034] Figure 3 This is a structural diagram of the OCT image feature encoder of the present invention;

[0035] Figure 4 This is a structural diagram of the multi-scale collaborative fusion module of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example

[0038] like Figure 1 As shown, the present invention provides a method for classifying ocular diseases based on a multimodal collaborative contrast fusion network, comprising the following steps:

[0039] S1. Obtain fundus photograph and OCT image data from Kaggle public dataset, and obtain eye disease type label , represents the total number of disease categories;

[0040] S2. Perform necessary standardization preprocessing operations on the obtained original fundus photograph and OCT image data to eliminate unnecessary variability introduced by imaging equipment, acquisition parameters, patient individual differences, etc., and unify the format and value range of the input data,

[0041] The preprocessing operation includes: uniformly adjusting the size of the original image data, using bilinear interpolation method to adjust its spatial resolution to a fixed pixel, ensuring that the CFP and OCT images have the same spatial dimension; normalizing the pixel value of the image, linearly scaling the pixel value to the range of [0, 1]; for the layered structure and contrast characteristics of OCT images, apply the contrast limited adaptive histogram equalization (CLAHE) method to locally enhance the contrast of OCT images, highlight the boundaries of each layer of the retina and the details of the lesions. The preprocessed eye medical images are divided to generate three subsets for model training, verification and testing, and the sample ratio is determined as 7:2:1;

[0042] S3. Perform data augmentation processing on the samples in the training set obtained by division to expand the training data set size and improve the robustness of the model and its adaptability to various image deformations, lighting, and color changes.

[0043] The data augmentation operation is randomly applied to each sample pair at each training round, including: random horizontal flip, random vertical flip, random plane rotation in the range of to , random scaling in the range of [0.8, 1.2], random translation in the range of pixels, and random color jittering (including brightness, contrast, saturation and hue fine tuning) only applied to fundus photographs.

[0044] In this embodiment, the diversified data augmentation strategy effectively simulates the variability of image data in real-world clinical environments, forcing the model to learn more generalizable feature representations;

[0045] S4. Establish an eye disease classification model based on a multi-modal collaborative contrast fusion network, the specific steps are as follows:

[0046] S4-1. First, build a fundus photograph feature encoder (CFP Encoder).

[0047] The encoder is used to extract hierarchical and multi-scale feature representation from 3-channel pre-processed fundus color images. The encoder is built based on Swin Transformer architecture and improved for the characteristics of eye medical images inside.

[0048] In this embodiment, the input of the CFP Encoder is fundus color image The model structure is built by a Modified Patch Embedding layer and multiple stage structures composed of Modified Swin Transformer V2 Blocks and PatchMerging layers. The Modified Swin Transformer V2 Block is the core unit for deep feature extraction. The structure is shown as Figure 2 .

[0049] In this embodiment, the first layer in the CFP Encoder structure is the Modified Patch Embedding layer, which is the first layer of the encoder and is responsible for converting the original image into a serialized feature representation. The invention uses overlapping convolution kernels to realize Patch Embedding, aiming to more effectively capture local texture and subtle structure information in fundus images, and make up for the details that may be lost by traditional non-overlapping Patch division.

[0050] In this embodiment, the Modified Swin Transformer V2 Block is based on the standard Swin Transformer Block structure and integrates a channel attention mechanism. The channel attention mechanism is located after the multi-layer perceptron (MLP) inside the Block, which receives the output feature map of the MLP as input. The channel attention first performs global average pooling (GAP) on the feature map to obtain the global spatial information of each channel; then passes the pooled features through a small fully connected network (MLP); finally, generates the weight of each channel through the Sigmoid activation function , which can be expressed as . Finally, multiply the channel weights with the original feature map in a channel-wise manner, that is to enhance the model's attention to the relevant feature channels of the fundus lesions, dynamically adjust the contribution of different channels, and thus improve the discriminative and expressive ability of the features. The Modified Swin Transformer V2 Block also contains a multi-head self-attention mechanism based on windows and shift windows, an LN layer, and a residual connection to effectively capture the local and non-local dependencies of the image.

[0051] The image feature extraction path performs spatial downsampling of the feature maps and expansion of the channel dimension between different stages through the Patch Merging layer, thereby forming a hierarchical multi-scale feature representation. In this embodiment, the CFP Encode contains multiple such stages (e.g., Stage 1, Stage 2, Stage 3, etc.), which gradually extract image features. Stage 1 is composed of several Modified Swin Transformer V2 Blocks, and its output is connected to the input of Stage 2 after passing through the Patch Merging layer. Stage 2 is also composed of several Modified Swin Transformer V2 Blocks, and is again connected to the subsequent stage through the Patch Merging layer. The encoder outputs feature maps of different scales at the end of a specific stage, denoted as These multi-scale features are used in the subsequent multi-modal collaborative fusion module.

[0052] S4-2. Next, an OCT image feature encoder (OCT Encoder) is constructed. The encoder is used to extract hierarchical and multi-scale feature representations from the preprocessed OCT image of 1 channel and enhance its ability to capture the unique layered structure of the OCT image. The encoder is based on the Modified ConvNeXt architecture, and the number of blocks in each stage is set corresponding to the CFP Encoder. In this embodiment, the input of the OCT Encoder is the OCT image The Stem layer, as the first layer of the encoder, converts the original OCT image into an initial feature representation through a two-dimensional convolution operation and connects a Layer Normalization layer as the input of Stage 1. The structure is shown in Figure 3 .

[0053] ​​In this embodiment, the Stem layer is the first layer of the encoder in the OCT Encoder structure, responsible for converting the single-channel raw OCT image into an initial feature map with a certain number of channels, providing input for the subsequent deep feature extraction stage. The Stem layer realizes initial feature conversion through a two-dimensional convolution layer and connects an LN layer, and its output is used as the input feature of Stage 1.

[0054] In this embodiment, the Modified ConvNeXt V2 Block is the basic unit of deep feature extraction in the network. It adds axial attention to the standard ConvNeXt V2 Block structure. Through this method, the model can effectively capture long-distance dependencies along the height direction (corresponding to the depth layered structure of the retina) and the width direction (corresponding to the transverse scan) of the OCT image, which is crucial for understanding and processing the significant layered structure and transverse correlation in the OCT image. The axial attention mechanism is realized by independently applying self-attention to the input features along the height and width axes. The standard multi-head self-attention calculation can be represented as:

[0055]

[0056] where , , represent the query, key and value matrices respectively, is the dimension of the key vector. The axial attention module projects the input feature F to obtain the query, key and value, and then calculates the attention along the height and width axes respectively to obtain the height direction attention calculation result. and the width direction attention calculation result . The final axial attention output is the combination of the two, which is combined by summation in this embodiment:

[0057]

[0058] The module also contains a 7x7 depth separable convolution, a 1x1 Pointwise convolution (for channel dimension transformation), a GELU activation function, a GRN (Global Response Normalization) layer, and a residual connection to effectively integrate local information and enhance the discriminability of features.

[0059] The image feature extraction pathway performs spatial down-sampling of the feature maps and expansion of the channel dimension between different stages through down-sampling layers, thereby forming a hierarchical multi-scale feature representation. In the present embodiment, the OCT Encoder comprises multiple such stages (e.g. Stage 1, Stage 2, Stage 3, etc.) that progressively extract image features. The encoder outputs feature maps of different scales at the end of Stage 1, Stage 2, and Stage 3, denoted as , , These multi-scale features will be used together with the multi-scale features of the fundus color photograph in the subsequent multi-modal collaborative fusion module. The final Stage 4 is composed of down-sampling layers and several Modified ConvNeXt V2 Blocks;

[0060] S4-3. Finally, a multi-scale collaborative fusion module (MSCF Module) is constructed, which is responsible for receiving the hierarchical feature maps ( , ) output by the fundus color photograph feature encoder and the OCT image feature encoder at multiple different scales (s4, s8, s16), and generating a high-dimensional feature representation for disease classification through a collaborative fusion process. The structure of the MSCF Module is shown in Figure 4 ;

[0061] S4-3-1. In the present embodiment, the MSCF Module structure mainly includes cross-modal interaction, disease-guided contrastive learning, and multi-scale feature aggregation. For each selected scale { s4, s8, s16}, first, the learned spatial alignment guide mechanism is used to predict the offset field between the two modalities. Based on this offset field, bidirectional deformable cross-attention calculation is performed, which allows the features of the two modalities to query and align with each other, generating collaboratively enhanced features ( , );

[0062] S4-3-2. A disease-guided contrastive learning mechanism is introduced. Through the disease-guided contrastive projection head, the collaboratively enhanced features at each scale ( , ) are respectively mapped to low-dimensional embedding vectors and ;

[0063] S4-3-3. The collaborative enhanced features of all scales { s4, s8, s16} are integrated through a multi-scale feature aggregation layer to generate a final fixed-length feature vector ;

[0064] S4-3-4. Receiving the final fused feature vector Through a Dropout layer, whose dropout probability is set to 0.5, to prevent overfitting, followed by a fully connected layer with input dimension of 512 and output dimension equal to the total number of disease classes The output of the fully connected layer is converted to a probability distribution over disease classes by a Softmax function ;

[0065] S5. After the data processed by S1 and S2 are input into the MSCF-Net established by S3, all trainable parameters of the model are optimized by minimizing the defined total loss function. The total loss function is composed of two parts: the cross-entropy loss of disease classification and the multi-scale disease-guided contrastive loss , whose expression is:

[0066]

[0067] where is a hyperparameter, whose optimal value is determined by experiments, to balance the contribution of the two losses to the total loss. The cross-entropy loss of disease classification is calculated by:

[0068]

[0069] where N is the number of samples in the current training batch, is the true label of sample belonging to class c, is the probability of sample belonging to class c predicted by the model. The multi-scale disease-guided contrastive loss is the weighted sum of the contrastive losses calculated in each scale of the MSCFModule:

[0070]

[0071] where is the weight of each scale contrastive loss, which is set to ; denotes the contrastive loss guided by disease label Y. The is calculated in the form of loss. For the CFP feature embedding of scale s, which is used as an anchor point, its corresponding OCT feature embedding ​The positive samples are the CFP or OCT feature embeddings of the same patient. The disease-guided mechanism is embodied in introducing weights when calculating the similarity between anchor points and negative samples :

[0072]

[0073] where, is the anchor point embedding, is the disease label of the anchor point, is the positive sample embedding (from patient a , and vice versa), is the cosine similarity function, is the temperature coefficient, P is the set of positive samples, is the set of negative samples. The disease-guided weight is defined as: when the disease label of the negative sample is different from the disease label of the anchor point a, ; when but k and a come from different patients = 1. Where is a hyperparameter, and > 1, aiming to increase the contribution of negative samples from different disease categories to the loss, forcing the model to push the features of different diseases further apart in the embedding space. In this embodiment, the AdamW optimizer is used to optimize the model parameters, which includes weight decay. The Cosine Annealing learning rate scheduling strategy is used. The batch size is set to 8 to balance the memory usage and gradient stability. The training period is set to 150 rounds, and the early stopping mechanism is configured (the patience value is set to 20 rounds), and the training is stopped when the validation set performance does not improve for 20 consecutive rounds. During the training process, the training loss and validation loss of the model are monitored in real time, and the key performance indicators are recorded;

[0074] S6. According to the prediction results of the eye disease classification method based on the multi-modal collaborative comparison fusion network on the test set and the real eye disease category label, the performance indicators are calculated to comprehensively and objectively verify the effectiveness of the model.

[0075] In this embodiment, in addition to using the commonly used precision and recall, the present application particularly introduces the following three core indicators to deeply evaluate the performance and characteristics of the model: overall diagnostic accuracy, cross-modal feature consistency index, and disease category embedding discriminability score.

[0076] Overall Diagnostic Accuracy: This represents the proportion of samples in the test set where the model correctly predicts the type of eye disease. High accuracy directly reflects the model's effectiveness in comprehensive diagnostic tasks. The formula for calculating it is as follows:

[0077]

[0078] in, This represents the total number of test samples. (⋅) is an indicator function that takes the value 1 when the condition inside the parentheses is true, and takes the value 0 otherwise. Indicates the first The model predicts the category for each sample. Indicates the first The true category of each sample.

[0079] Cross-Modal Feature Consistency Index: This index measures the similarity of feature embeddings from different modalities in the embedding space by calculating the average cosine similarity of feature embedding pairs across all patients in the test set at each scale s∈{s4,s8,s16}. The calculation formula is as follows:

[0080]

[0081] Disease Category Embedding Discriminant Score (DSEDC): This metric is used to evaluate the final fused feature vector learned by the MSCF-Net of this invention. Its effectiveness in distinguishing different categories of eye diseases. The calculation formula is as follows:

[0082]

[0083] in, This represents the total number of disease categories. This indicates the calculation of Euclidean distance.

[0084] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal collaborative contrast fusion network-based eye disease classification method, characterized in that, The method comprises the following steps: S1. Obtain fundus color image and OCT image data from Kaggle public dataset, and obtain the labeled disease type label; S2. Preprocess the fundus color image and OCT image data, and divide the processed data set into a training set, a test set and a validation set according to a 7:2:1 ratio; S3. Perform data augmentation on the data in the divided training set by random deformation, flipping, rotation, random scaling and random color jittering to expand the data set; S4. Establish an eye disease classification model based on a multi-modal collaborative comparison fusion network; S5. Input the preprocessed and data-augmented eye medical image training set and validation set into the eye disease classification model based on the multi-modal collaborative comparison fusion network, and calculate the model loss through a loss function; S6. Train the model in S5 and analyze the final data results, and use weighted fusion precision, recall, cross-modal consistency analysis, cross-modal feature consistency index and disease category embedding discriminative score as evaluation indexes.

2. The eye disease classification method based on the multi-modal collaborative comparison fusion network according to claim 1, characterized in that: The specific steps of preprocessing in S2 are as follows: S2-1. Adopt unified size adjustment and pixel value normalization to eliminate the influence of imaging equipment, acquisition parameters, patient individual differences and other factors; S2-2. Apply CLAHE method to enhance the contrast of OCT image and highlight the boundaries of each layer of retina and lesion details.

3. The eye disease classification method based on the multi-modal collaborative comparison fusion network according to claim 1, characterized in that: The specific steps of data augmentation and data set expansion in S3 are as follows: S3-1. Random horizontal flipping, random vertical flipping, random plane rotation in the range of-10° to +10°, random scaling in the range of [0.8, 1.2], and random translation in the range of ±20 pixels; S3-2. Random color jittering of fundus color image, including fine tuning of brightness, contrast, saturation and hue.

4. The eye disease classification method based on the multi-modal collaborative comparison fusion network according to claim 1, characterized in that: The specific steps of establishing an eye disease classification model based on a multi-modal collaborative comparison fusion network in S4 are as follows: S4-1. Construct a fundus color image feature encoder, which includes multiple stages containing Modified SwinTransformer V2 Block layers and Patch Merging layers, and gradually extract image features through feature extraction and processing at different levels, so that the model can more effectively capture local texture and subtle structure information in fundus images; S4-2. Construct an OCT image feature encoder, which includes multiple stages containing Modified ConvNext layers and axial attention layers, and through spatial downsampling of feature maps and expansion of channel dimensions between different stages, hierarchical multi-scale feature representation is formed; S4-3. Construct a multi-scale collaborative fusion module, which includes cross-modal interaction, disease-guided contrast learning and multi-scale feature aggregation, and generates high-dimensional feature representation for disease classification through collaborative fusion process.

5. The eye disease classification method based on the multi-modal collaborative contrast fusion network according to claim 1, characterized in that: The S6 further comprises an accurate evaluation index at the end of prediction to illustrate the advantages of the eye disease classification method based on the multi-modal collaborative comparison fusion network, including the precision, the overall diagnostic accuracy, the disease category embedding discriminant score and the cross-modal consistency analysis as the evaluation index.

Images