Classification Method for Age-Related Macular Degeneration Based on Hierarchical Fusion of Fundus Bimodal Images

Abstract

The present invention discloses a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images, which relates to the technical field of image segmentation. The method includes training a feature extraction and fusion network using paired CFP images and OCT images to obtain CFP features, OCT features, and fused high-dimensional features; inputting the CFP features and OCT features into a modality discriminator to enhance the attention of the feature extractor to the inter-modal relevant information; inputting the CFP features and OCT features into a multi-modal classifier trained by the fused features, and calculating a modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier. The classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the present invention enables the network to effectively distinguish features of different modalities, reduce the deviation between modalities, and enhance the complementarity of features through a modality discriminator based on adversarial learning.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and specifically to a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images. Background Art

[0002] Age-related Macular Degeneration (AMD) is a common ophthalmic disease. Color Fundus Photography (CFP) and Optical Coherence Tomography (OCT) images are widely used in the diagnosis of ophthalmic diseases. In recent years, significant progress has been made in Deep Learning (DL) image processing technology, which is widely applied to tasks such as classification, segmentation, and registration of medical images. Especially in classification tasks, the research has gradually expanded from single-modal to multi-modal. Different imaging modalities provide different perspectives and information. For example, CFP images show the vascular and pigment changes in the retina, while OCT images reveal the microscopic structural changes in the retinal layers. Some lesions may not be obvious in a specific modality but may be clearly visible in other modalities. By fusing the information of these modalities, the classification model can reduce missed diagnoses and misdiagnoses and improve the diagnostic accuracy.

[0003] Currently, multi-modal DL network frameworks can be divided into five types according to the information fusion steps: input fusion, single-level fusion, hierarchical fusion, attention-based fusion, and output fusion. Input fusion refers to the information fusion performed before the DL backbone network. Single-level fusion uses different DL backbone networks to extract features of different modalities respectively, then performs information fusion, and finally makes a final decision. Hierarchical fusion is an improved form of single-level fusion. This method fuses different-dimensional features extracted by the DL backbone at the network level and can also perform further feature fusion after the DL backbone to learn more complex features. Attention-based fusion uses the attention mechanism, especially the Transformer structure, to extract and combine features in the DL backbone network to enhance the feature expression ability. Output fusion, on the other hand, sends the features of different modalities extracted by the DL backbone into different classifiers respectively, and then integrates these classification results through fusion techniques (such as voting or averaging) to generate the final classification result.

[0004] Although significant success has been achieved in DL-based multi-modal classification methods, the existing methods still have the following deficiencies:

[0005] Input fusion: It is not suitable for processing heterogeneous modal data and cannot fully utilize the complementary information between modalities.

[0006] Single - level fusion: Since information fusion only occurs at the end of the network architecture, it is unable to effectively jointly analyze low - dimensional features.

[0007] Hierarchical fusion: The registration problem of multi - modal data may affect the fusion of low - dimensional features, thus affecting the classification performance.

[0008] Attention - based fusion: Although Transformer performs well on large datasets, the insufficient number of samples in medical datasets limits its training effect.

[0009] Output fusion: Lack of feature fusion makes it difficult to fully utilize the complementary information between modalities. If the classification performance of different modalities varies greatly, output fusion may not significantly improve the classification performance. Summary of the Invention

[0010] In view of the above - mentioned problems, the present invention is proposed.

[0011] Therefore, the technical problem to be solved by the present invention is: The multi - modal classification method based on DL has problems such as the inability to fully utilize the complementary information between modalities, the inability to effectively jointly analyze low - dimensional features, being limited by the number of samples, poor training effect, and how to fully utilize the complementary information between modalities to significantly improve the classification performance.

[0012] To solve the above - mentioned technical problems, the present invention provides the following technical solution: A method for classifying age - related macular degeneration based on hierarchical fusion of fundus bimodal images, including training a feature extraction and fusion network using paired CFP images and OCT images to obtain CFP features, OCT features, and fused high - dimensional features; inputting the CFP features and OCT features into a modality discriminator to enhance the attention of the feature extractor to the relevant information between modalities; inputting the CFP features and OCT features into a multi - modal classifier trained by fused features, and calculating the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier.

[0013] As a preferred embodiment of the method for classifying age - related macular degeneration based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: The training of the feature extraction and fusion network using paired CFP images and OCT images includes performing 4×4 convolutional downsampling on the CFP and OCT images by adopting a parallel structure of the feature extraction and fusion network. The processed CFP and OCT images respectively pass through their own feature extraction channels and undergo four - stage feature extraction and fusion operations. In the first stage, the CFP and OCT images respectively pass through two feature extraction modules to obtain CFP image features C1 ∈ R 56×56×96 and OCT image features O1 ∈ R 56×56×96 , perform feature fusion to obtain the fused feature F1 ∈ R 56×56×96 ;

[0014] In the second to fourth stages, the CFP feature extraction channel downsamples by image block merging, while the OCT feature extraction channel downsamples through 2×2 convolution;

[0015] In the second stage, the downsampled C1 and O1 respectively pass through two feature extraction modules to obtain the CFP feature C2 ∈ R 28×28×192 and the OCT feature O2 ∈ R 28×28×192 , and the features in the second stage are processed by a feature fusion module to obtain the fused feature F2 ∈ R 28×28×192 ;

[0016] In the third stage, 12 feature extraction modules are used to process the downsampled C2 and O2 respectively to obtain the feature C3 ∈ R 14 ×14×384 and O3 ∈ R 14×14×384 , and they are fused through a feature fusion module to obtain the fused feature F3 ∈ R 14×14×384 ;

[0017] In the fourth stage, two feature extraction modules are used to process the downsampled C3 and O3 to obtain C4 ∈ R 7×7×768 and O4 ∈ R 7×7×768 , and they are fused through a feature fusion module to obtain the fused feature F4 ∈ R 7×7×768 , and the category attention module is used to process F4.

[0018] As a preferred scheme of the method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: inputting the CFP feature and the OCT feature into the modality discriminator includes processing the input feature map F4, processing it through a 1×1 convolutional layer to generate a feature F' ∈ R H×W×kC , where C is the number of channels required to detect the discriminant regions for each class, k represents the number of classes, obtaining a new feature F” by performing global max pooling on F', and inputting F” into Gaussian similarity clustering to calculate the class scores S = {S1, S2,..., S k}.

[0019] As a preferred embodiment of the method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: inputting the CFP features and OCT features into the modality discriminator further includes converting the input feature map into a feature vector through a global average pooling layer, and the feature vector passes through a gradient reversal layer GRL. During backpropagation, the gradient calculated from the discriminator error is multiplied by a negative scaling factor, the sign of the gradient is reversed, and the magnitude is scaled. The negative gradient is passed to the feature extractor to update the gradient. After the update is completed, the feature vector passes through a fully connected neural network to output the probability distribution of different modality images.

[0020] As a preferred embodiment of the method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: calculating the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier includes using the CFP feature C i , OCT feature O i and the fused feature F i to train three classifiers, namely the CFP classifier, the OCT classifier, and the multi-modal classifier. The output probability distributions of each feature are obtained through the three classifiers, denoted as P C , P O and P M respectively. Then, C i and O i are input into the multi-modal classifier to obtain the output probability distributions P M (C i ) and P M (O i ). Calculate the mean square error loss between P C and P M (C i ) and the mean square error loss between P O and P M (O i ).

[0021] As a preferred embodiment of the method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: calculating the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier further includes that the loss function consists of three parts, including the discriminator loss, the modality consistency loss, and the classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multi-modal classifier and the true label, expressed as:

[0022] L c (P M , CE(P M , y)) = α · (1 - exp(-CE(P M , y)))γ ·CE(P M ,y)

[0023] where CE represents the cross - entropy loss, P M represents the output of the multi - modal classifier, y represents the true label, α represents the class balance factor, and γ represents the focal factor.

[0024] As a preferred solution of the age - related macular degeneration classification method based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: the calculation of the modality consistency loss includes that the total loss function is expressed as:

[0025] L total = L c + L mc + L D .

[0026] Another object of the present invention is to provide an age - related macular degeneration classification system based on hierarchical fusion of fundus bimodal images, which can use the Gaussian similarity clustering method for class attention based on Gaussian similarity clustering to evaluate the similarity between features of different classes, thereby generating weights for specific classes, helping the model to focus more on class features related to the current task, improving the accuracy and robustness of classification, and solving the problem of insufficient accuracy in current DL - based multi - modal classification methods.

[0027] As a preferred solution of the age - related macular degeneration classification system based on hierarchical fusion of fundus bimodal images according to the present invention, wherein: it includes

[0028] A computer device, including a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it realizes the steps of the age - related macular degeneration classification method based on hierarchical fusion of fundus bimodal images.

[0029] A computer - readable storage medium, on which a computer program is stored, and when the computer program is executed by the processor, it realizes the steps of the age - related macular degeneration classification method based on hierarchical fusion of fundus bimodal images.

[0030] Advantages of the present invention: The classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the present invention introduces a category attention module based on Gaussian similarity clustering to enhance the model's attention to task-related category features. Through a modality discriminator based on adversarial learning, the network can effectively distinguish features of different modalities, reduce the deviation between modalities, and enhance the complementarity of features. Through the modality consistency loss, valuable information is extracted from the modality with better performance to improve the performance of other modalities, thereby optimizing the overall classification effect and significantly improving the classification performance of age-related macular degeneration. The present invention achieves better results in terms of accuracy and classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0032] Figure 1 It is the overall flowchart of a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention.

[0033] Figure 2 It is the structural diagram of the feature extraction and fusion network of a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention.

[0034] Figure 3 It is the schematic diagram of the CFP feature extraction module of a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention.

[0035] Figure 4 It is the schematic diagram of the OCT feature extraction module of a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention.

[0036] Figure 5 It is the schematic diagram of the feature fusion module of a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention.

[0037] Figure 6 It is the schematic diagram of the category attention module of a classification system for age-related macular degeneration based on hierarchical fusion of fundus bimodal images provided by the first embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] To make the above objects, features, and advantages of the present invention more apparent and understandable, the following provides a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0039] Example 1, referring to Figures 1-6 , which is an embodiment of the present invention, provides a classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images, including:

[0040] S1: By using paired CFP images and OCT images to train a feature extraction and fusion network, CFP features, OCT features, and fused high-dimensional features are obtained.

[0041] Furthermore, the training of the feature extraction and fusion network by using paired CFP images and OCT images includes performing 4×4 convolutional downsampling on the CFP and OCT images using a parallel structure of the feature extraction and fusion network. The processed CFP and OCT images respectively pass through their respective feature extraction channels and undergo four stages of feature extraction and fusion operations. In the first stage, the CFP and OCT images respectively pass through two feature extraction modules to obtain CFP image features C1 ∈ R 56×56×96 and OCT image features O1 ∈ R 56×56×96 , and feature fusion is performed to obtain the fusion feature F1 ∈ R 56×56×96 ;

[0042] In the second to fourth stages, the CFP feature extraction channel uses image block merging for downsampling, while the OCT feature extraction channel performs downsampling through 2×2 convolution;

[0043] In the second stage, the downsampled C1 and O1 respectively pass through two feature extraction modules to obtain the CFP feature C2 ∈ R 28×28×192 and OCT feature O2 ∈ R 28×28×192 . The second-stage features are processed through a feature fusion module to obtain the second-stage fusion feature F2 ∈ R 28×28×192 ;

[0044] In the third stage, 12 feature extraction modules are used to process the downsampled C2 and O2 respectively to obtain features C3 ∈ R 14 ×14×384 and O3 ∈ R 14×14×384 , and they are fused through a feature fusion module to obtain the third-stage fusion feature F3 ∈ R 14×14×384 ;

[0045] In the fourth stage, the downsampled C3 and O3 are processed by two feature extraction modules to obtain C4 ∈ R 7×7×768 and O4 ∈ R 7×7×768 , and they are fused through a feature fusion module to obtain the fused feature F4 ∈ R 7×7×768 , and the category attention module is used to process F4.

[0046] It should be noted that the overall structure of the CFP feature extraction module is as shown in the appendix Figure 3 . This module introduces the Window Multi-head Self-Attention (W-MSA) and Shift Windows Multi-head Self-Attention (SW-MSA) mechanisms. W-MSA divides the input sequence into multiple fixed-size windows and calculates self-attention within each window; SW-MSA calculates self-attention using a regular window partitioning scheme at the l-th layer and generates new windows through shifted window partitioning at the l + 1-th layer, so that the self-attention calculation in the new windows can cross the window boundaries of the l-th layer, providing connections between windows.

[0047] The input CFP feature is first processed by layer normalization. Then, the feature is sequentially processed by W-MSA, 1×1 convolution, and the GELU activation function. Subsequently, the processed feature is subjected to a residual connection with the original feature F CFP to obtain F' CFP ∈ R H×W×C . The GELU activation function adjusts the output value through a gating mechanism, and its specific formula is shown in Equation (1). Next, after F' CFP is subjected to layer normalization and relative position offset, it is sequentially processed by SW-MSA, 1×1 convolution, and the GELU activation function. Finally, the processed feature is subjected to a residual connection with F' CFP to obtain the final output CFP feature where H, W, and C represent the height, width, and number of channels of the feature map, respectively.

[0048] GELU(x) = xP(x) (1)

[0049] where P(x) is the cumulative distribution function of the Gaussian distribution N(μ, σ2) with a mean of 0 and a variance of 1.

[0050] It should also be noted that the overall structure of the OCT feature extraction module is as shown in the appendix Figure 4 . The input OCT feature First, it undergoes 3×3 depth convolution and layer normalization. Then, 1×1 convolution is used for cross-channel information interaction. Subsequently, the processed features are connected with the original features through residual connection to obtain the final output OCT features

[0051] Furthermore, the overall structure of the feature fusion module is as shown in the appendix Figure 5 The input CFP feature C i ∈R H×W×C and the OCT feature O i ∈R H×W×C are processed by the channel attention module and the self-attention module respectively to obtain and W, H, and C represent the width, height, and number of channels of the feature map. The fused feature of the previous stage is downsampled through 1×1 convolution and average pooling to obtain C i , O i and are concatenated in the channel dimension, and the concatenated features are processed by layer normalization (Layer Normalization, LN) and 1×1 convolution to obtain a comprehensive feature representation Finally, and are concatenated in the channel dimension, processed by the Residual Multi-layer Perceptron (RMLP), and added to the processed features through skip connection to obtain the final fused feature F . The feature fusion operation process is as shown in formula (2). i .

[0052]

[0053] Among them, CA represents channel attention, SA represents self-attention, Avgpool represents average pooling, Concat represents concatenation in the channel dimension, LN represents layer normalization, RMLP represents the Residual Multi-layer Perceptron, and f 1×1 represents 1×1 convolution.

[0054] The operation process of the channel attention module is as follows. First, the input feature F is spatially squeezed through global average pooling to obtain the statistical information z of each channel. Next, an excitation operation is performed on the squeezed channel statistical information to fully capture the dependencies between channels. The excitation operation consists of two fully connected layers and a non-linear activation function ReLU. Specifically, first, the channel dimension is reduced by a dimensionality reduction layer with a reduction ratio r = 16, and then the channel dimension is restored to the dimension of the input feature through a dimensionality increase layer. Finally, the channel weight F is obtained through the Sigmoid activation function calculation. ex The channel weight F ex is multiplied element-wise with the input feature F to obtain the output of the channel attention module. The calculation process is expressed as:

[0055]

[0056]

[0057] where δ is the ReLU activation function, ε is the Sigmoid activation function, W1 and W2 are the weight vectors of the dimensionality reduction layer and the dimensionality increase layer respectively, and p, q, m are the sequence numbers of rows, columns, and channels respectively. denotes element-wise multiplication.

[0058] The operation process of the self-attention module is as follows. First, the input feature map F is passed through three learnable 1×1 convolutions respectively to generate three tensors: query F q ∈R WW×H×C , key F k ∈R WW×H×C , and value F v ∈R WW×H×C . By calculating the dot product of F q and F k , the attention weight map is obtained. Finally, the self-attention feature F SA is obtained by performing weighted summation of the attention weight map and F v , which is expressed as:

[0059]

[0060] RMLP consists of a 3×3 deep convolutional residual layer and two linear transformation layers, where the GELU activation function is applied after the first linear transformation layer to introduce non-linear transformation. This design effectively makes up for the deficiencies of FFN and alleviates the problems of gradient vanishing, gradient explosion, and network degradation to a certain extent, which is expressed as:

[0061] RMLP(x) = f 1×1 (f 1×1 (f depth3×3 (x) + x))

[0062] Among them, f 1×1 represents a 1×1 convolution, and f depth3×3 represents a 3×3 depth convolution.

[0063] S2: Input the CFP features and OCT features into the modality discriminator to enhance the feature extractor's attention to the inter-modal relevant information.

[0064] Furthermore, the inputting of the CFP features and OCT features into the modality discriminator includes processing the input feature map F4 through a 1×1 convolutional layer to generate a feature F′ ∈ R H×W×kC , where C is the number of channels required to detect the discriminant regions for each category, k represents the number of categories, obtaining a new feature F″ from F′ through global max pooling operation, inputting F″ into Gaussian similarity clustering, and calculating the category scores S = {S1, S2, …, S k}, which is expressed as:

[0065]

[0066] where d is the Gaussian similarity metric, σ is a hyperparameter, and w i ∈ R 1×1×kC is a learnable weight vector.

[0067] Perform intra-class cross-channel average pooling operation on F′ to obtain the feature map for each category:

[0068]

[0069] where f′ i，j represents the j-th feature map of the i-th category in F′, and F′ i_avg ∈ R H×W×1 represents the feature map of the i-th category.

[0070] The operation process of the category attention ATT CAB ∈ RH ×W×1 is as follows:

[0071]

[0072] ATT CAB can highlight the regions discriminative for AMD classification. Finally, multiply the input feature map F CAB-IN and the category attention ATT CAB element-wise to obtain the output feature map F CAB-OUT :

[0073]

[0074] where represents element-wise multiplication.

[0075] It should be noted that the operation process of the modality discriminator includes the following steps. First, the input feature map is converted into a feature vector through a global average pooling layer. Then, this feature vector passes through a Gradient Reversal Layer (GRL). The role of GRL in forward propagation is to output the input data as it is without any transformation. In backpropagation, it multiplies the gradient calculated from the discriminator error by a negative scaling factor, reverses the sign of the gradient, and scales its magnitude. Then, the negative gradient is passed to the feature extractor to update its gradient. Next, the feature vector passes through a fully connected neural network, which includes an input layer, a hidden layer consisting of 512 neurons (applying the ReLU activation function), and an output layer (using the softmax activation function for classification). Finally, the discriminator outputs the probability distribution of different modality images.

[0076] Adversarial learning is a max - min game. The feature extractor is trained so that the features it generates are not easily classified correctly by the modality discriminator. The discriminator, on the other hand, aims to maximize its ability to assign correct labels to the two modality features. The discriminator loss is expressed as:

[0077]

[0078] where D represents the discriminator, C4 is the CFP feature in the fourth stage, and O4 is the OCT feature in the fourth stage.

[0079] S3: Input the CFP feature and the OCT feature into the multi - modality classifier trained with the fused feature, and calculate the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier.

[0080] Furthermore, the operation process of the modality consistency loss is as follows: First, use the CFP feature C i , the OCT feature O i and the fused feature F i to train three classifiers, namely the CFP classifier, the OCT classifier, and the multi - modality classifier. Through these three classifiers, the output probability distributions of each feature are obtained, denoted as P C , P O and P M . Then input C i and O i into the multi - modality classifier to obtain the corresponding output probability distributions P M (C i ) and P M (O i ). Calculate P C and P M (Ci ) the Mean Squared Error (MSE) loss between, and P O and P M (O i ) the MSE loss between. The modal consistency loss is expressed as:

[0081]

[0082] where j is the class index and k is the total number of classes.

[0083] It should be noted that calculating the modal consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier also includes that the loss function consists of three parts, including the discriminator loss, the modal consistency loss, and the classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multi-modal classifier and the true label, and is expressed as:

[0084] L c (P M , CE(P M , y)) = α · (1 - exp(-CE(P M , y))) γ ·CE(P M , y)

[0085] where CE represents the cross-entropy loss, P M represents the output of the multi-modal classifier, y represents the true label, α represents the class balance factor, and γ represents the focal factor.

[0086] Example 2, an embodiment of the present invention, provides a method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images. To verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0087] First, the dataset used in this embodiment is a publicly available dataset, including 1094 CFP images and 1289 OCT images of 1093 eyes from 829 subjects. Among them, 817 eyes are equipped with 1 to 5 OCT images. The CFP images are collected by a Topcon fundus camera, while the OCT images are collected by a Topcon OCT camera and a Heidelberg OCT camera respectively, both of which are central B-Scan images and are manually selected by technicians. The classification labels for each eye are obtained by comprehensive evaluation of two ophthalmologists based on CFP images, OCT images, fluorescein angiography, and indocyanine green angiography. The classification of eyes includes normal, dry AMD, wet AMD, or PCV, and all images of the same eye are labeled consistently. During the data processing, the CFP images and OCT images are paired by eye. The paired multimodal samples are then divided into a training set, a validation set, and a test set in a ratio of 7:1.5:1.5.

[0088] The accuracy, precision, recall, F1-score, kappa coefficient, and AUC value are used as evaluation metrics for classification.

[0089] Accuracy: The proportion of the number of correctly classified samples to the total number of samples.

[0090]

[0091] Precision: The proportion of samples actually being positive among the samples predicted as positive by the model.

[0092]

[0093]

[0094] Among them, TP i represents the number of true positive samples in the i-th class; FP i represents the number of false positive samples in the i-th class.

[0095] Recall: The proportion of samples actually being positive among the samples predicted as positive by the model.

[0096]

[0097]

[0098] Among them, FN i represents the number of false negative samples in the i-th class; k is the number of classes.

[0099] F1-score: The harmonic mean of precision and recall, used to comprehensively evaluate the performance of the model, and is applicable to the case of imbalanced class distributions.

[0100]

[0101]

[0102] Kappa coefficient: Used to measure the consistency of a classification model, which takes into account the difference between the proportion of correct predictions by the model and the proportion of random predictions, and can be used to address situations of sample imbalance or a large number of classification categories.

[0103]

[0104] Among them, P o represents the observed accuracy, and P e represents the accuracy of random prediction.

[0105] AUC value: The area under the ROC curve, used to measure the average performance of the model at all thresholds. The larger the AUC value, the better the model performance.

[0106] The present invention conducts corresponding ablation experiments on category attention, modality discriminator, and modality distillation methods. In the comparative experiment, the method of the present invention is compared with other excellent deep learning-based classification networks, including MM-CNN, AM-FAM, MSAN, CRD-Net, and Hifuse, to verify the effectiveness of the method proposed by the present invention.

[0107] Table 1 Ablation experiment results of multimodal age-related macular degeneration classification

[0108]

[0109] Table 1 lists the results of the ablation experiment. Compared with the baseline network, the method of the present invention shows significant improvements in multiple performance indicators. The accuracy, precision, recall, F1 score, Kappa, and AUC values are improved by 4.4%, 3.2%, 2.9%, 3%, 6.5%, and 0.6% respectively compared with the baseline method. The improvements in these indicators all reflect the advantages of the model in classification performance and consistency.

[0110] Table 2 Comparative experiment results of multimodal age-related macular degeneration classification

[0111] Method Accuracy Precision Recall F1 Kappa AUC MSAN 0.670 0.700 0.675 0.662 0.487 0.899 MM-CNN 0.813 0.836 0.845 0.837 0.724 0.949 Hifuse 0.819 0.823 0.812 0.817 0.727 0.945 AM-FAM 0.824 0.820 0.833 0.826 0.739 0.944 CRD-Net 0.852 0.856 0.893 0.871 0.782 0.961 The present invention 0.885 0.875 0.883 0.879 0.827 0.959

[0112] Table 2 lists the results of the comparative experiment. The present invention shows significant improvements in all indicators. The accuracy, precision, F1 score, and Kappa value are all the highest, indicating that the present invention is superior to the existing methods in terms of classification performance and prediction consistency. Although it is slightly inferior to CRD-Net in terms of the AUC indicator, the overall performance is still excellent.

[0113] The present invention proposes an innovative hierarchical fusion method for fundus bimodal images for the automatic classification of AMD. This method processes features of different modalities in a feature extraction and fusion network through four stages, overcoming the deficiencies of existing multi-scale feature fusion methods. Specifically, a feature fusion module combining channel attention and self-attention mechanisms is designed. This module effectively fuses features of different scales and modalities at different levels of the network and suppresses irrelevant or noisy features through the attention mechanism. In addition, a category attention module based on Gaussian similarity clustering is introduced to help the model focus more on category features related to the current task, so as to enhance the model's attention to the lesion area. To reduce the deviation between modalities, the present invention adopts a modality discriminator based on adversarial learning. Finally, through the modality consistency loss, valuable information is extracted from the modality with better performance and used to improve the performance of other modalities, so as to make full use of the information of the existing modalities and improve the overall classification effect. Experiments show that the network proposed by the present invention can overcome the difficult problems in multi-modal image classification, such as the insufficient ability to fuse multi-scale features, the neglect of information related to different modalities, the insufficient utilization of complementary information between modalities, and the insufficient attention to the lesion area. The hierarchical fusion network proposed by the present invention can achieve accurate classification of AMD, exceeding many other advanced classification networks.

[0114] Example 3, an embodiment of the present invention, provides a classification system for age-related macular degeneration based on hierarchical fusion of fundus bimodal images, including a feature extraction module, an information attention enhancement module, and a loss module.

[0115] Among them, the feature extraction module is used to obtain CFP features, OCT features, and fused high-dimensional features by training a feature extraction and fusion network using paired CFP images and OCT images; the information attention enhancement module is used to input the CFP features and OCT features into a modality discriminator to enhance the feature extractor's attention to information related between modalities; the loss module is used to input the CFP features and OCT features into a multi-modal classifier trained by fused features, and calculate the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier.

[0116] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs, Read-Only Memories), random access memories (RAMs, Random Access Memories), magnetic disks, or optical discs.

[0117] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device.

[0118] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part with one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, a computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0119] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc. It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present invention, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A classification method for age-related macular degeneration based on hierarchical fusion of fundus bimodal images, characterized in that, Including: By using paired CFP images and OCT images to train a feature extraction and fusion network, CFP features, OCT features, and fused high-dimensional features are obtained; Input the CFP features and OCT features into a modality discriminator to enhance the attention of the feature extractor to the inter-modal relevant information; Input the CFP features and OCT features into a multi-modal classifier trained with fused features, and calculate the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier; The input of the CFP feature and the OCT feature into the modality discriminator includes processing the input feature map F4 through a 1×1 convolutional layer to generate a feature F'∈R H×W×kC , where C is the number of channels required to detect the discriminant regions for each class, k represents the number of classes, the new feature F” is obtained by performing a global max pooling operation on F', and F” is input into a Gaussian similarity clustering to calculate the class scores S = {S1, S2, …, S k}; Perform intra-class cross-channel average pooling operation on F' to obtain the feature map of each category: where f' i,j represents the j-th feature map of the i-th class in F', and F' i_avg ∈R H×W×1 represents the feature map of the i-th class; Category Attention ATT CAB ∈R H×W×1 The operation process is as follows: ATT CAB Highlight the regions discriminative for age-related macular degeneration classification; finally, multiply the input feature map F CAB-IN and the class attention ATT CAB pixel by pixel to obtain the output feature map F CAB-OUT : Among them, represents element-wise multiplication; The step of inputting the CFP features and OCT features into the modality discriminator further includes converting the input feature map into a feature vector through a global average pooling layer. The feature vector passes through a gradient reversal layer GRL. In backpropagation, the gradient calculated from the discriminator error is multiplied by a negative scaling factor, the sign of the gradient is reversed, and the magnitude is scaled. The negative gradient is passed to the feature extractor to update the gradient. After the update is completed, the feature vector passes through a fully-connected neural network, and the discriminator outputs the probability distributions of different modality images. The discriminator loss is expressed as: Where D represents the discriminator, C4 is the CFP feature in the fourth stage, and O4 is the OCT feature in the fourth stage; The calculation of the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier includes using the CFP feature C i , the OCT feature O i , and the fusion feature F i to train three classifiers, namely the CFP classifier, the OCT classifier, and the multi-modal classifier. The output probability distributions of each feature are obtained through the three classifiers, denoted as P C , P O , and P M . Then, C i and O i are input into the multi-modal classifier to obtain the output probability distributions P M (C i ) and P M (O i ). Calculate the mean square error loss between P C and P M (C i ) and the mean square error loss between P O and P M (O i ). The modality consistency loss is expressed as: Where j is the category index and k is the total number of categories; The step of calculating the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier further includes that the loss function consists of three parts, including the discriminator loss, the modality consistency loss, and the classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multi-modal classifier and the true label, and is expressed as: L c (P M ,CE(P M ,y)) = α·(1 - exp(-CE(P M ,y))) γ ·CE(P M ,y) Among them, CE represents the cross-entropy loss, P M represents the output of the multi-modal classifier, y represents the true label, α represents the class balance factor, and γ represents the focal factor; The step of calculating the modality consistency loss includes that the total loss function is expressed as: L total = L c + L mc + L D 。 2. The method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images according to claim 1, wherein: The training of the feature extraction and fusion network by using paired CFP images and OCT images includes performing 4×4 convolutional downsampling on the CFP and OCT images by adopting the parallel structure of the feature extraction and fusion network. The processed CFP and OCT images respectively pass through their respective feature extraction channels, and after four stages of feature extraction and fusion operations. In the first stage, the CFP and OCT images respectively pass through two feature extraction modules to obtain the CFP image feature C1 ∈ R 56×56×96 and the OCT image feature O1 ∈ R 56×56×96 , and feature fusion is performed to obtain the fusion feature F1 ∈ R 56×56×96 ; In the second to fourth stages, the CFP feature extraction channel performs downsampling by image block merging, and the OCT feature extraction channel performs downsampling by 2×2 convolution; In the second stage, the downsampled C1 and O1 respectively pass through two feature extraction modules to obtain the CFP feature C2 ∈ R of the second stage 28×28×192 and the OCT feature O2 ∈ R 28×28×192 . The features of the second stage are processed by a feature fusion module to obtain the fused feature F2 ∈ R of the second stage 28×28×192 ; In the third stage, 12 feature extraction modules are used to process the downsampled C2 and O2 respectively, obtaining feature C3 ∈ R 14×14×384 and O3 ∈ R 14×14×384 , and they are fused through a feature fusion module to obtain the fused feature F3 ∈ R 14×14×384 ; In the fourth stage, the downsampled C3 and O3 are processed using two feature extraction modules to obtain C4 ∈ R 7×7×768 and O4 ∈ R 7 ×7×768 , and they are fused through a feature fusion module to obtain the fused feature F4 ∈ R 7×7×768 , and F4 is processed using a class attention module.

3. A system adopting the method for classifying age-related macular degeneration based on hierarchical fusion of fundus bimodal images as described in any one of claims 1 and 2, characterized in that: Including a feature extraction module, an information attention enhancement module, and a loss module; The feature extraction module is used to obtain CFP features, OCT features, and fused high-dimensional features by using paired CFP images and OCT images to train a feature extraction and fusion network; The information attention enhancement module is used to input the CFP features and OCT features into a modality discriminator to enhance the attention of the feature extractor to the inter-modal relevant information; The loss module is used to input the CFP features and OCT features into a multi-modal classifier trained with fused features, and calculate the modality consistency loss by combining the obtained probability distribution with the probability distributions output by the CFP classifier and the OCT classifier.

4. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the age-related macular degeneration classification method based on hierarchical fusion of fundus bimodal images according to any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the age-related macular degeneration classification method based on hierarchical fusion of fundus bimodal images according to any one of claims 1 to 2.

Images