Elderly macular degeneration classification method based on fundus bimodal image hierarchical fusion
By adopting a dual-modal image hierarchical fusion method in multimodal image classification, combining category attention and modal consistency losses, the problem of insufficient information utilization in modes in deep learning is solved, and the classification of macular degeneration in the elderly is significantly improved performance and accuracy.
Patent Information
- Application Number
- CN202411488839.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-10-24
AI Technical Summary
The multimodal image classification method based on deep learning has problems such as inability to fully utilize the complementary information between modals, unable to effectively analyze low-dimensional features, limited by sample number, poor training effect, and difficult to fully utilize the complementary information between modals. Causes insufficient classification performance.
Using a method based on the hierarchical fusion of fundus dual-modal images, the feature extraction and fusion network is trained through paired color illumination and OCT images to obtain high-dimensional features, and input features into the modal discriminator to enhance the correlation between the feature extractors for modality. The attention of information is combined with the category attention module and modal consistency loss, and the output of the classifier is optimized.
The classification performance of macular degeneration in the elderly has been significantly improved, the model's attention to task-related category characteristics has been enhanced, the deviation between modes has been reduced, and the overall classification accuracy and robustness have been improved.
Smart Images

Figure CN119006943A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images. Background Art
[0002] Age-related macular degeneration (AMD) is a common ophthalmic disease. Color fundus photography (CFP) and optical coherence tomography (OCT) images are widely used in the diagnosis of ophthalmic diseases. In recent years, deep learning (DL)-based image processing technology has made significant progress and is widely used in tasks such as classification, segmentation, and registration of medical images. Especially in classification tasks, research has gradually expanded from single modality to multimodality. Different imaging modalities provide different perspectives and information. For example, CFP images show retinal vascular and pigment changes, while OCT images reveal microstructural changes in the retinal layer. Some lesions may not be obvious in a specific modality, but may be clearly visible in other modalities. By fusing the information of these modalities, the classification model can reduce missed diagnoses and misdiagnoses and improve diagnostic accuracy.
[0003] At present, multimodal DL network frameworks can be divided into five types according to the information fusion steps: input fusion, single-level fusion, hierarchical fusion, attention-based fusion, and output fusion. Input fusion refers to the information fusion performed before the DL backbone network. Single-level fusion uses different DL backbone networks to extract features of different modalities respectively, then performs information fusion and finally makes a final decision. Hierarchical fusion is an improved form of single-level fusion. This method fuses different dimensional features extracted by DL backbone at the network level, and can also perform further feature fusion after DL backbone to learn more complex features. Attention-based fusion uses the attention mechanism, especially the Transformer structure, to extract and combine features in the DL backbone network to enhance the feature expression ability. Output fusion generates the final classification result by feeding the different modal features extracted by DL backbone into different classifiers respectively, and then integrating these classification results through fusion techniques (such as voting or averaging).
[0004] Although DL-based multimodal classification methods have achieved remarkable success, existing methods still have the following shortcomings:
[0005] Input fusion: It is not suitable for processing heterogeneous modal data and cannot fully utilize the complementary information between modalities.
[0006] Single-level fusion: Since information fusion only occurs at the end of the network architecture, it is impossible to effectively jointly analyze low-dimensional features.
[0007] Hierarchical fusion: The registration problem of multimodal data may affect the fusion of low-dimensional features, thereby affecting the classification performance.
[0008] Attention-based fusion: Although Transformer performs well on large datasets, the insufficient number of samples in medical datasets limits its training effect.
[0009] Output fusion: Lack of feature fusion makes it difficult to fully utilize the complementary information between modalities. If the classification performance of different modalities varies greatly, output fusion may not significantly improve the classification performance. Summary of the invention
[0010] In view of the above-mentioned problems, the present invention is proposed.
[0011] Therefore, the technical problem solved by the present invention is: the multimodal classification method based on DL cannot fully utilize the complementary information between modalities, cannot effectively jointly analyze low-dimensional features, is limited by the number of samples, has poor training effect, and how to fully utilize the complementary information between modalities to significantly improve the classification performance.
[0012] To solve the above technical problems, the present invention provides the following technical solutions: a classification method for age-related macular degeneration based on hierarchical fusion of fundus dual-modality images, comprising training a feature extraction and fusion network using paired CFP images and OCT images to obtain CFP features, OCT features and fused high-dimensional features; inputting the CFP features and OCT features into a modality discriminator to enhance the feature extractor's attention to inter-modality correlation information; inputting the CFP features and OCT features into a multimodal classifier trained by fused features, and calculating the modality consistency loss by combining the obtained probability distribution with the probability distribution output by the CFP classifier and the OCT classifier.
[0013] As a preferred solution of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in the present invention, wherein: the training of the feature extraction and fusion network by using paired CFP images and OCT images includes adopting a parallel structure of the feature extraction and fusion network to perform 4×4 convolution downsampling processing on the CFP and OCT images, and the processed CFP and OCT images pass through their respective feature extraction channels, and undergo four stages of feature extraction and fusion operations. In the first stage, the CFP and OCT images pass through two feature extraction modules respectively to obtain the CFP image feature C1∈R 56×56×96 and OCT image features O1∈R 56×56×96 , perform feature fusion to obtain the first stage fusion feature F1∈R 56×56×96 ;
[0014] In the second to fourth stages, the CFP feature extraction channel uses image block merging for downsampling, and the OCT feature extraction channel uses 2×2 convolution for downsampling;
[0015] In the second stage, the downsampled C1 and O1 are respectively passed through two feature extraction modules to obtain the second stage CFP feature C2∈R 28×28×192 and OCT feature O2∈R 28×28×192 , the second stage features are processed by the feature fusion module to obtain the second stage fusion feature F2∈R 28×28×192 ;
[0016] In the third stage, 12 feature extraction modules are used to process the downsampled C2 and O2 respectively, and the feature C3∈R 14 ×14×384 and O3∈R 14×14×384 , and fused through the feature fusion module to obtain the third stage fusion feature F3∈R 14×14×384 ;
[0017] In the fourth stage, two feature extraction modules are used to process the downsampled C3 and O3 to obtain C4∈R 7×7×768 and O4∈R 7×7×768 , and fused through the feature fusion module to obtain the fourth stage fusion feature F4∈R 7×7×768 , F4 is processed using the category attention module.
[0018] As a preferred solution of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in the present invention, wherein: the input of CFP features and OCT features into the modality discriminator includes processing the input feature map F4, processing through a 1×1 convolution layer, and generating a feature F'∈R H×W×kC , where C is the number of channels required to detect the discriminant area of each category, k represents the number of categories, and F' is subjected to the global maximum pooling operation to obtain a new feature F'. F' is input into the Gaussian similarity clustering to calculate the category score S = {S1, S2, ..., S k}.
[0019] As a preferred solution of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modal images described in the present invention, the input of CFP features and OCT features into the modality discriminator also includes converting the input feature map into a feature vector through a global average pooling layer, and the feature vector passes through a gradient reverse layer GRL. In the back propagation, the gradient calculated from the discriminator error is multiplied by a negative scaling factor, the sign of the gradient is reversed, and the size is scaled, and the negative gradient is passed to the feature extractor to update the gradient. After the update is completed, the feature vector passes through a fully connected neural network to output the probability distribution of different modality images.
[0020] As a preferred embodiment of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in the present invention, wherein: the probability distribution obtained by combining the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss includes using the CFP feature C i OCT features i And the fusion feature F i Train three classifiers, namely CFP classifier, OCT classifier and multimodal classifier, and obtain the output probability distribution of each feature through the three classifiers, which are denoted as P C , P O and P M , and then C i and O i Input to the multimodal classifier and get the output probability distribution P M (C i ) and P M (O i ), calculate P C and P M (C i ) and the mean square error loss between O and P M (O i ) is the mean square error loss between .
[0021] As a preferred embodiment of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modal images described in the present invention, wherein: the obtained probability distribution is combined with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss, and the loss function is composed of three parts, including the discriminator loss, the modality consistency loss and the classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multimodal classifier and the true label, and is expressed as:
[0022] L c (P M ,CE(P M ,y))=α·(1-exp(-CE(P M ,y)))γ ·CE(P M ,y)
[0023] Among them, CE represents the cross entropy loss, P M represents the output of the multimodal classifier, y represents the true label, α represents the category balance factor, and γ represents the focus factor.
[0024] As a preferred solution of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in the present invention, the calculation of modality consistency loss includes a total loss function expressed as:
[0025] L total =L c +L mc +L D .
[0026] Another object of the present invention is to provide an age-related macular degeneration classification system based on hierarchical fusion of fundus bimodal images, which can use the Gaussian similarity clustering method to evaluate the similarity between features of different categories through category attention based on Gaussian similarity clustering, thereby generating weights for specific categories, helping the model to focus more on category features related to the current task, improving classification accuracy and robustness, and solving the problem of insufficient accuracy of current DL-based multimodal classification methods.
[0027] As a preferred solution of the age-related macular degeneration classification system based on hierarchical fusion of fundus dual-modality images described in the present invention, it includes:
[0028] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a method for classifying age-related macular degeneration based on hierarchical fusion of fundus dual-modality images.
[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for classifying age-related macular degeneration based on hierarchical fusion of fundus dual-modality images.
[0030] Beneficial effects of the present invention: The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modal images provided by the present invention introduces a category attention module based on Gaussian similarity clustering to enhance the model's attention to task-related category features. Through a modality discriminator based on adversarial learning, the network can effectively distinguish the features of different modalities, reduce the deviation between modalities and enhance the complementarity of features. Through modality consistency loss, valuable information is extracted from the modality with better performance, and the performance of other modalities is improved, thereby optimizing the overall classification effect and significantly improving the classification performance of age-related macular degeneration. The present invention achieves better results in terms of accuracy and classification performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0032] Figure 1 An overall flow chart of an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images provided in the first embodiment of the present invention.
[0033] Figure 2 A structural diagram of a feature extraction and fusion network for an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images provided in the first embodiment of the present invention.
[0034] Figure 3 A schematic diagram of a CFP feature extraction module of an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images provided in the first embodiment of the present invention.
[0035] Figure 4 A schematic diagram of an OCT feature extraction module of an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images provided in the first embodiment of the present invention.
[0036] Figure 5 A schematic diagram of a feature fusion module of an age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images provided in the first embodiment of the present invention.
[0037] Figure 6 A schematic diagram of a category attention module of an age-related macular degeneration classification system based on hierarchical fusion of fundus bimodal images provided in the first embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0039] Example 1, reference Figure 1-Figure 6 , which is an embodiment of the present invention, provides a method for classifying age-related macular degeneration based on hierarchical fusion of fundus dual-modality images, comprising:
[0040] S1: By using paired CFP images and OCT images to train feature extraction and fusion networks, CFP features, OCT features, and fused high-dimensional features are obtained.
[0041] Furthermore, the method of training the feature extraction and fusion network by using the paired CFP image and OCT image includes adopting a parallel structure of the feature extraction and fusion network to perform 4×4 convolution downsampling processing on the CFP and OCT images. The processed CFP and OCT images pass through their respective feature extraction channels and undergo four stages of feature extraction and fusion operations. In the first stage, the CFP and OCT images pass through two feature extraction modules respectively to obtain the CFP image feature C1∈R 56×56×96 and OCT image features O1∈R 56×56×96 , perform feature fusion to obtain the first stage fusion feature F1∈R 56×56×96 ;
[0042] In the second to fourth stages, the CFP feature extraction channel uses image block merging for downsampling, and the OCT feature extraction channel uses 2×2 convolution for downsampling;
[0043] In the second stage, the downsampled C1 and O1 are respectively passed through two feature extraction modules to obtain the second stage CFP feature C2∈R 28×28×192 and OCT feature O2∈R 28×28×192 , the second stage features are processed by the feature fusion module to obtain the second stage fusion feature F2∈R 28×28×192 ;
[0044] In the third stage, 12 feature extraction modules are used to process the downsampled C2 and O2 respectively, and the feature C3∈R 14 ×14×384 and O3∈R 14×14×384 , and fused through the feature fusion module to obtain the third stage fusion feature F3∈R 14×14×384 ;
[0045] In the fourth stage, two feature extraction modules are used to process the downsampled C3 and O3 to obtain C4∈R 7×7×768 and O4∈R 7×7×768 , and fused through the feature fusion module to obtain the fourth stage fusion feature F4∈R 7×7×768 , F4 is processed using the category attention module.
[0046] It should be noted that the overall structure of the CFP feature extraction module is shown in the attached Figure 3 As shown in the figure. This module introduces the Windows Multi-head Self-Attention (W-MSA) and ShiftWindows Multi-head Self-Attention (SW-MSA) mechanisms. W-MSA divides the input sequence into multiple windows of fixed size and calculates self-attention in each window; SW-MSA calculates self-attention using a regular window partitioning scheme at the lth layer, and generates a new window by shifting the window partition at the l+1th layer, so that the self-attention calculation in the new window can cross the window boundary of the lth layer, providing connections between windows.
[0047] Enter CFP characteristics First, the features are normalized. Then, they are processed by W-MSA, 1×1 convolution, and GELU activation functions, and then the processed features are compared with the original features F. CFP Perform residual connection to obtain F' CFP ∈R H×W×C The GELU activation function adjusts the output value through a gating mechanism. The specific formula is shown in formula (1). Next, F' CFP After layer normalization and relative position shift, it is processed by SW-MSA, 1×1 convolution and GELU activation function in turn, and finally the processed features are compared with F' CFP Perform residual connection to obtain the final output CFP feature Among them, H, W, and C represent the height, width, and number of channels of the feature map, respectively.
[0048] GELU(x)=xP(x) (1)
[0049] Where P(x) is the cumulative distribution function of the Gaussian distribution N(μ, σ2) with mean 0 and variance 1.
[0050] It should also be noted that the overall structure of the OCT feature extraction module is shown in the attached Figure 4 Input OCT features First, it undergoes 3×3 depth convolution and layer normalization. Then, 1×1 convolution is used for cross-channel information interaction. Then, the processed features are compared with the original features. Perform residual connection to obtain the final output OCT features
[0051] Furthermore, the overall structure of the feature fusion module is shown in the attached figure. Figure 5 As shown. Input CFP feature C i ∈R H×W×C and OCT features i ∈R H×W×C After being processed by the channel attention module and the self-attention module, we get and W, H, C represent the width, height and number of channels of the feature map. The fusion features of the previous stage After 1×1 convolution and average pooling, we get C i , O i and The concatenated features are processed by layer normalization (LN) and 1×1 convolution to obtain a comprehensive feature representation. Finally, and The concatenation is done in the channel dimension, processed by the Residual Multi-layer Perceptron (RMLP), and connected by skip connections. Add it to the processed features to get the final fusion feature F i The feature fusion operation process is shown in formula (2).
[0052]
[0053] Among them, CA represents channel attention, SA represents self-attention, Avgpool represents average pooling, Concat represents concatenation in the channel dimension, LN represents layer normalization, RMLP represents residual multilayer perceptron, and f 1×1 Represents a 1×1 convolution.
[0054] The operation process of the channel attention module is as follows. First, the spatial dimension of the input feature F is squeezed through global average pooling to obtain the statistical information z of each channel. Next, the channel statistics obtained by squeezing are excitation operations to fully capture the dependencies between channels. The excitation operation includes two fully connected layers and a nonlinear activation function ReLU. Specifically, the channel dimension is first reduced through a dimensionality reduction layer with a reduction ratio of r=16, and then the channel dimension is restored to the dimension of the input feature through a dimensionality increase layer. Finally, the channel weight F is calculated by the Sigmoid activation function. ex . Set the channel weight F ex Multiply the input feature F channel by channel to get the output of the channel attention module. The calculation process is expressed as:
[0055]
[0056]
[0057] Among them, δ is the ReLU activation function, ε is the Sigmoid activation function, W1 and W2 are the weight vectors of the dimensionality reduction layer and the dimensionality increase layer, respectively, and p, q, and m are the serial numbers of rows, columns, and channels, respectively. Represents channel-by-channel multiplication.
[0058] The operation process of the self-attention module is as follows. First, the input feature map F is passed through three learnable 1×1 convolutions to generate three tensors: query F q ∈R WW×H×C , key F k ∈R WW×H×C Sum value F v ∈R WW×H×C By calculating F q and F k Finally, the self-attention feature F SA is to combine the attention weight map with F v The weighted sum is expressed as:
[0059]
[0060] RMLP consists of a 3×3 deep convolution residual layer and two linear transformation layers, where the GELU activation function is applied after the first linear transformation layer to introduce nonlinear transformation. This design effectively makes up for the lack of FFN and alleviates the problems of gradient disappearance, gradient explosion and network degradation to a certain extent, which can be expressed as:
[0061] RMLP(x)=f 1×1 (f 1×1 (f depth3×3 (x)+x))
[0062] Among them, f 1×1 represents 1×1 convolution, f depth3×3 Represents a 3×3 depthwise convolution.
[0063] S2: CFP features and OCT features are fed into the modality discriminator to enhance the feature extractor’s attention to inter-modality correlation information.
[0064] Furthermore, the input of the CFP features and the OCT features into the modality discriminator includes processing the input feature map F4 through a 1×1 convolutional layer to generate a feature F′∈R H×W×kC , where C is the number of channels required to detect the discriminant region of each category, k represents the number of categories, and F′ is subjected to the global maximum pooling operation to obtain a new feature F″. F″ is input into the Gaussian similarity clustering to calculate the category score S = {S1, S2, …, S k}, expressed as:
[0065]
[0066] Where d is the Gaussian similarity measure, σ is a hyperparameter, and w i ∈R 1×1×kC is a learnable weight vector.
[0067] Perform intra-class cross-channel average pooling on F′ to obtain the feature map of each category:
[0068]
[0069] Among them, f′ i,j represents the jth feature map of the i-th category in F′, F′ i_avg ∈R H×W×1 Represents the feature map of the i-th category.
[0070] Category Attention ATT CAB ∈RH ×W×1 The operation process is as follows:
[0071]
[0072] ATT CAB The discriminative regions for AMD classification can be highlighted. Finally, the input feature map F CAB-IN and category attention ATT CAB Multiply pixel by pixel to get the output feature map F CAB-OUT :
[0073]
[0074] in, Represents element-wise multiplication.
[0075] It should be noted that the operation flow of the modality discriminator includes the following steps. First, the input feature map is converted into a feature vector through a global average pooling layer. Then, the feature vector passes through a gradient reversal layer (GRL), and the role of GRL in the forward propagation is to output the input data as is without any transformation. In the back propagation, it multiplies the gradient calculated from the discriminator error by a negative scaling factor, reverses the sign of the gradient, and scales its magnitude. The negative gradient is then passed to the feature extractor to update its gradient. Next, the feature vector passes through a fully connected neural network, which includes an input layer, a hidden layer consisting of 512 neurons (ReLU activation function is applied), and an output layer (classification is performed using the softmax activation function). Finally, the discriminator outputs the probability distribution of images of different modalities.
[0076] Adversarial learning is a maximum-minimum game that trains the feature extractor so that the features it generates are not easily correctly classified by the modality discriminator. The discriminator is to maximize the ability to assign correct labels to the two modal features. The discriminator loss is expressed as:
[0077]
[0078] Among them, D represents the discriminator, C4 is the CFP feature of the fourth stage, and O4 is the OCT feature of the fourth stage.
[0079] S3: Input the CFP features and OCT features into the multimodal classifier trained by the fusion features, and combine the obtained probability distribution with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss.
[0080] Furthermore, the operation process of modal consistency loss is as follows: First, use the CFP feature C i OCT features i And the fusion feature F i Train three classifiers, namely CFP classifier, OCT classifier and multimodal classifier. Through these three classifiers, the output probability distribution of each feature is obtained, which is denoted as P C , P O and P M Then C i and O i Input to the multimodal classifier to obtain the corresponding output probability distribution P M (C i ) and P M (O i ). Calculate P C and P M (Ci ) between the mean squared error (MSE) loss, and P O and P M (O i ). The modality consistency loss is expressed as:
[0081]
[0082] Here, j is the category index and k is the total number of categories.
[0083] It should be noted that the probability distribution obtained by combining the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss also includes a loss function consisting of three parts including discriminator loss, modality consistency loss and classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multimodal classifier and the true label, which is expressed as:
[0084] L c (P M ,CE(P M ,y))=α·(1-exp(-CE(P M ,y))) γ ·CE(P M , y)
[0085] Among them, CE represents the cross entropy loss, P M represents the output of the multimodal classifier, y represents the true label, α represents the category balance factor, and γ represents the focus factor.
[0086] Example 2, an embodiment of the present invention, provides a classification method for age-related macular degeneration based on hierarchical fusion of fundus dual-modality images. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.
[0087] First, the data set used in this embodiment is a public data set, including 1094 CFP images and 1289 OCT images from 1093 eyes of 829 subjects. Among them, 817 eyes are equipped with 1 to 5 OCT images. CFP images are collected by Topcon fundus camera, while OCT images are collected by Topcon OCT camera and Heidelberg OCT camera respectively. They are all central B-Scan images and are manually selected by technicians. The classification label of each eye is obtained by two ophthalmologists based on comprehensive evaluation of CFP images, OCT images, fluorescein angiography and indocyanine green angiography. The classification of eyes includes normal, dry AMD, wet AMD or PCV, and all images of the same eye are labeled consistently. During data processing, CFP images and OCT images are paired by eye. The paired multimodal samples are then divided into training set, validation set and test set in a ratio of 7:1.5:1.5.
[0088] Accuracy, precision, recall, F1 score, kappa coefficient and AUC value were used as evaluation indicators for classification.
[0089] Accuracy: The ratio of correctly classified samples to the total number of samples.
[0090]
[0091] Precision: The proportion of samples predicted by the model to be positive that are actually positive.
[0092]
[0093]
[0094] Among them, TP i Represents the number of true positive samples of the i-th category; FP i Represents the number of false positive samples in the i-th category.
[0095] Recall: The proportion of samples that are actually positive examples that are predicted as positive examples by the model.
[0096]
[0097]
[0098] Among them, FN i represents the number of false negative samples in the i-th category; k is the number of categories.
[0099] F1 score: The harmonic mean of precision and recall, used to comprehensively evaluate the performance of the model, and is suitable for cases with unbalanced category distribution.
[0100]
[0101]
[0102] Kappa coefficient: It is used to measure the consistency of the classification model. It takes into account the difference between the proportion of correct predictions by the model and the proportion of random predictions. It can be used to solve the situation of sample imbalance or too many classification categories.
[0103]
[0104] Among them, P o represents the observed accuracy, P e Indicates the accuracy of random prediction.
[0105] AUC value: The area under the ROC curve is used to measure the average performance of the model under all thresholds. The larger the AUC value, the better the model performance.
[0106] The present invention has conducted corresponding ablation experiments on category attention, modality discriminator and modality distillation methods. In the comparative experiment, the method of the present invention is compared with other excellent classification networks based on deep learning, including MM-CNN, AM-FAM, MSAN, CRD-Net, and Hifuse, which verifies the effectiveness of the method proposed in the present invention.
[0107] Table 1 Ablation experiment results of multimodal age-related macular degeneration classification
[0108]
[0109] Table 1 lists the results of the ablation experiment. Compared with the baseline network, the proposed method shows significant improvements in multiple performance indicators. The accuracy, precision, recall, F1 score, Kappa and AUC values are improved by 4.4%, 3.2%, 2.9%, 3%, 6.5% and 0.6% respectively compared with the baseline method. The improvement of these indicators reflects the advantages of model classification performance and consistency.
[0110] Table 2 Comparative experimental results of multimodal age-related macular degeneration classification
[0111] method Accuracy Accuracy Recall F1 Kappa AUC MSAN 0.670 0.700 0.675 0.662 0.487 0.899 MM-CNN 0.813 0.836 0.845 0.837 0.724 0.949 Hifuse 0.819 0.823 0.812 0.817 0.727 0.945 AM-FAM 0.824 0.820 0.833 0.826 0.739 0.944 CRD-Net 0.852 0.856 0.893 0.871 0.782 0.961 The present invention 0.885 0.875 0.883 0.879 0.827 0.959
[0112] Table 2 lists the results of the comparative experiments. The present invention shows significant improvements in all indicators. The accuracy, precision, F1 score and Kappa value are all the highest, indicating that the present invention is superior to the existing methods in terms of classification performance and prediction consistency. Although it is slightly inferior to CRD-Net in terms of AUC, the overall performance is still excellent.
[0113] The present invention proposes an innovative hierarchical fusion method of fundus bimodal images for automatic classification of AMD. The method processes features of different modalities in feature extraction and fusion networks through four stages, overcoming the shortcomings of existing multi-scale feature fusion methods. Specifically, a feature fusion module combining channel attention and self-attention mechanisms is designed, which effectively fuses features of different scales and modalities at different levels of the network and suppresses irrelevant or noise features through the attention mechanism. In addition, a category attention module based on Gaussian similarity clustering is introduced to help the model focus more on category features related to the current task to enhance the model's attention to the lesion area. In order to reduce the deviation between modalities, the present invention adopts a modality discriminator based on adversarial learning. Finally, through the modality consistency loss, valuable information is extracted from the modality with better performance, and it is used to improve the performance of other modalities, thereby making full use of the information of the existing modalities and improving the overall classification effect. Experiments show that the network proposed in the present invention can overcome the difficult problems of multimodal image classification, such as insufficient multi-scale feature fusion ability, neglect of related information of different modalities, insufficient use of complementary information between modalities, and insufficient attention to the lesion area. The hierarchical fusion network proposed in the present invention can achieve accurate classification of AMD, surpassing many other advanced classification networks.
[0114] Embodiment 3, an embodiment of the present invention, provides an age-related macular degeneration classification system based on hierarchical fusion of fundus dual-modality images, including a feature extraction module, an information attention enhancement module, and a loss module.
[0115] Among them, the feature extraction module is used to train the feature extraction and fusion network by using paired CFP images and OCT images to obtain CFP features, OCT features and fused high-dimensional features; the information attention enhancement module is used to input CFP features and OCT features into the modal discriminator to enhance the feature extractor's attention to the relevant information between modalities; the loss module is used to input CFP features and OCT features into the multimodal classifier trained by the fused features, and combine the obtained probability distribution with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modal consistency loss.
[0116] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0117] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0118] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.
[0119] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.
[0120] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A classification method for age-related macular degeneration based on hierarchical fusion of fundus dual-modality images, characterized in that: include: By using paired CFP images and OCT images to train feature extraction and fusion networks, CFP features, OCT features, and fused high-dimensional features are obtained; The CFP features and OCT features are input into the modality discriminator to enhance the feature extractor's attention to the inter-modality correlation information; The CFP features and OCT features are input into the multimodal classifier trained by the fusion features, and the obtained probability distribution is combined with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss.
2. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 1, characterized in that: The method of training the feature extraction and fusion network by using the paired CFP image and OCT image includes adopting a parallel structure of the feature extraction and fusion network to train the CFP and OCT images. Convolution downsampling processing, the processed CFP and OCT images pass through their own feature extraction channels, and undergo four stages of feature extraction and fusion operations. In the first stage, the CFP and OCT images pass through two feature extraction modules respectively to obtain the CFP image features. and OCT image features , perform feature fusion to obtain the fusion features of the first stage ; In the second to fourth stages, the CFP feature extraction channel uses image block merging for downsampling, and the OCT feature extraction channel uses Convolution is used for downsampling; In the second stage, after downsampling and Through two feature extraction modules respectively, the CFP features of the second stage are obtained and OCT features The second stage features are processed by the feature fusion module to obtain the second stage fusion features ; The third stage uses 12 feature extraction modules to process the downsampled and , get the features and , and fused through the feature fusion module to obtain the fusion features of the third stage ; In the fourth stage, two feature extraction modules are used to process the downsampled and ,get and , and fused through the feature fusion module to obtain the fusion features of the fourth stage , processed using the category attention module .
3. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 2, characterized in that: The inputting of CFP features and OCT features into the modality discriminator includes processing the input feature map ,pass Convolutional layer processing, generating features ,in is the number of channels required to detect the discriminative region for each category, represents the number of categories, New features are obtained through global maximum pooling operation Will Input into Gaussian similarity clustering to calculate category scores .
4. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 3, characterized in that: The method of inputting CFP features and OCT features into a modality discriminator also includes converting an input feature map into a feature vector through a global average pooling layer. The feature vector passes through a gradient reverse layer GRL. In back propagation, the gradient calculated from the discriminator error is multiplied by a negative scaling factor to reverse the sign of the gradient and scale the size. The negative gradient is passed to the feature extractor to update the gradient. After the update is completed, the feature vector passes through a fully connected neural network to output the probability distribution of different modality images.
5. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 4, characterized in that: The obtained probability distribution is combined with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss, including using the CFP feature OCT features and fusion features Three classifiers are trained, namely CFP classifier, OCT classifier and multimodal classifier. The output probability distribution of each feature is obtained through the three classifiers, which are recorded as , and , and then and Input to the multimodal classifier to get the output probability distribution and ,calculate and The mean square error loss between and The mean square error loss between .
6. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 5, characterized in that: The obtained probability distribution is combined with the probability distribution output by the CFP classifier and the OCT classifier to calculate the modality consistency loss. The loss function is composed of three parts, including discriminator loss, modality consistency loss and classification loss. The classification loss is obtained by calculating the focal loss between the prediction of the multimodal classifier and the true label, which is expressed as: , in, represents the cross entropy loss, represents the output of the multimodal classifier, represents the true label, represents the category balance factor, Represents the focus factor.
7. The age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as claimed in claim 6, characterized in that: The calculation of the modality consistency loss includes a total loss function expressed as: 。 8. A system using the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images as described in any one of claims 1 to 7, characterized in that: Including feature extraction module, information attention enhancement module, and loss module; The feature extraction module is used to train a feature extraction and fusion network by using paired CFP images and OCT images to obtain CFP features, OCT features, and fused high-dimensional features; The information attention enhancement module is used to input CFP features and OCT features into the modality discriminator to enhance the feature extractor's attention to inter-modality related information; The loss module is used to input CFP features and OCT features into a multimodal classifier trained by fusion features, and calculate the modality consistency loss by combining the obtained probability distribution with the probability distribution output by the CFP classifier and the OCT classifier.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the age-related macular degeneration classification method based on hierarchical fusion of fundus dual-modality images described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Elderly macular lesion priori interpretable method and system and storage medium
CN114820509A
Multi-modal retina fundus image classification method
CN116824217A
Visible light and infrared image fusion method based on spatial gradient guide network
CN117058505A
Multi-modal macular degeneration recognition model construction method and device, equipment and medium
CN117788881A
Glaucoma image detection method based on multi-modal fusion
CN118299007A
