A multi-modal sugar net image lesion grading method

Abstract

The multi-modal sugar net image lesion grading method comprises a backbone network, a category attention module, a modality interaction information guided spatial attention module and a classifier, the backbone network is two independent backbone networks, respectively extracting features from the read-in color fundus image and the fluorescein angiography image; the category attention module uses the color fundus image and the fundus fluorescein angiography image features to obtain the attention weight of each category, obtains the category weighted feature map, and sends it to the modality interaction information guided spatial attention module; the modality interaction information guided spatial attention module models the feature dependency relationship between the modalities through the modality information interaction module and extracts the context correlation, then uses the spatial attention to include the key lesion position captured by the modality information interaction module into the attention weighted fusion process, realizes the accurate positioning of the key lesion; the classifier sends the output of the spatial attention into the classifier containing a global average pooling operation and a fully connected layer to perform severity grading. The image lesion grading method relates to the fusion of color fundus features and fluorescein angiography features, focuses on combining the information with grading value in the two modalities to make decisions, and has the advantages of good reliability, high use value and the like.

Description

(I) Technical Field

[0001] This invention belongs to the field of pattern recognition technology, specifically relating to multimodal diabetic retinopathy image lesion grading technology. (II) Background Technology

[0002] With the rapid development of computer technology and artificial intelligence, retinal vessel segmentation and automatic grading technologies have become important tools for assisting clinical diagnosis and combining prevention and treatment of diabetic retinopathy. However, due to the multi-scale changes in retinal vessels and the complex pathological relationship between fundus images and diabetic retinopathy, traditional retinal vessel segmentation and automatic grading algorithms for diabetic retinopathy have limitations in accuracy and reliability. Multimodal diabetic retinopathy grading methods can comprehensively utilize pathological information from multiple imaging modalities, thus exhibiting superior grading performance. However, the scarcity of multimodal data and the lack of effective multimodal feature fusion methods have greatly limited the development of multimodal grading algorithms. Therefore, this invention focuses on a multimodal feature fusion grading method to improve the reliability of automatic diagnosis of retinal diseases. (III) Summary of the Invention

[0003] The purpose of this invention is to provide an effective multimodal diabetic retinopathy image lesion grading method. This classification model, by combining category attention and spatial attention guided by the modal interaction module, can simultaneously enhance the model's ability to extract intra-class information and important lesion information of fused features, so as to obtain more accurate diabetic retinopathy grading results.

[0004] A multimodal diabetic retinopathy image lesion grading method includes a backbone network, a category attention module, a spatial attention module guided by modal interaction information, and a classifier. The backbone network consists of two independent backbone networks, which extract features from the read-in color fundus images and fluorescein angiography images, respectively. The category attention module uses the features of the color fundus images and fluorescein angiography images to obtain the attention weights for each category, resulting in a category-weighted feature map, which is then fed into the spatial attention module guided by modal interaction information. The spatial attention module models the feature dependencies between modalities and extracts contextual relevance through the modal information interaction module. Then, it uses spatial attention to incorporate the key lesion locations captured by the modal information interaction module into the attention-weighted fusion process, achieving accurate localization of key lesions. The classifier feeds the output of the spatial attention into a classifier containing a global average pooling operation and a fully connected layer for severity grading.

[0005] The category attention module first processes the feature map F output by the backbone network through a 1×1 convolutional layer to generate a feature map Y∈R. H×W×kLWhere L is the number of categories, k is the fixed number of channels allocated to learning features for each category, and then a "channel splitting" operation is used to divide Y into L groups, i.e., Y′=[Y1′,…,Y L ′],Y i ′∈R H×W×k The intra-class features of each class are processed in parallel, and then, in each Y... i Dilated convolutions are used to increase intra-class receptive fields and reduce interference between inter-class information.

[0006] Then, the inverse operation of channel splitting is used to connect all branches into a unified feature map. Global average pooling and channel average pooling along the category dimension are then used to obtain the attention weights for each category, resulting in the category-weighted feature map S1. The feature map F of the category attention block is then output by the following formula. category :

[0007]

[0008] The spatial attention module guided by modal interaction information comprises two parts: First, it models the feature dependencies between modalities through the proposed multimodal information interaction module, enhances attention to common key lesions, and extracts contextual relevance. Then, it uses spatial attention to incorporate the key lesion locations captured by the multimodal information interaction module into the attention-weighted fusion process, achieving accurate localization of key lesions. The specific process is as follows:

[0009] First, in the multimodal information interaction module, the feature information F extracted from the color fundus image is... CF As the main feature, the fluorescein angiography image feature F, which has similar semantics and contains important supplementary information, is utilized. FFA Generate interactive weighted feature W1;

[0010] Then, the importance of each channel in the main feature is adjusted using the interactive weighted feature W1 to obtain F′. interaction F′ interaction The calculation process is as follows:

[0011]

[0012] The adjusted feature map F′ interaction A lesion heatmap M is generated by 1×1 convolution to guide spatial attention in locating key lesion regions. The calculation process of M is as follows:

[0013] M = Conv 1×1 (F′ interaction )

[0014] Spatial attention captures the spatial correlation between features, enabling the network to focus on key lesion feature regions and suppress irrelevant regions. In the spatial attention branch, the multimodal fusion feature F, which has been optimized by category attention, is first processed. category Average pooling is performed along the channel dimension, and spatial feature description W2 is obtained by applying the sigmoid activation function:

[0015] W2 = f sigmoid (CGAP(F category ))

[0016] Here, CGAP represents the average pooling operation in the channel dimension. Next, the lesion heatmap M obtained from the multimodal information interaction module is used to adjust the feature description W2 to obtain the final score W3 in the spatial dimension, resulting in the output F of the spatial attention branch. spatial W3 and F spatial The calculation formula is as follows:

[0017]

[0018]

[0019] Guided by the lesion heatmap M generated by the multimodal information interaction module, spatial attention can focus on lesion feature information in multiple modalities, thereby more accurately locating key lesion positions. Finally, F spatial The severity level of diabetic retinopathy is generated by global average pooling and fully connected layers.

[0020] The main contributions and features of this invention are as follows:

[0021] This invention addresses the problem that fluorescein angiography images and color fundus images contain complementary pathological information, and that the fusion and interaction of these two modalities can comprehensively model important pathological regions in the classification task of diabetic retinopathy. It also addresses the issue of uneven distribution of data across different categories within the dataset. To solve this problem, a novel multimodal category-relational attention classification model is proposed. This model mainly consists of a backbone network, a category attention module, and a spatial attention module guided by modal interaction information. It combines information with classification value from both modalities to make decisions, aiming to obtain a highly practical classification model for diabetic retinopathy. (iv) Description of the attached drawings

[0022] Figure 1 The main flowchart of this invention.

[0023] Figure 2 Basic architecture of multimodal category-relational attention network.

[0024] Figure 3 Multimodal Feature Information Interaction Module (MFIM).

[0025] Figure 4 Spatial Attention Module (SAM). (V) Detailed Implementation

[0026] The invention will now be described in more detail with reference to the accompanying drawings:

[0027] This invention proposes a multimodal category-relational attention network to achieve disease level classification of color fundus images and fluorescein angiography images. The classification model mainly consists of a backbone network, a category attention module, a spatial attention module guided by modal interaction information, and a classifier.

[0028] 1. Multimodal Category-Relational Attention Network

[0029] Given the complementary feature information between color fundus images and fluorescein angiography images, this invention designs a category-relationship attention network based on multimodal feature fusion, abbreviated as MCRA-Net. For example... Figure 2 The MCRA-Net shown inputs images from two different modalities into two separate backbone networks for feature extraction. The backbone network uses the Xception network, and then integrates multimodal information through category attention (CAB) and spatial attention (SAM) guided by the modality interaction module (MFIM). It learns general feature representations in diabetic retinopathy images by combining color fundus images and corresponding fluorescein angiography images, focusing on combining information with hierarchical value from the two modalities to make decisions.

[0030] The input to MCRA-Net is a pair of color fundus images and fluorescein angiography images. The two modalities are fed into two separate backbone networks for feature extraction, and then modal information is fused through channel concatenation. Due to the imbalance of samples from different disease severity categories in the retinal image dataset, the recognition accuracy of small sample categories is low, and the symptom presentation of diabetic retinopathy at adjacent severity levels is similar, which also increases the difficulty of model judgment. To address these issues, the MCRA-Net designed in this invention embeds a category attention module into the backbone network for further processing of multimodal fusion features. Since category attention can effectively enhance the capture of intra-class information and help distinguish inter-class information, it has the ability to optimize the semantic information of fused feature categories in multimodal classification methods.

[0031] Furthermore, this invention proposes a multimodal information interaction module that fully integrates lesion features across different modalities, highlighting important lesion information shared by two modalities and further supplementing lesion information missed in a single modality. This generates a lesion heatmap to guide the spatial attention module for accurate lesion localization. Finally, the output of the spatial attention module is fed into a classifier containing a global average pooling operation and a fully connected layer for severity grading.

[0032] The MCRA-Net proposed in this invention has two significant advantages: (1) It utilizes color fundus images and fluorescein angiography images to provide grading decision-making basis under different imaging modes, achieving a grading of diabetic retinopathy that is more consistent with the clinical diagnostic process. (2) By combining category attention and spatial attention guided by the modal interaction module, it can simultaneously enhance the model's ability to extract intra-class information and important lesion information of fused features, thereby achieving more accurate grading results for diabetic retinopathy.

[0033] 2-category attention module

[0034] First, the feature map F output from the backbone network of the original image is processed by a 1×1 convolutional layer to generate the feature map Y∈ H×W×kL Where L is the number of categories, and k is the fixed number of channels allocated to learning features for each category. Then, a "channel splitting" operation is used to divide Y into L groups, i.e., Y′=[Y1′,…,Y L ′],Y i ′∈ H×W×k The intra-class features of each class are processed in parallel. Then, in each Y... i Dilated convolutions are used to increase the intra-class receptive field and reduce interference between inter-class information.

[0035] Next, the inverse operation of channel splitting is used to connect all branches into a unified feature map. Global average pooling and channel average pooling along the category dimension are then used to obtain the attention weights for each category, resulting in a category-weighted feature map S1. Finally, the feature map F of the category attention block is output using equation (1). category :

[0036]

[0037] 3-modal interactive information-guided spatial attention module

[0038] Multiple lesions are typically present on a single fundus image, but their importance varies significantly during grading. Paying closer attention to relatively important lesion information is crucial for accurate grading. Since microaneurysms are important lesions for distinguishing the early stages of diabetic retinopathy, neglecting their corresponding characteristics will inevitably lead to a decrease in grading accuracy. Considering that fluorescein angiography images and color fundus images contain complementary lesion information, the fusion and interaction of these two modalities can comprehensively model important pathological regions in the diabetic retinopathy grading task.

[0039] Therefore, to avoid missing lesion information with significant grading value, this invention designs a modal interaction-guided spatial attention mechanism in MCRA-Net to accurately locate important lesions. This modal interaction-guided spatial attention generates a lesion heatmap through a multimodal feature information interaction module, which then guides the spatial attention module in lesion region localization. By combining lesion information from different modalities, such as clearly characterized microaneurysms and non-perfused capillary areas in fluorescein angiography images, and easily extractable exudates in color fundus images, MCRA-Net achieves better lesion capture capabilities. Figure 3 and Figure 4 The internal structural details of the multimodal information interaction module and the spatial attention module are shown respectively.

[0040] The spatial attention mechanism guided by modal interaction information comprises two parts: First, it models the feature dependencies between modalities using a proposed multimodal information interaction module, enhancing attention to shared key lesions and extracting contextual relevance. Then, it utilizes spatial attention to incorporate the key lesion locations captured by the multimodal information interaction module into an attention-weighted fusion process, achieving accurate localization of key lesions. The specific process is as follows:

[0041] First, in the multimodal information interaction module, the feature information F extracted from the color fundus image is... CF As the main feature, the fluorescein angiography image feature F, which has similar semantics and contains important supplementary information, is utilized. FFA Generate interactive weighted features W1.

[0042] Then, the importance of each channel in the main feature is adjusted by the interactive weighted feature W1, and the adjusted feature map F′ is then generated. interaction A lesion heatmap M is generated using 1×1 convolution to guide spatial attention for locating key lesion regions.

[0043]

[0044] M = Conv 1×1 (F′interaction (3)

[0045] Spatial attention captures the spatial correlation between features, enabling the network to focus on key lesion feature regions and suppress irrelevant regions. In the spatial attention branch, the multimodal fusion feature F, which has been optimized by category attention, is first processed using Equation (4). category Average pooling is performed along the channel dimension, and spatial feature description W2 is obtained by applying the sigmoid activation function:

[0046] W2 = f sigmoid (CGAP(F category (4)

[0047] Where CGAP represents the average pooling operation in the channel dimension. Next, the feature description W2 is adjusted from the lesion heatmap M obtained from the multimodal information interaction module by equation (5) to obtain the final score W3 in the spatial dimension, and the output F of the spatial attention branch is obtained by equation (6). spatial :

[0048]

[0049]

[0050] Guided by the lesion heatmap M generated by the multimodal information interaction module, spatial attention can focus on lesion feature information in multiple modalities, thereby more accurately locating key lesion positions. Finally, F spatial The severity level of diabetic retinopathy is generated by global average pooling and fully connected layers.

[0051] The multimodal category-relation attention network designed in this invention adds a spatial attention module guided by modal interaction information after the category attention module. This enables the modeling of key pathological information and then guides spatial attention to achieve accurate lesion area localization. It is more suitable for multimodal grading models and can improve the accuracy of grading and class consistency.

Claims

1. A multimodal diabetic retinopathy image lesion grading method, comprising a backbone network, a category attention module, a spatial attention module guided by modal interaction information, and a classifier, wherein the backbone network consists of two independent backbone networks, which extract features from the read-in color fundus images and fluorescein angiography images, respectively; The category attention module uses the features of color fundus images and fundus fluorescein angiography images to obtain the attention weight of each category, resulting in a category-weighted feature map, which is then sent to the spatial attention module guided by modal interaction information. The modal interaction information-guided spatial attention module models the feature dependencies between modalities and extracts contextual relevance through the modal information interaction module. Then, it uses spatial attention to incorporate the key lesion locations captured by the modal information interaction module into the attention-weighted fusion process to achieve accurate localization of key lesions. The classifier feeds the output of spatial attention into a classifier containing a global average pooling operation and a fully connected layer for severity classification. The spatial attention module guided by modal interaction information comprises two parts: First, it models the feature dependencies between modalities through the proposed multimodal information interaction module, enhances attention to common key lesions, and extracts contextual relevance. Then, it uses spatial attention to incorporate the key lesion locations captured by the multimodal information interaction module into the attention-weighted fusion process, achieving accurate localization of key lesions. The specific process is as follows: First, in the multimodal information interaction module, the feature information extracted from the color fundus image is... As the primary feature, fluorescein angiography image features with similar semantics and containing important supplementary information are utilized. Generate interactive weighted features ; Then through interactive weighted features The importance of each channel in the main feature is adjusted to obtain the following: , The calculation process is as follows: The adjusted feature map Generating lesion heatmaps using 1×1 convolution. Used to guide spatial attention to locate key lesion areas. The calculation process is as follows: Spatial attention captures the spatial correlation between features, enabling the network to focus on key lesion feature regions and suppress irrelevant regions. In the spatial attention branch, the multimodal fusion features optimized by category attention are first processed. Average pooling is performed along the channel dimension, and spatial feature descriptions are obtained through the sigmoid activation function. : in This represents the average pooling operation along the channel dimension. Next, the lesion heatmap obtained from the multimodal information interaction module... Adjust feature description To obtain the final score in the spatial dimension. The output of the spatial attention branch is obtained. , and The calculation formula is as follows: The lesion heatmap generated by the multimodal information interaction module Guided by this principle, spatial attention focuses on lesion feature information across multiple modalities, thereby more accurately locating key lesion sites. Finally, The severity level of diabetic retinopathy is generated by global average pooling and fully connected layers.

2. The multimodal diabetic retinopathy image lesion grading method according to claim 1, characterized in that: The category attention module first processes the feature maps output by the backbone network through 1×1 convolutional layers. To generate feature maps ,in It is the number of categories. It allocates a fixed number of channels to learn each category feature, and then uses a "channel splitting" operation to... Divided into Group, i.e. , Process the intra-class features of each class in parallel, and then, in each Dilated convolutions are used to increase intra-class receptive fields and reduce interference between inter-class information. Then, the inverse operation of channel splitting is used to connect all branches into a unified feature map. Global average pooling and channel average pooling along the category dimension are then applied to obtain the attention weights for each category, resulting in a category-weighted feature map. The feature map of the category attention block is output by the following formula. ; 。

Images