A Multimodal Attention-Based Segmentation Method for Retinal Capillaries

Abstract

A multimodal attention-based refinement segmentation method for retinal capillaries addresses the problem that the original UNet model, lacking any filtering module, easily introduces new noise into the decoder, hindering accurate segmentation. The method includes the following steps: training a CycleGAN generative network on a multimodal retinal image dataset to learn the mapping relationship between a first color retinal angiography and a first fluorescein angiography, and synthesizing a second fluorescein angiography image; performing multi-scale Laplacian filtering on the second fluorescein angiography image to obtain a coarser vascular structure map; synchronously inputting the first color retinal angiography and the coarse vascular structure map into the UNet encoder to obtain prior features for auxiliary supervision; introducing a gated spatial attention mechanism on the UNet skip connections; and applying a contextual feature fusion attention mechanism to the multi-scale pyramid map generated by the UNet decoder.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to a multimodal attention-based thinning segmentation method for retinal capillaries. Background Technology

[0002] Studies have shown that various ophthalmic and cardiovascular diseases can cause varying degrees of deformation and hemorrhage in retinal vessels. With improved living standards, the incidence of these diseases is increasing year by year. Clinically, medical personnel can extract retinal vessels from color fundus images acquired through ophthalmoscopy and then diagnose these diseases by analyzing the morphology of the vessels. However, due to limitations in fundus image acquisition technology, images often contain a large amount of noise. Furthermore, the complex and variable structure of retinal vessels themselves makes retinal vessel segmentation extremely difficult. The most traditional method relies on manual segmentation of retinal vessels, which is not only extremely labor-intensive and time-consuming but also heavily influenced by subjective factors. Traditional methods utilize extracted known image features to obtain the desired vascular region image, achieving good results, but still suffer from problems such as manual parameter tuning and feature selection. Therefore, it is crucial to find an algorithm using computer technology that can quickly and accurately segment retinal vessels, enabling real-time extraction of vascular features from fundus images, which plays an important role in assisting medical personnel in diagnosing ophthalmic and cardiovascular diseases.

[0003] In the field of medical image segmentation, UNet is a common and well-known baseline network. Due to its excellent performance, many state-of-the-art methods for retinal vessel segmentation are based on UNet. DEU-Net significantly enhances the network's ability to segment retinal vessels using both end-to-end and pixel-to-pixel methods; VesselNet was the first to combine the advantages of initial and residual methods for retinal vessel segmentation. AG-Net employs a mechanism called an attention-guided filter to better preserve structural information. While these UNet variants perform well, they inevitably make the network more complex and harder to interpret. More importantly, the differences between vascular and non-vascular features in retinal fundus images are not significant, especially in vascular regions with fine edges. By introducing a few additional parameters to enhance important features (such as vascular features) and suppress unimportant features, the network's representational power can be improved.

[0004] While the skip connections in the original UNet model address the loss of low-level features caused by downsampling to some extent, they also introduce new noise into the decoder without any filtering module, making it difficult to achieve accurate segmentation. Summary of the Invention

[0005] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a multimodal attention-based segmentation method for retinal capillaries. This method effectively solves the problem that the original UNet model, lacking any filtering module, easily introduces new noise into the decoder, making it difficult to achieve accurate segmentation results.

[0006] To achieve the above objectives, the technical solution of the present invention is: a multimodal attention refinement segmentation method for retinal capillaries, comprising the following steps:

[0007] S1. Train the CycleGAN generative network on a multimodal retinal image dataset to learn the mapping relationship between the first color retinal angiography and the first fluorescein angiography, and synthesize the second fluorescein angiography image;

[0008] S2, Perform multi-scale Laplacian filtering on the synthesized second fluorescein angiography image to obtain a coarser vascular structure map;

[0009] S3, the first color retinal angiography and the rough vascular structure map are simultaneously input into the UNet encoder to obtain prior features for auxiliary supervision;

[0010] S4 introduces a gated spatial attention mechanism on UNet skip connections;

[0011] S5 applies a contextual feature fusion attention mechanism to the multi-scale pyramid map generated by the UNet decoder.

[0012] Preferably, the specific steps of S1 are as follows: The multimodal retinal angiography image dataset contains 59 pairs of unannotated color retinal angiography and fluorescein angiography images, each pair of images coming from the same patient. The color retinal angiography image is used as network input, and the corresponding fluorescein angiography image is used as the label to train a pre-defined CycleGAN generation network. The trained CycleGAN generation network learns the mapping relationship between the two by generating a first fluorescein angiography image based on the first color retinal angiography image, and synthesizes the corresponding second fluorescein angiography image on the DRIVE dataset.

[0013] Preferably, the specific step of S2 is as follows: performing a multi-scale Laplacian transform on the synthesized second fluorescein angiography image, wherein the Laplacian scale is matched with the width of the blood vessel, so that the two peaks in the blood vessel completely overlap, thereby enhancing the entire blood vessel region.

[0014] Preferably, the specific steps of S3 are as follows: the rough vascular structure map containing auxiliary supervision information is used as a priori features and simultaneously input into the same UNet encoder along with the first color retinal angiography, and the encoding vectors of the two are concatenated by channel.

[0015] Preferably, the specific steps of S4 are as follows: the input of the skip connection is split into two, a spatial attention module and a regular two-dimensional convolution are added to each, and a gating mechanism is added to them using the sigmoid function, that is, one is activated by the sigmoid function and the other is not activated, and then residuals are added to the entire structure.

[0016] Preferably, the specific steps of S5 are as follows: on the UNet decoder, the multi-level feature maps are gradually refined through a top-down path, and a fusion attention module that combines contextual features is introduced at each layer, so that the high-level features provide semantic and spatial information of the target entity for the low-level details.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1) Multimodal data was introduced to achieve complementary image information, present the retinal structure and the appearance of lesions from multiple angles, and minimize the objective impact of acquisition equipment and patient condition;

[0019] 2) The vascular structure map generated by image processing technology creates auxiliary supervision information, demonstrating the important guiding role of prior knowledge for the model;

[0020] 3) Introduce a gated attention mechanism on the UNet skip connections to achieve selective multi-channel transmission of fundus feature information;

[0021] 4) It avoids the indiscriminate feature extraction during the bilinear interpolation (upsampling) process of the UNet decoder, and captures the visual details of the target object under the guidance of high-level semantics, thus refining the segmentation results. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the refined segmentation method of the present invention;

[0023] Figure 2 A flowchart of the gating spatial attention mechanism for UNet jump connections. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. The drawings are simplified schematic diagrams, illustrating only the basic structure of the invention in a schematic manner, and therefore only show the components relevant to the invention.

[0025] like Figure 1-2As shown, a multimodal attention-based refinement segmentation method for retinal capillaries is proposed. This method deeply mines the correlation between multimodal data in the field of medical image processing and the pyramid feature maps generated by the intermediate steps of the commonly used network UNet. It utilizes multiple attention mechanisms to enhance effective information and suppress irrelevant noise, thereby enabling the model to pay attention to some capillary pixels that are difficult to detect or prone to errors, thus improving the segmentation accuracy. The specific method is as follows.

[0026] Step S1: The multimodal retinal angiography image dataset contains 59 pairs of unannotated color retinal angiography and fluorescein angiography images, each pair of images coming from the same patient. The color retinal angiography images are used as network input, and the corresponding fluorescein angiography images are used as labels to train a pre-defined CycleGAN generative network.

[0027] The trained CycleGAN generative network learns the mapping relationship between a first color retinal angiography and a first fluorescein angiography by generating the first fluorescein angiography from the first color retinal angiography. Then, it synthesizes the corresponding fluorescein angiography on the DRIVE dataset. Since fluorescein angiography is invasive, acquisition of images from many clinical sites is difficult. Therefore, this invention uses CycleGAN to synthesize fluorescein angiography images, allowing the multimodal attention-based segmentation method to still be implemented even when only color retinal angiography is available. Specifically, the generative model is trained on a multimodal retinal image dataset, which easily learns the mapping relationship between color retinal angiography and fluorescein angiography. The CycleGAN loss function is adversarial loss and cycle consistency loss, trained for 500 epochs at a learning rate of 0.0001. Then, the corresponding fluorescein angiography modality is synthesized in the DRIVE retinal fundus dataset to complete the learning process of the multimodal attention-based segmentation method.

[0028] Step S2: Perform multi-scale Laplacian filtering on the synthesized second fluorescein angiography image to obtain a coarser vascular structure map.

[0029] A Laplacian operator is convolved with a Gaussian filter to obtain a Laplacian-Gaussian filter. Given the varying widths of vessels in the entire fluorescein angiography image, applying the Laplacian operator at multiple scales yields multiple single-scale LoG response maps. These LoG response maps from different scales are then merged to obtain a multi-scale Laplacian map, where vessels of varying widths are enhanced. Specifically, for each pixel in the fluorescein angiography at different scales, the scale corresponding to the largest Laplacian response value is selected.

[0030] Step S3: The first color retinal angiography and the rough vascular structure map are simultaneously input into the UNet encoder to obtain prior features for auxiliary supervision.

[0031] For the same patient, two images of the same size—a color retinal angiography and a coarse vascular structure map (the latter containing more clearly defined vascular plexus structures)—are treated as a pair and simultaneously input into a UNet encoder with the same structure. This results in two 30×30 encoded vectors with 512 channels each. These two vectors are then concatenated by channel, followed by a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function layer, resulting in a single 30×30 encoded vector with 512 channels. This single encoded vector contains the auxiliary supervision information used for prior features.

[0032] Step S4: Introduce a gated spatial attention mechanism on the UNet skip connections.

[0033] This invention adds a gated spatial attention mechanism with residual structure to the original skip connection. First, the input X of the skip connection is split into two paths, which are then fed into two structures, each containing a 2D convolutional layer and a spatial attention layer. The gate mechanism, as the name suggests, uses a sigmoid function σ with a range of (0,1) to add a gate to the ordinary 2D convolution Conv2D.

[0034]

[0035] SA(X) represents the computation result of the spatial attention mechanism. Then, the two Conv2D convolutions are multiplied bit by bit. Although their kernel numbers, window sizes, etc., are the same, their weights are independent (not shared). One convolution is activated using the sigmoid function; the other is not activated, thus reducing the risk of gradient vanishing. When the input and output dimensions are the same, the gated attention mechanism can incorporate residuals and be represented in a more visually intuitive equivalent form.

[0036]

[0037] In the above equation, Conv2D1 is a linear transformation (without sigmoid activation), so during training, Conv2D1(SA(X))-SA(X) is equivalent to a single Conv2D1. Therefore, it can be transformed into:

[0038]

[0039] The above equation can be rewritten as:

[0040]

[0041] q=σ(Conv2D2(SA(X)))

[0042] The above derivation process demonstrates the selective multi-channel transmission of information: it passes directly with a probability of 1-q, and passes through after transformation with a probability of q.

[0043] Step S5: Apply a context feature fusion attention mechanism to the multi-scale pyramid map generated by the UNet decoder.

[0044] In the layer-by-layer bilinear interpolation (upsampling) process of the UNet decoder, this invention further fuses high-level semantic features with low-level visual features in the multi-scale pyramid map, thereby fusing the feature map G of the second image of the i-th layer in the UNet decoder. i A context-based attention mechanism is applied to extract detailed information about the target object. The calculation process is as follows:

[0045] A i =σ(Conv c (Conv a (G i )+Conv b (Up(G i-1 ))))

[0046]

[0047] Among them, A i Fuse attention weights to the context features of the i-th layer of the UNet decoder. a v b v c All are 1×1 convolution kernels. Up represents the deconvolution (upsampling) operation. G i-1 This represents the last feature map of the (i-1)th layer. Indicates to G i The computational results of this context-fusion attention mechanism are applied. In this top-down path, high-level abstract features with semantic and spatial information are integrated into the next layer of feature map after bilinear interpolation (upsampling), thereby guiding the model to assign more weights to the correct target region and refine the segmentation edges.

[0048] The embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that various variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom remain within the scope of this invention.

Claims

1. A multimodal attention-based fine segmentation method for retinal capillaries, characterized in that, Includes the following steps: S1. Train the CycleGAN generative network on the multimodal retinal image dataset to learn the mapping relationship between the first color retinal angiography and the first fluorescein angiography. Based on the color retinal angiography, use the CycleGAN generative network to synthesize the second fluorescein angiography image. S2, Perform a multi-scale Laplacian filtering operation on the second fluorescein angiography image to obtain a coarse vascular structure map; S3, the first color retinal angiography and the rough vascular structure map are synchronously input into the same UNet encoder. The encoding vectors of the two are concatenated by channel, and then passed through a 3×3 convolutional layer, a batch normalization layer and a ReLU activation function layer to obtain an encoding vector containing auxiliary supervision information for prior features; where the encoding vector for auxiliary supervision information for prior features is the input of the UNet decoder. S4, a UNet skip connection is set between the UNet encoder and UNet decoder, whose input is color retinal angiography. A gated spatial attention mechanism with residual structure is added to the UNet skip connection. This includes: splitting the input of the skip connection into two parts, adding a spatial attention module and a regular 2D convolution to each part, and adding a gating mechanism using the sigmoid function, i.e., one part is activated by the sigmoid function, while the other part is not activated. Then, a residual is added to the entire structure, including: in, The input represents the skip connection; The computational results represent the spatial attention mechanism; and Represents a two-dimensional convolutional layer; This belongs to a linear transformation without sigmoid activation; Indicates two Multiply digit by digit. Indicates the range is The sigmod function embodies selective multichannel transmission of information: The probability is directly passed through, with The probability is only passed after transformation; S5 applies a contextual feature fusion attention mechanism to the multi-scale pyramid map generated by the UNet decoder, including: fusing high-level semantic features with low-level visual features in the multi-scale pyramid map during the layer-by-layer bilinear interpolation process of the UNet decoder, and fusing the high-level semantic features with low-level visual features in the multi-scale pyramid map. The second feature map of the layer Applying a context-fusion attention mechanism to extract detailed information about the target object includes: in, For UNet decoder number Layer context features are fused with attention weights; , , All are 1×1 convolutional kernels; Indicates the deconvolution operation; Indicates the first The last feature map of the layer; Indicates to The computational results of this context-fusion attention mechanism are applied; in this top-down path, high-level abstract features with semantic and spatial information are integrated into the next layer of feature map after bilinear interpolation, thereby guiding the model to assign more weight to the correct target region and refine the segmentation edges.

2. The multimodal attention-based segmentation method for retinal capillaries according to claim 1, characterized in that, The specific steps of S1 are as follows: The multimodal retinal angiography image dataset contains 59 pairs of unannotated color retinal angiography and fluorescein angiography, each pair of images comes from the same patient; the color retinal angiography is used as the network input, and the corresponding fluorescein angiography is used as the label to train the pre-defined CycleGAN generative network; the network is trained for 500 epochs with a learning rate of 0.0001.

3. The multimodal attention-based segmentation method for retinal capillaries according to claim 1, characterized in that, S3 includes: for color retinal angiography and coarse vascular structure map of the same size from the same patient, these two images are treated as a pair and simultaneously input into a UNet encoder with the same structure to obtain two 30×30 encoded vectors with 512 channels each. After concatenating these two vectors according to channels, they are then passed through a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function layer to obtain a 30×30 encoded vector with 512 channels. This encoded vector contains auxiliary supervision information for prior features.

Images