A variant UNet retinal segmentation method based on attention mechanism
By introducing attention mechanism and SoftPool pooling into UNet retinal segmentation method, the problem of complex structure segmentation of retinal vessels is solved, segmentation accuracy and detail retention are improved, and the effect of retinal vascular segmentation is achieved.
Patent Information
- Application Number
- CN202310034105.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-01-10
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-01-10
AI Technical Summary
The complex structure and high curvature of retinal blood vessels make it difficult to segment the existing technology with high precision, affecting the diagnostic effect of doctors.
Using the mutation UNet retinal segmentation method based on attention mechanism, the convolution attention module, cross-fusion channel attention module and improved attention gate model are used, and the Encode-Decode structure and SoftPool pooling method are used to enhance the channels of retinal blood vessel characteristics. and spatial information processing to reduce information loss.
It significantly improves the segmentation accuracy and detail retention of retinal vascular images, effectively solves the problem of insufficient feature extraction capabilities in the U-Net algorithm, and ensures the integrity and continuity of vascular segmentation.
Smart Images

Figure CN116051594B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a retinal segmentation method, in particular to a variant UNet retinal segmentation method based on an attention mechanism. Background Art
[0002] Since the retinal blood vessels in medical images are crucial in the systemic blood circulation, most fundus diseases are relatively easy to track and diagnose, and changes in their structure can reflect the impact of diabetic retinopathy (DR) on the human body. Therefore, whether the distribution and detailed information of retinal blood vessels can be observed intuitively has a very important impact on the doctor's diagnosis. However, the structure of retinal blood vessels is very complex, with high curvature and various shapes. The present invention adds an attention mechanism on the basis of deep learning technology, and proposes a variant UNet network model (abbreviated as AMV-UNet) to solve the problem of high-precision retinal segmentation. The experimental results show that the method of the present invention has achieved good segmentation effect in segmenting retinal blood vessel images. Summary of the invention
[0003] To address the above problems, a variant UNet retinal segmentation method based on attention mechanism was proposed to solve the above technical problems.
[0004] A variant UNet retinal segmentation method based on an attention mechanism is characterized by: comprising a network model, an attention module and an improved attention gate model; the attention module also comprises a convolutional attention module, a cross-fusion channel attention module and a multi-layer perceptron MLP module; the cross-fusion channel attention module is included in the convolutional attention module and is embedded in the tail of the convolutional attention module;
[0005] The network model is a four-layer Encode-Decode structure, and the network model includes an encoding module on the left, a skip connection module in the middle, and a decoding module on the right;
[0006] First, the encoding operation inputs the preprocessed fundus image into the encoding module of the network; the encoding module contains the convolutional attention module and the SoftPool downsampling pooling operation; there are three such operations in the encoding module: the image first passes through the convolutional attention module to extract features, and then uses the SoftPool downsampling operation;
[0007] Secondly, the decoding operation is performed in a symmetrical manner with the encoding module. This process also includes three deconvolution upsampling operations. Each step of the decoding module includes a convolution attention module and a deconvolution layer. The feature map is jump-connected with the corresponding feature map of the encoding module. In the feature connection process, the AG+ attention gate is added to enhance the feature map to strengthen the segmentation ability of the detail area and improve the segmentation accuracy.
[0008] Finally, the segmentation operation uses the fully connected layer and the excitation function to segment the retinal image blood vessels from the background and output the segmentation result;
[0009] In the encoding and decoding stage of the network, a structured convolutional attention module is used to perform channel enhancement on the retinal vascular features; then an improved attention gate module is introduced at the jump connection of the network to perform spatial enhancement on the retinal vascular features; then the SoftPool pooling method is used to reduce information loss during the downsampling process.
[0010] The beneficial effects of the present invention are as follows: the model method first uses a structured convolutional attention module (DC-Conv) to perform channel enhancement on retinal vascular features in the encoding and decoding stage of the network; then an improved attention gate (AG+) module is introduced at the jump connection of the network to perform spatial enhancement on retinal vascular features; then a SoftPool pooling method is used to reduce information loss during the downsampling process. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 Network model architecture;
[0012] Figure 2 Cross-fusion channel attention module;
[0013] Figure 3 .MLP module;
[0014] Figure 4 Attention gate model;
[0015] Figure 5 Fundus vascular medical images;
[0016] Figure 6 Comparative experiment;
[0017] Figure 7 A diagram showing the detailed segmentation effects of different algorithms on CHASEDB1 and DRIVE. DETAILED DESCRIPTION
[0018] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0019] like Figure 1 A variant UNet retinal segmentation method based on an attention mechanism includes a network model, an attention module and an improved attention gate (AG+) model; the attention module also includes a convolutional attention module (DC-Conv), a cross-fusion channel attention module and a multi-layer perceptron MLP module; the cross-fusion channel attention module is included in the convolutional attention module (DC-Conv), and is embedded in the tail of the convolutional attention module (DC-Conv);
[0020] Firstly, the knot convolution attention module (DC-Conv) is used to perform channel enhancement on the retinal vascular features in the encoding and decoding stage of the network. Then, the improved attention gate (AG+) module is introduced at the jump connection of the network to perform spatial enhancement on the retinal vascular features. Then, the SoftPool pooling method is used to reduce the information loss in the downsampling process.
[0021] The network model is a four-layer Encode-Decode structure, and the network model includes three parts, namely, the encoding module on the left, the skip connection module in the middle, and the decoding module on the right;
[0022] First, the encoding operation inputs the preprocessed fundus image into the network encoder; the encoder contains the convolutional attention module DC-Conv and the SoftPool downsampling pooling operation. There are three such operations in the encoder: the image first passes through the DC-Conv convolutional attention module to extract features, and then uses SoftPool for downsampling.
[0023] Secondly, the decoding operation is performed in a symmetrical manner with the encoder. This process also includes three deconvolution upsampling operations. Each step of the decoder contains a DC-Conv and a deconvolution layer. The feature map is jump-connected with the corresponding encoder feature map. In the feature connection process, AG+ attention gate is added to enhance the feature map to strengthen the segmentation ability of the detail area and improve the segmentation accuracy.
[0024] Finally, in the decoding operation, the retinal image blood vessels are segmented from the background using the fully connected layer and the excitation function, and the segmentation result is output.
[0025] like Figure 2 The network model of the present invention integrates the cross-fusion channel attention module, the improved attention gate (AG+) model and the SoftPool downsampling pooling method. The following will introduce these three parts in detail.
[0026] Cross-fusion channel attention module:
[0027] In order to solve the problem of inconsistent thickness of retinal blood vessels, enhance the local detail information of the image, and make it have a local receptive field, the present invention uses the global maximum pooling to compress the feature map; then the global and average pooling results are sent to the multi-layer perceptron MLP module with feature crossover mechanism for learning, the results are output and added; finally, the final channel attention weight is obtained after mapping by sigmoid function. Figure 2The cross-fusion channel attention module shown in the figure puts the global mean feature and the maximum pooling feature weight into the Product layer in a certain combination for cross-multiplication, so that the global and local features in the retinal blood vessels are cross-learned to compensate for the local detail features. This module is mainly divided into five steps:
[0028] (1) Squeeze: Define a feature map with M feature channels and obtain a tensor F formed by global maximum pooling and average pooling max and F avg , Sigmoid is used to obtain the weight tensor A of the feature channel of the global maximum pooling and average pooling thickness respectively max and A avg .
[0029]
[0030] (2) Feature Cross: The four tensors obtained after compression are sent to the Product layer for four feature cross operations to fuse global and local features. The four feature cross operations result in four tensors, as shown in the specific formula:
[0031]
[0032] (3) Excitation: These four tensors are sent to the MLP module for feature learning. The MLP in this paper shares weight coefficients. Figure 3 The figure shows a MLP module with shared weights. After the MLP module, four Attention coefficients are obtained, namely A1, A2, A3, and A4.
[0033]
[0034] (4) Element-wise addition: In order to make the dependencies between channels contain both global features and local detail features, the obtained 4 channel importance weights are weighted, and the Sigmoid function is used to make the new channel feature weights between 0 and 1. The obtained new channel feature weights are expressed as:
[0035]
[0036] (5) Recalibration (Sale): Use the feature weight A to recalibrate the original feature map to obtain a new feature map F'
[0037]
[0038] SoftPool pooling method:
[0039] It is a new pooling method based on softmax normalization that can be used to downsample 2D (image) and 3D (video) activation maps. It uses a softmax weighted sum of the activations. SoftPool largely preserves the descriptive activation characteristics while remaining computationally and memory efficient. Models containing SoftPool consistently show improved classification performance compared to the original implementation due to better feature preservation.
[0040] SoftPool uses a smooth maximum approximation of the activations within the kernel region R, where each activation a with index i i A weight w is applied to each i , the weight w i It is calculated as the ratio of the natural index of this activation to the sum of the natural indexes of all activations in the neighborhood R:
[0041]
[0042] Output value of SoftPool operation is generated by the normalized sum of all weighted activations within the kernel neighborhood R:
[0043]
[0044] Compared to other pooling methods based on maximum and average, using region-based softmax produces a normalized result with a probability distribution that is proportional to the value of each activation relative to the neighboring activations of the kernel region. This is in direct contrast to the popular selection of the maximum activation value or averaging all activations over the kernel region, where the output activations are not normalized.
[0045] Convolutional Attention Module:
[0046] The model designed by the present invention uses a convolutional attention module (DC-Conv module) to replace the double-layer convolution structure of the traditional U-Net codec. The DC-Conv module contains two sub-convolution blocks and cross-fusion channel attention. Each sub-convolution in the structure of the sub-convolution block is composed of 3×3-Conv, DropBlock, BN, and ReLU in sequence. The present invention adds a DropBlock layer, a batch normalization layer, and a fusion channel attention to the basic double-layer convolution structure of UNet, wherein DropBlock is used to regularize the network. The model using DropBlock is more adaptable to different semantic segmentation scenarios, and has stronger sensitivity and robustness. Retinal vascular images often have insufficient brightness in a certain area and small and difficult-to-capture blood vessels. Using the DropBlock model will make it easier to learn a more complete vascular structure and the bifurcation characteristics of the blood vessels. The role of the BN layer is to maintain the stability of the input and output data distribution and reduce the model's dependence on the initial input data. The purpose of adding the channel fusion attention mechanism is to enable the network model to take into account both global and local features, perform channel enhancement on retinal blood vessels, and improve segmentation accuracy.
[0047] Improved Attention Gate (AG+) module:
[0048] In order to highlight the details of microvessels, the present invention introduces a spatial attention mechanism to spatially enhance microvessels. Inspired by Attention U-Net, the present invention conducts innovative research and names it AG+ (Attention Gate+). Figure 4 shown.
[0049] This attention mechanism takes two feature maps as input: the first is the feature g during upsampling; the second is the feature x during skip connection. l The two input features are added after a 1×1×1 convolution operation to obtain a feature map with the same size and C channels. The intermediate feature map is then obtained through the ReLU activation function, and then the attention coefficient α is obtained after a 1×1 convolution operation, a Sigmoid activation function and resampling. To output the feature map, its expression is:
[0050]
[0051] In formula (7), x l is the input feature, l represents the number of pixels of each feature. The attention coefficient α is
[0052] α=σ2(ψ(σ1(ω x x l +ω g gl +b g ))+b ψ ) (8)
[0053] ω i (i=x,g) are the input features x l and g l The weight of ψ is the standard convolution function; b g g l The bias value of b ψ is the bias value of ψ. The input features provide contextual information to the attention mechanism, which can determine which input features are related to retinal blood vessels. It is a weighted operation on low-level features to highlight the importance of retinal blood vessels.
[0054] In the segmentation task, due to the existence of multiple semantic categories, the present invention learns the coefficient of multi-dimensional attention, so the attention mechanism can pay more attention to the main situation in the segmentation process. Therefore, the segmentation accuracy of the present invention using the additive attention algorithm is higher and the segmentation effect is better.
[0055] Experimental data
[0056] Dataset:
[0057] This paper uses two public datasets, CHASEDB1 and DRIVE. The information of the two fundus retinal vascular image datasets is as follows: Figure 5 As shown:
[0058] In order to demonstrate the superiority of the algorithm in this paper, this paper compares the proposed algorithm with other algorithms on the CHASEDB1 and DRIVE datasets. All segmentation results are obtained in the same experimental environment. Figure 6 As shown in the comparative experiment, (a) is the original fundus image; (b) is the gold standard for segmentation; (c) is the segmentation result of the algorithm in this paper; (d) is the segmentation result of the U-Net network. The original U-Net codec uses the traditional continuous convolution and jump connection structure. The first and second rows give the CHASEDB1 retinal images and the segmentation results of each network, and the third and fourth rows give the DRIVE retinal images and the segmentation results of each network.
[0059] Figure 7 The diagram shows the effect of different algorithms on the detailed segmentation of CHASEDB1 and DRIVE, (a) original image (b) original image details (c) label details (d) segmentation result of this algorithm (e) U-Net segmentation result.
[0060] In comparison, the segmentation algorithm in the present invention can well segment some of the tiny blood vessels lost when the U-Net algorithm segmented the retinal blood vessels, and retain more details in the blood vessels. Therefore, the results verify that the algorithm model in the present invention can effectively solve the problem of insufficient feature extraction capability in the U-Net algorithm, and ensure the integrity and continuity of blood vessel segmentation.
[0061] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the elements of the claims be included in the present invention. Any reference numeral in a claim should not be regarded as limiting the claim to which it relates.
Claims
1. A variant UNet retinal segmentation method based on attention mechanism, characterized by: It includes a network model, an attention module and an improved attention gate model; the attention module also includes a convolutional attention module, a cross-fusion channel attention module and a multi-layer perceptron MLP module; the cross-fusion channel attention module is included in the convolutional attention module and is embedded in the tail of the convolutional attention module; the network model integrates the cross-fusion channel attention module, the improved attention gate model and the SoftPool pooling method; The feature map is compressed using the global maximum pooling. The global and average pooling results are then sent to the multi-layer perceptron MLP module with a feature crossover mechanism for learning, and the results are output and added. Finally, the final channel attention weight is obtained after mapping through the sigmoid function. The cross-fusion channel attention module puts the global mean feature and the maximum pooling feature weight into the Product layer in a certain combination for cross-multiplication, so that the global and local features in the retinal blood vessels are cross-learned to compensate for the local detail features; The improved attention gate model takes two feature maps as input: the first is the feature g in the upsampling process; The second is the feature x during skip connection l ; After the two input features are subjected to a 1×1×1 convolution operation, a feature map with the same size and C channels is obtained, and then an addition operation is performed; then the intermediate feature map is obtained through the ReLU activation function, and then a 1×1 convolution operation, a Sigmoid activation function and resampling are performed to obtain the attention coefficient α; The network model is a four-layer Encode-Decode structure, and the network model includes an encoding module on the left, a skip connection module in the middle, and a decoding module on the right; First, the encoding operation inputs the preprocessed fundus image into the encoding module of the network; the encoding module contains the convolutional attention module and the SoftPool downsampling pooling operation; there are three such operations in the encoding module: the image first passes through the convolutional attention module to extract features, and then uses the SoftPool downsampling operation; Secondly, the decoding operation is performed to perform the decoding module operation symmetrical to the encoding module; this process also includes three deconvolution upsampling operations, and each step of the decoding module includes a convolution attention module and a deconvolution layer, and the feature map is jump-connected with the corresponding encoding module feature map. The improved attention gate model is added in the feature connection process to enhance the feature map to strengthen the segmentation ability of the detail area and improve the segmentation accuracy; Finally, the segmentation operation uses the fully connected layer and the excitation function to segment the retinal image blood vessels from the background and output the segmentation result; In the encoding and decoding stage of the network, a convolutional attention module is used to enhance the channel of retinal vascular features; then an improved attention gate module is introduced at the jump connection of the network to spatially enhance the retinal vascular features; then the SoftPool pooling method is used to reduce the information loss in the downsampling process; The cross-fusion channel attention module is divided into five steps: (1) Squeeze: Define a feature map with M feature channels and obtain a tensor F formed by global maximum pooling and average pooling max and F avg , Sigmoid is used to obtain the weight tensor A of the feature channel of the global maximum pooling and average pooling thickness respectively max and A avg ; A max =Sigmoid(F max )(1) A avg =Sigmoid(F avg ) (2) (2) Feature Crossover: The four tensors obtained after compression are sent to the Product layer for four feature crossovers to fuse global and local features. Four tensors are obtained after four feature crossover operations. The specific formula is as follows: (3) Stimulation: These four tensors are sent to the MLP module for feature learning. The MLP module shares weight coefficients. After passing through the MLP module, four attention coefficients are obtained, namely A1, A2, A3, A4, (4) Weighted summation of features: To ensure that the dependencies between channels include both global features and local detail features; The obtained four channel importance weights are weighted, and the Sigmoid function is used to make the new channel feature weights between 0 and 1. The obtained new channel feature weights are expressed as: A=Sigmoid(A1+A2+A3+A4)(5) (5) Recalibration: Use the feature weight A to recalibrate the original feature map to obtain a new feature map F' 2. According to claim 1, a variant UNet retinal segmentation method based on attention mechanism is characterized in that: The convolutional attention module contains two sub-convolution blocks and cross-fusion channel attention. Each sub-convolution in the structure of the sub-convolution block consists of 3×3-Conv, DropBlock, BN, and ReLU in sequence. The DropBlock layer, batch normalization layer and fused channel attention are added to the basic two-layer convolutional structure of UNet, and DropBlock is used to regularize the network.
Citation Information
Patent Citations
Retinal blood vessel image segmentation method based on double-channel U-shaped improved Transform network
CN114820632A
Retina image blood vessel segmentation method based on improved U-Net network
CN114881962A
Anchor-free polyp colonoscopy and pathological classification method
WO2022247486A1
