Retinal blood vessel image segmentation method and system
By introducing the residual attention module and spatial activation module into the U-Net architecture, combined with skip connections and attention weighting, the accuracy and robustness of retinal vascular image segmentation are improved, the difficulty of small blood vessel segmentation is solved, and noise interference is reduced.
Patent Information
- Application Number
- CN202311125881.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-02
- Publication Date
- 2025-09-12
- Estimated Expiration
- 2043-09-02
AI Technical Summary
Existing retinal vessel image segmentation methods are prone to confusing segmented areas when processing fundus retinal images and are easily interfered by noise, especially in the segmentation of small blood vessels.
An image segmentation model based on the U-Net architecture is adopted, combined with the residual attention module and the spatial activation module. Through skip connections, channel attention weighting and pixel attention weighting, an attention matrix is generated to reduce noise interference and improve the model's segmentation performance for small blood vessels.
It effectively improves the problem of confused areas during image segmentation, improves segmentation performance, especially the segmentation effect of small blood vessels, reduces noise interference, and alleviates the problems of gradient vanishing and gradient exploding in the model.
Smart Images
Figure CN117152090B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a retinal blood vessel image segmentation method and system. Background Art
[0002] The diagnostic process in ophthalmology relies on fundus images. Typically, medical equipment captures high-quality images of the patient's eye using non-invasive methods. These images contain a wealth of vascular information, such as morphology, thickness, and branching, which is crucial for diagnosing ophthalmic diseases. Due to the large number of ophthalmology patients and limited physician resources, manual segmentation of blood vessels is time-consuming and prone to errors, prolonging the patient's diagnostic cycle. In this context, artificial intelligence technology is particularly important, as it can improve physician efficiency, save manpower and resources, and contribute to patients' rapid recovery.
[0003] Retinal vascular image segmentation is an important medical image analysis technology that helps doctors accurately locate and diagnose blood vessels and other tissues in retinal images. It is widely used in medical diagnosis, surgical planning, simulation, and training. Over the past few decades, many researchers have devoted themselves to developing automated vascular image segmentation algorithms from various perspectives and strategies. Although research on retinal vascular image segmentation methods has been relatively extensive, there is still much room for improvement. Major issues include the different contrasts between fundus retinal images, the confusion when segmenting the intersection of blood vessels, and the susceptibility to lesions in certain areas. Therefore, there is an urgent need to provide a solution to improve these issues. Summary of the Invention
[0004] The purpose of the present invention is to provide a retinal vascular image segmentation method and system, which can pay more attention to the small blood vessels in the fundus retinal image, reduce the interference of noise, further improve the segmentation performance, and effectively improve the problem of easily confusing segmented areas during image segmentation.
[0005] The present invention provides a retinal blood vessel image segmentation method using the following technical solutions:
[0006] Acquiring image data and constructing a data set based on the image data;
[0007] A retinal vascular image segmentation model is constructed based on a U-Net architecture, wherein a residual attention module is embedded between sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and a spatial activation module is embedded in the final output layer of the decoding module; wherein the residual attention module is used to perform a jump connection between the input features and the input features subjected to the convolution operation, and then sequentially perform channel attention weighting and pixel attention weighting to generate an attention matrix, and then perform an element-by-element multiplication operation on the attention matrix and the input features before outputting the result;
[0008] Training the image segmentation model based on the data set to obtain a trained image segmentation model;
[0009] The image to be segmented is input into the trained image segmentation model to obtain a segmentation result.
[0010] The retinal vascular image segmentation method as described above, wherein the process of performing jump connection between the input feature and the input feature map subjected to the convolution operation comprises:
[0011] After inputting the input feature into the convolution layer for convolution operation, the first feature is output;
[0012] Perform a jump connection between the input feature and the first feature to obtain a second feature.
[0013] In the above-mentioned method for retinal vessel image segmentation, the process of performing channel attention weighting includes:
[0014] After the second feature undergoes global average pooling, convolution and activation operations, the channel attention weight is output;
[0015] Perform an element-by-element multiplication operation on the channel attention weight and the second feature to obtain a channel attention matrix.
[0016] In the above-mentioned retinal vessel image segmentation method, the second feature is subjected to a global average pooling process and is expressed as:
[0017] ;
[0018] in, Represents global space information, represents the global average pooling operation, Represents the second feature, The i-th row and j-th column of the c-th channel feature map representing the second feature, Indicates the height of the second feature, Indicates the width of the second feature.
[0019] In the above-mentioned method for retinal vessel image segmentation, the process of performing pixel attention weighting includes:
[0020] After convolution and activation operations on the channel attention matrix, the pixel attention weight is output;
[0021] Perform element-by-element multiplication on the pixel attention weight and the channel attention matrix to obtain an attention matrix.
[0022] The retinal vascular image segmentation method described above, wherein when the image segmentation model is trained based on the data set, the process of processing by the spatial activation module includes:
[0023] After convolution and activation operations are performed on the input of the last output layer of the decoding module, up-sampled features are obtained and used as input of the spatial activation module;
[0024] After the upsampled features are subjected to a 1×1 convolution operation, global average pooling and global maximum pooling operations are performed to obtain average pooling features and maximum pooling features respectively;
[0025] After performing element-by-element addition operation on the average pooling feature and the maximum pooling feature, the first fusion feature is obtained by activating the feature through a multi-layer perceptron and an activation function;
[0026] Performing an element-by-element multiplication operation on the first fused feature and the upsampled feature to obtain a second fused feature;
[0027] An activation feature is obtained by performing a jump connection between the second fusion feature and the upsampled feature.
[0028] In the above-mentioned retinal vascular image segmentation method, the process of constructing a retinal vascular image segmentation model based on the U-Net architecture includes:
[0029] Constructing a U-Net architecture, wherein the encoding module of the U-Net architecture includes a first encoding sub-block, a second encoding sub-block, a third encoding sub-block, and a fourth encoding sub-block, the decoding module of the U-Net architecture correspondingly includes a first decoding sub-block, a second decoding sub-block, a third decoding sub-block, and a fourth decoding sub-block, and embedding a spatial attention module in the first decoding sub-block;
[0030] Embedding a first residual attention module between the first coding sub-block and the first decoding sub-block, embedding a second residual attention module between the second coding sub-block and the second decoding sub-block, embedding a third residual attention module between the third coding sub-block and the third decoding sub-block, and embedding a fourth residual attention module between the fourth coding sub-block and the fourth decoding sub-block;
[0031] The output of the first residual attention module is fused with the output of the second coding sub-block and used as the input of the second residual attention module; the output of the second residual attention module is fused with the output of the third coding sub-block and used as the input of the third residual attention module; the output of the third residual attention module is fused with the output of the fourth coding sub-block and used as the input of the fourth residual attention module;
[0032] The output of the fourth residual attention module is fused with the aggregated scale feature maps and used as the input of the fourth decoding sub-block. The output of the fourth decoding sub-block is fused with the output of the third residual attention module and used as the input of the third decoding sub-block. The output of the third decoding sub-block is fused with the output of the second residual attention module and used as the input of the second decoding sub-block. The output of the second decoding sub-block is fused with the output of the first residual attention module and used as the input of the first decoding sub-block.
[0033] A retinal blood vessel image segmentation system as described above, comprising:
[0034] A data set construction module, configured to obtain image data and construct a data set based on the image data;
[0035] The residual attention module is used to perform jump connections on the input features and the input features after the convolution operation, and then perform channel attention weighting and pixel attention weighting in sequence to generate an attention matrix, and then perform element-by-element multiplication of the attention matrix with the input features and output the result;
[0036] A model construction module, which constructs a retinal vascular image segmentation model based on a U-Net architecture, wherein the residual attention module is embedded between the sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and the spatial activation module is embedded in the final output layer of the decoding module;
[0037] A training module, which trains the image segmentation model based on the data set to obtain a trained image segmentation model;
[0038] The segmentation module is used to input the image to be segmented into the trained image segmentation model to obtain a segmentation result.
[0039] The present invention provides a retinal blood vessel image segmentation method and system, which have the following beneficial effects:
[0040] 1. This paper proposes a residual attention module and combines it with the U-Net network architecture to build a segmentation model. This model can pay more attention to the small blood vessels in the fundus retinal images, reduce the interference of noise, further improve the segmentation performance, and effectively improve the problem of easily confused segmentation areas during image segmentation.
[0041] 2. The present invention also introduces a spatial activation module and adds skip connections, which effectively alleviates the problems of gradient vanishing and gradient exploding in the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A structural diagram of the residual attention module provided in an embodiment of the present invention;
[0043] Figure 2 A structural diagram of the attention module provided in an embodiment of the present invention;
[0044] Figure 3 A structural diagram of a space activation module provided in an embodiment of the present invention;
[0045] Figure 4 This is a structural diagram of the image segmentation model provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.
[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0048] An embodiment of the present invention provides a retinal blood vessel image segmentation method, comprising:
[0049] Acquiring image data and constructing a data set based on the image data;
[0050] A retinal vascular image segmentation model is constructed based on a U-Net architecture, wherein a residual attention module is embedded between sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and a spatial activation module is embedded in the final output layer of the decoding module; wherein the residual attention module is used to perform a jump connection between the input features and the input features subjected to the convolution operation, and then sequentially perform channel attention weighting and pixel attention weighting to generate an attention matrix, and then perform an element-by-element multiplication operation on the attention matrix and the input features before outputting the result;
[0051] Training the image segmentation model based on the data set to obtain a trained image segmentation model;
[0052] The image to be segmented is input into the trained image segmentation model to obtain a segmentation result.
[0053] In some embodiments, all experiments are performed on the DRIVE dataset.
[0054] In some embodiments, see Figure 1 , performing a jump connection between the input feature and the input feature map after the convolution operation, including:
[0055] After inputting the input feature into the convolution layer for convolution operation, the first feature is output;
[0056] Perform a jump connection between the input feature and the first feature to obtain a second feature.
[0057] In some embodiments, see Figure 2 , the process of performing channel attention weighting includes:
[0058] After the second feature undergoes global average pooling, convolution and activation operations, the channel attention weight is output;
[0059] Perform an element-by-element multiplication operation on the channel attention weight and the second feature to obtain a channel attention matrix.
[0060] In some embodiments, the second feature is subjected to a global average pooling process, which is expressed as:
[0061] ;
[0062] in, Represents global space information, represents the global average pooling operation, Represents the second feature, The i-th row and j-th column of the c-th channel feature map representing the second feature, Indicates the height of the second feature, Indicates the width of the second feature.
[0063] Specifically, the size of the second feature is C×H×W, and after global average pooling, a pooled feature of size C×1×1 is obtained.
[0064] Furthermore, after two 3×3 convolutional layers, ReLU and Sigmoid activation functions, the channel attention weights are obtained. The specific process can be expressed as:
[0065] ;
[0066] Where, represents the channel attention weight, represents the Sigmoid activation function, represents the convolution operation, represents the ReLU activation function, Represents pooled features.
[0067] In some embodiments, performing pixel attention weighting includes:
[0068] After convolution and activation operations on the channel attention matrix, the pixel attention weight is output;
[0069] Perform element-by-element multiplication on the pixel attention weight and the channel attention matrix to obtain an attention matrix.
[0070] Specifically, the channel attention matrix is processed by two 3×3 convolutional layers, ReLU and Sigmoid activation functions to obtain pixel attention weights. Specifically, the size of the channel attention matrix is C×H×W, and the size of the pixel attention weight is 1×H×W.
[0071] In some embodiments, see Figure 3 When the image segmentation model is trained based on the data set, the process of processing by the spatial activation module includes:
[0072] After convolution and activation operations are performed on the input of the last output layer of the decoding module, up-sampled features are obtained and used as input of the spatial activation module;
[0073] After the upsampled features are subjected to a 1×1 convolution operation, global average pooling and global maximum pooling operations are performed to obtain average pooling features and maximum pooling features respectively;
[0074] After performing element-by-element addition operation on the average pooling feature and the maximum pooling feature, the first fusion feature is obtained by activating the feature through a multi-layer perceptron and an activation function;
[0075] Performing an element-by-element multiplication operation on the first fused feature and the upsampled feature to obtain a second fused feature;
[0076] An activation feature is obtained by performing a jump connection between the second fusion feature and the upsampled feature.
[0077] Specifically, the processing process of the spatial activation module can be expressed as:
[0078] ;
[0079] Where, Indicates the activation feature, represents the upsampled features, represents the Sigmoid activation function, represents global average pooling, Represents global maximum pooling.
[0080] In some embodiments, see Figure 4 The process of building a retinal vascular image segmentation model based on the U-Net architecture includes:
[0081] Constructing a U-Net architecture, wherein the encoding module of the U-Net architecture includes a first encoding sub-block, a second encoding sub-block, a third encoding sub-block, and a fourth encoding sub-block, the decoding module of the U-Net architecture correspondingly includes a first decoding sub-block, a second decoding sub-block, a third decoding sub-block, and a fourth decoding sub-block, and embedding a spatial attention module in the first decoding sub-block;
[0082] Embedding a first residual attention module between the first coding sub-block and the first decoding sub-block, embedding a second residual attention module between the second coding sub-block and the second decoding sub-block, embedding a third residual attention module between the third coding sub-block and the third decoding sub-block, and embedding a fourth residual attention module between the fourth coding sub-block and the fourth decoding sub-block;
[0083] The output of the first residual attention module is fused with the output of the second coding sub-block and used as the input of the second residual attention module; the output of the second residual attention module is fused with the output of the third coding sub-block and used as the input of the third residual attention module; the output of the third residual attention module is fused with the output of the fourth coding sub-block and used as the input of the fourth residual attention module;
[0084] The output of the fourth residual attention module is fused with the aggregated scale feature maps and used as the input of the fourth decoding sub-block. The output of the fourth decoding sub-block is fused with the output of the third residual attention module and used as the input of the third decoding sub-block. The output of the third decoding sub-block is fused with the output of the second residual attention module and used as the input of the second decoding sub-block. The output of the second decoding sub-block is fused with the output of the first residual attention module and used as the input of the first decoding sub-block.
[0085] Specifically, the decoding module uses a bilinear interpolation method to upsample, adjusts to the size of the original image, and then inputs it into the spatial activation module.
[0086] In some embodiments, a retinal vessel image segmentation system includes:
[0087] A data set construction module, configured to obtain image data and construct a data set based on the image data;
[0088] The residual attention module is used to perform jump connections on the input features and the input features after the convolution operation, and then perform channel attention weighting and pixel attention weighting in sequence to generate an attention matrix, and then perform element-by-element multiplication of the attention matrix with the input features and output the result;
[0089] A model construction module, which constructs a retinal vascular image segmentation model based on a U-Net architecture, wherein the residual attention module is embedded between the sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and the spatial activation module is embedded in the final output layer of the decoding module;
[0090] A training module, which trains the image segmentation model based on the data set to obtain a trained image segmentation model;
[0091] The segmentation module is used to input the image to be segmented into the trained image segmentation model to obtain a segmentation result.
[0092] Performance testing
[0093] The image segmentation model provided by the present invention is consistent with Mo (Mo J, Zhang L. Multi-level deepsupervised networks for retinal vessel segmentation[J]. International journal of computer assisted radiology and surgery, 2017, 12: 2181-2193), Residual U-Net (Alom MZ, Hasan M, Yakopcic C, et al. Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image arXiv preprint arXiv:1802.06955, 2018.), Yan(Yan Z, Yang et al. Attention guided u-net with atrous convolution for accurate retinal vessels segmentation[J]. IEEE Access, 2020, 8: 32826-32839.) and CHEN(Chen J, Lu Y, Yu Q, et al. Transunet: Transformers make strongencoders for medical image segmentation[J]. 2021, arXiv preprint arXiv:2102.04306.) were compared on the DRIVE dataset. The results are shown in Table 1.
[0094]
[0095] It can be seen that the method provided by the present invention has improved both AUC and ACC indicators compared to other methods. However, the Residual U-Net segmentation results for the DRIVE dataset showed low sensitivity and high specificity. This phenomenon may be due to the network structure's poor segmentation of tiny edge vessels and excessive noise. The method provided by the present invention utilizes a residual attention module to reduce information loss during the downsampling process. The attention mechanism enables the model to focus more on tiny vessels in fundus images, reducing noise interference and further improving segmentation performance.
[0096] While the embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations of these embodiments are possible. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the claims. Furthermore, the invention described herein is susceptible to other embodiments and may be practiced or implemented in a variety of ways.
Claims
1. A retinal vascular image segmentation method, characterized in that: The following steps are involved: Acquiring image data and constructing a data set based on the image data; A retinal vascular image segmentation model is constructed based on a U-Net architecture, wherein a residual attention module is embedded between sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and a spatial activation module is embedded in the final output layer of the decoding module; wherein the residual attention module is used to perform a jump connection between the input features and the input features subjected to the convolution operation, and then sequentially perform channel attention weighting and pixel attention weighting to generate an attention matrix, and then perform an element-by-element multiplication operation on the attention matrix and the input features before outputting the result; Training the image segmentation model based on the data set to obtain a trained image segmentation model; Inputting the image to be segmented into the trained image segmentation model to obtain a segmentation result; The process of building a retinal vascular image segmentation model based on the U-Net architecture includes: Constructing a U-Net architecture, wherein the encoding module of the U-Net architecture includes a first encoding sub-block, a second encoding sub-block, a third encoding sub-block, and a fourth encoding sub-block, the decoding module of the U-Net architecture correspondingly includes a first decoding sub-block, a second decoding sub-block, a third decoding sub-block, and a fourth decoding sub-block, and embedding a spatial attention module in the first decoding sub-block; Embedding a first residual attention module between the first coding sub-block and the first decoding sub-block, embedding a second residual attention module between the second coding sub-block and the second decoding sub-block, embedding a third residual attention module between the third coding sub-block and the third decoding sub-block, and embedding a fourth residual attention module between the fourth coding sub-block and the fourth decoding sub-block; The output of the first residual attention module is fused with the output of the second coding sub-block and used as the input of the second residual attention module; the output of the second residual attention module is fused with the output of the third coding sub-block and used as the input of the third residual attention module; the output of the third residual attention module is fused with the output of the fourth coding sub-block and used as the input of the fourth residual attention module; The output of the fourth residual attention module is fused with the aggregated scale feature maps and used as the input of the fourth decoding sub-block. The output of the fourth decoding sub-block is fused with the output of the third residual attention module and used as the input of the third decoding sub-block. The output of the third decoding sub-block is fused with the output of the second residual attention module and used as the input of the second decoding sub-block. The output of the second decoding sub-block is fused with the output of the first residual attention module and used as the input of the first decoding sub-block.
2. The retinal blood vessel image segmentation method according to claim 1, characterized in that: The process of performing a jump connection between the input feature and the input feature map after the convolution operation includes: After inputting the input feature into the convolution layer for convolution operation, the first feature is output; Perform a jump connection between the input feature and the first feature to obtain a second feature.
3. The retinal blood vessel image segmentation method according to claim 2, characterized in that: The process of performing channel attention weighting includes: After the second feature undergoes global average pooling, convolution and activation operations, the channel attention weight is output; Perform an element-by-element multiplication operation on the channel attention weight and the second feature to obtain a channel attention matrix.
4. The retinal blood vessel image segmentation method according to claim 3, characterized in that: The second feature is subjected to the global average pooling process and is expressed as: ； in, Represents global space information, represents the global average pooling operation, Represents the second feature, The i-th row and j-th column of the c-th channel feature map representing the second feature, Indicates the height of the second feature, Indicates the width of the second feature.
5. The retinal blood vessel image segmentation method according to claim 3, characterized in that: The process of performing pixel attention weighting includes: After convolution and activation operations on the channel attention matrix, the pixel attention weight is output; Perform element-by-element multiplication on the pixel attention weight and the channel attention matrix to obtain an attention matrix.
6. The retinal blood vessel image segmentation method according to claim 1, characterized in that: When the image segmentation model is trained based on the data set, the process of processing by the spatial activation module includes: After convolution and activation operations are performed on the input of the last output layer of the decoding module, up-sampled features are obtained and used as input of the spatial activation module; After the upsampled features are subjected to a 1×1 convolution operation, global average pooling and global maximum pooling operations are performed to obtain average pooling features and maximum pooling features respectively; After performing element-by-element addition operation on the average pooling feature and the maximum pooling feature, the first fusion feature is obtained by activating the feature through a multi-layer perceptron and an activation function; Performing an element-by-element multiplication operation on the first fused feature and the upsampled feature to obtain a second fused feature; An activation feature is obtained by performing a jump connection between the second fusion feature and the upsampled feature.
7. A retinal vascular image segmentation system, used to implement the retinal vascular image segmentation method according to any one of claims 1 to 6, characterized in that: include: A data set construction module, configured to obtain image data and construct a data set based on the image data; The residual attention module is used to perform jump connections on the input features and the input features after the convolution operation, and then perform channel attention weighting and pixel attention weighting in sequence to generate an attention matrix, and then perform element-by-element multiplication of the attention matrix with the input features and output the result; A model construction module, which constructs a retinal vascular image segmentation model based on a U-Net architecture, wherein the residual attention module is embedded between the sub-blocks corresponding to the encoding module and the decoding module in the image segmentation model, and the spatial activation module is embedded in the final output layer of the decoding module; A training module, which trains the image segmentation model based on the data set to obtain a trained image segmentation model; A segmentation module is used to input the image to be segmented into the trained image segmentation model to obtain a segmentation result; The process of building a retinal vascular image segmentation model based on the U-Net architecture includes: Constructing a U-Net architecture, wherein the encoding module of the U-Net architecture includes a first encoding sub-block, a second encoding sub-block, a third encoding sub-block, and a fourth encoding sub-block, the decoding module of the U-Net architecture correspondingly includes a first decoding sub-block, a second decoding sub-block, a third decoding sub-block, and a fourth decoding sub-block, and embedding a spatial attention module in the first decoding sub-block; Embedding a first residual attention module between the first coding sub-block and the first decoding sub-block, embedding a second residual attention module between the second coding sub-block and the second decoding sub-block, embedding a third residual attention module between the third coding sub-block and the third decoding sub-block, and embedding a fourth residual attention module between the fourth coding sub-block and the fourth decoding sub-block; The output of the first residual attention module is fused with the output of the second coding sub-block and used as the input of the second residual attention module; the output of the second residual attention module is fused with the output of the third coding sub-block and used as the input of the third residual attention module; the output of the third residual attention module is fused with the output of the fourth coding sub-block and used as the input of the fourth residual attention module; The output of the fourth residual attention module is fused with the aggregated scale feature maps and used as the input of the fourth decoding sub-block. The output of the fourth decoding sub-block is fused with the output of the third residual attention module and used as the input of the third decoding sub-block. The output of the third decoding sub-block is fused with the output of the second residual attention module and used as the input of the second decoding sub-block. The output of the second decoding sub-block is fused with the output of the first residual attention module and used as the input of the first decoding sub-block.