Retinal vessel image segmentation method based on edge and detail co-conducting topology perception

Abstract

The present application belongs to the technical field of medical image semantic segmentation method, aiming at solving the problem of low precision of existing algorithm in extracting blood vessels and serious loss of detail information. A retinal blood vessel image segmentation method based on edge and detail co-conduction topological perception is provided, comprising the following steps: building a retinal blood vessel image segmentation network initial model; training and iterating the retinal blood vessel image segmentation network initial model, verifying to obtain a retinal blood vessel image segmentation network model; collecting retinal blood vessel image data for segmentation and image processing, inputting into the retinal blood vessel image segmentation network model, and the retinal blood vessel image segmentation network model outputting corresponding fundus retinal image semantic segmentation result. The present application solves the problems of easy interference of similar external tissues, blurred boundary between blood vessels and surrounding tissues, uneven illumination and blood spot shielding, and serious loss of detail information in extracting blood vessels from fundus retinal image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image semantic segmentation methods, and particularly relates to a retinal blood vessel image segmentation method based on edge and detail co-guided topological perception. BACKGROUND

[0002] Analyzing blood vessel structure from fundus images has been widely used in the diagnosis of various ophthalmic diseases caused by arteriosclerosis, diabetes and stroke. To analyze the morphology of blood vessels, it is necessary to accurately segment blood vessels from fundus images, so as to completely reflect the changes of blood vessel width, blood vessel density and blood vessel branch pattern. Since manually labeling blood vessels is a difficult and time-consuming task, it is particularly important to automatically segment blood vessels from fundus images.

[0003] Existing algorithms for automatically segmenting blood vessels from fundus images divide the pixels in the fundus retinal image into fundus retinal blood vessels and background. The key to these methods is to obtain various features of the fundus retinal blood vessels, so as to effectively distinguish them from the complex surrounding tissues. Among them, the traditional segmentation method uses image intensity, blood vessel edge and morphological characteristics to design various artificially designed features for extracting fundus retinal blood vessels. However, these methods rely heavily on low-level semantic features designed by humans, and cannot be well applied to complex scenes such as high-resolution remote sensing images, thus having poor generalization performance and being easily affected by various noise interference. The fully convolutional neural network based on deep learning can automatically obtain high-level semantic features from remote sensing images through multi-layer network structure and nonlinear transformation. In addition, the end-to-end and pixel-to-pixel implementation of the fully convolutional neural network can provide pixel-level blood vessel recognition and positioning.

[0004] The fundus retinal blood vessel extraction method based on the fully convolutional network can effectively improve the extraction accuracy and generalization performance compared with the traditional method. However, due to the inherent locality of convolution operation, the fully convolution-based method shows obvious limitations in modeling the explicit long-distance pixel relationship. Transformer is initially proposed for natural language processing, which uses self-attention mechanism to obtain the long-distance dependence relationship of words. In 2020, it was applied to computer vision tasks and named visual transformer (ViT). In ViT, patches from different regions of the image are regarded as different words, and the dependence relationship between different positions in the image is obtained through self-attention mechanism. With the continuous application of visual Transformer in various vision tasks, a large number of fundus retinal blood vessel extraction methods based on visual Transformer have emerged recently.

[0005] However, in actual use of ViT, huge memory requirements and computing costs are often generated, which seriously affects its potential in downstream tasks. In addition, ViTs mainly focus on capturing global context, while ignoring the preservation of spatial detail context, but spatial details are also crucial for the extraction of small blood vessels. Although many algorithms combine ViT with fully convolutional networks to extract global and detail features, these methods directly combine the two, which easily causes the details to be overwhelmed. More importantly, the blood vessel extraction method based on ViT will lose a lot of spatial structure information due to the two-dimensional image conversion to one-dimensional sequence by the self-attention mechanism, causing the model to rely too much on the surface features of blood vessels, and thus being easily disturbed by similar surface tissues, uneven illumination and blood spot shielding. In addition, the boundary between blood vessels and surrounding tissues is not obvious, which seriously affects the accuracy of the extracted blood vessels; the network loses a lot of detail information. SUMMARY

[0006] The present application provides a retinal blood vessel image segmentation method based on edge and detail co-guided topological perception to solve at least one of the above technical problems in the prior art.

[0007] The present application adopts the following technical solution: a retinal blood vessel image segmentation method based on edge and detail co-guided topological perception, comprising the following steps:

[0008] Retinal blood vessel image data is acquired, and a data set is constructed based on the retinal blood vessel image data; the data of the data set is subjected to image processing to serve as input data of a retinal blood vessel image segmentation network initial model; a retinal blood vessel image segmentation network initial model is built, which comprises a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder comprises a detail branch, a context branch and an edge branch, which are used to extract the detail, semantic and edge features in the retinal blood vessel image; wherein the detail branch, the edge branch, the context branch and the decoder are each divided into n units, the units in the detail branch and the edge branch are connected with the units in the same layer of the context branch, the units in the context branch are connected with the units in the corresponding layer of the decoder through a coordinate attention module, and the output end of the context branch is connected with the input end of the decoder through the bottleneck module; the retinal blood vessel image segmentation network initial model is used to output a fundus retinal image semantic segmentation result; the retinal blood vessel image segmentation network initial model is trained and iterated based on the retinal blood vessel image data subjected to image processing, and is verified to obtain a retinal blood vessel image segmentation network model; retinal blood vessel image data to be segmented is collected and subjected to image processing, and is input into the retinal blood vessel image segmentation network model, and the retinal blood vessel image segmentation network model outputs a corresponding fundus retinal image semantic segmentation result.

[0009] Preferably, the up-down branch comprises a patch embedding layer and n down-sampling units; the down-sampling unit comprises a omnidirectional state space unit, a 2*2 max pooling layer and a boundary attention guidance fusion module connected in sequence, wherein the omnidirectional state space unit is used to extract global context feature maps at different scales, the 2*2 max pooling layer is used to down-sample the feature map size to half of the original size, and the boundary attention guidance fusion module is used to fuse the feature maps extracted by the down-sampling unit and the feature maps extracted by the unit layer of the corresponding detail branch in the edge region of the blood vessel, and to retain the feature maps obtained by the down-sampling unit in other regions; the output of the omnidirectional state space unit is connected with the detail branch and the edge branch, and the output end of the boundary attention guidance fusion module is connected with the decoder through a coordinate attention module; the input data of the model is input into the patch embedding layer, the output end of the patch embedding layer is connected with the first down-sampling unit, the n down-sampling units are connected in sequence from top to bottom, and the output end of the nth down-sampling unit is connected with the input end of the decoder through a bottleneck module.

[0010] Preferably, the detail branch comprises a segmentation head and n detail units; the detail unit comprises a 3*3 convolution layer and a pixel attention guidance fusion module, wherein the 3*3 convolution layer is used to extract detail features; the pixel attention guidance fusion module selectively aggregates the detail features extracted by the 3*3 convolution layer and the semantic features extracted by the unit layer of the corresponding context branch; the pixel attention guidance fusion module is respectively connected with the output end of the omnidirectional state space unit in the context branch and the input end of the boundary attention guidance fusion module; the n detail units are connected in sequence from top to bottom, the input data of the model is input into the first detail unit, and the segmentation head is connected with the output end of the nth detail unit; the segmentation head of the detail branch is used to generate a detail prediction map.

[0011] Preferably, the edge branch comprises an edge head and n edge units; the edge unit comprises a 3*3 convolution layer and an addition module, wherein the 3*3 convolution layer is used to extract edge features; the addition module is used to add the feature maps extracted by the 3*3 convolution layer and the result of up-sampling the output feature maps of the omnidirectional state space unit of the context branch; the input end of the addition module is connected with the 3*3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected with the input end of the boundary attention guidance fusion module; the n edge units are connected in sequence from top to bottom, the input data of the model is input into the first edge unit, and the edge head is connected with the output end of the nth edge unit; the edge head of the edge branch is used to generate an edge prediction map.

[0012] Preferably, the bottle neck module comprises an all-vector state space unit, and the decoder comprises a segmentation head and n up-sampling units; the up-sampling unit comprises a dynamic graph convolution module and a 3*3 transpose convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3*3 transpose convolution layer is used for up-sampling; the output end of the dynamic graph convolution module is connected with the input end of the 3*3 transpose convolution layer; the n up-sampling units are connected in turn from bottom to top, wherein the first down-sampling unit is connected with the bottle neck module, the output end of the n-th up-sampling unit is connected with the segmentation head of the decoder, and the segmentation head of the decoder is used to output a blood vessel prediction map; wherein the first up-sampling unit is connected with the n-th down-sampling unit, the second up-sampling unit is connected with the n-1-th down-sampling unit, and so on until the n-th up-sampling unit is connected with the first down-sampling unit.

[0013] Preferably, the method for data processing based on the coordinate attention module comprises: adopting two one-dimensional global pooling operations to perform global pooling along the width and height directions of the input feature respectively to obtain two direction perception feature maps with dimensions of H*1*C and 1*W*C respectively; performing spatial dimension splicing on the two direction perception feature maps, performing channel number compression by using a 1*1 convolution to obtain a feature map with a dimension of 1*(W+H)*C / r; after the compressed feature map is sequentially subjected to a batch normalization operation and a ReLU activation function, the feature map is re-divided into feature vectors in two directions; the dimensions of the two feature vectors are restored to H*1*C and C*1*W respectively by using a 1*1 convolution and a Sigmoid activation function; and the two dimension-restored feature vectors and the coordinate attention module input feature map are subjected to weighted calculation in two directions, and a feature is output.

[0014] Preferably, the calculation of the dynamic graph convolution module comprises dynamic graph generation and graph convolution calculation, wherein the dynamic graph generation comprises a graph node matrix and an adjacency matrix The graph convolution calculation is as follows:

[0015] Z = ReLU (AF' Θ)

[0016] wherein is the result of the dynamic graph convolution, is a parameter matrix;

[0017] The graph node matrix is obtained by inputting the dynamic graph convolution module input feature F H×W×C (H, W, C are the height, width and channel number of the feature map respectively) into a transformation matrix;

[0018] The calculation of the adjacency matrix firstly transforms the dynamic graph convolution module input feature F H×W×CThe input 1x1 convolution operation reduces the channel dimension to M, passes through a ReLU activation function, and then is converted into F through a transformation function HW×M and F M×HW , F HW×M The input features of the dynamic graph convolution module are subjected to global mean pooling, 1x1 convolution operation, Sigmoid activation function and diagonalization operation, and the result φ M×M and F M×HW are sequentially multiplied by matrix multiplication, and the obtained product is input into a Softmax activation function to obtain a normalized adjacency matrix.

[0019] Preferably, the pixel attention guidance fusion module specifically calculates the formula as follows:

[0020]

[0021] wherein and represent the vectors corresponding to any one pixel in the feature maps of the detail branch and the context branch respectively, σ represents the probability that the two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[ ] represents the Sigmoid function, f p () and f i () represent the 1x1 convolution operations on the detail branch and the context branch respectively, and Out Pag is the output of the pixel attention guidance fusion module.

[0022] Preferably, the specific calculation formula of the boundary attention guidance fusion module is as follows:

[0023]

[0024] wherein represents the vector corresponding to any one pixel in the feature map of the edge branch, f out [] represents a 3x3 convolution layer, and Out bag represents the output of the boundary attention guidance fusion module.

[0025] Preferably, the calculation formula of the loss function L total of the model iterative optimization is as follows:

[0026] L total = 0.4l0+20l1+l2+l3

[0027] wherein l0 is the loss function of the detail branch, which adopts a cross-entropy loss function, and l1 is the loss function of the edge branch, which adopts a weighted binary cross-entropy loss function, and the specific calculation is as follows:

[0028]

[0029] wherein s i,c and are the true value and the prediction result of the i-th pixel of the category c, and beta is a weight parameter;

[0030] l2 and l3 belong to the semantic branch loss function, and cross-entropy loss function and edge perception loss function are adopted respectively, and the specific calculation is as follows:

[0031]

[0032] wherein t is a predefined threshold, t=0.8, b i is the edge head output of the i-th pixel.

[0033] Compared with the prior art, the beneficial effects of the present application are:

[0034] 1) The present application adopts an additional edge branch and a detail branch to guide the learning of blood vessel features, so that the model learns more detailed and structural features, and the recognition ability of blood vessels and the edge positioning accuracy are enhanced;

[0035] 2) The pixel attention guidance fusion module in the present application selectively aggregates the semantic features in the detail feature and the context branch and avoids being overwhelmed by the semantic features in the context branch;

[0036] 3) The omnidirectional state space unit in the network of the present application can efficiently extract global blood vessel features with low computational complexity;

[0037] 4) The boundary attention guidance fusion module of the present application makes the model fuse the feature map of the detail branch only with the feature map obtained by the context branch in the blood vessel edge region, and retains the feature map obtained by the context branch in other regions;

[0038] 5) The coordinate attention module is adopted in the skip connection of the present application, which is conducive to selecting appropriate channel information for fusion and avoiding interference and loss of spatial structure information, so as to better improve the blood vessel extraction accuracy of the model;

[0039] 6) The dynamic graph convolution module is adopted in the decoder of the present application, which has strong topological form perception ability, so that the model can still effectively identify the blood vessel target when the blood vessel is occluded or other appearance clues are poor;

[0040] Through the above beneficial effects, the present application comprehensively solves the problems that the existing automatic blood vessel extraction method of fundus retinal image is easily affected by similar external tissues, blurred boundary between blood vessels and surrounding tissues, uneven illumination and blood spot occlusion interference, and serious loss of detail information. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed in the embodiments will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained from these drawings without creative labor.

[0042] Fig. 1 is a schematic diagram of the overall structure of the fundus retina image blood vessel extraction network constructed in the method of the present application;

[0043] Fig. 2 is a schematic diagram of the composition structure of the omnidirectional state space unit in the fundus retina image blood vessel extraction network constructed in the method of the present application;

[0044] Fig. 3 is a schematic diagram of the composition structure of the coordinate attention module in the fundus retina image blood vessel extraction network constructed in the method of the present application;

[0045] Fig. 4 is a schematic diagram of the composition structure of the dynamic graph convolution module in the fundus retina image blood vessel extraction network constructed in the method of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application are clearly and completely described in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0047] It should be noted that the structures, proportions, sizes, etc. shown in the drawings of the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the limiting conditions of the embodiments of the present application, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that can be achieved by the present application, should fall within the scope of the technical content disclosed by the present application. It should be noted that in the present specification, relationship terms such as first and second are only used to distinguish one entity from another entity, and do not necessarily require or imply any actual relationship or order between the entities.

[0048] The present application provides an embodiment:

[0049] As shown in Figs. 1 to 4 , a retinal blood vessel image segmentation method based on edge and detail co-conduction topology perception includes the following steps:

[0050] obtaining retinal blood vessel image data, constructing a data set based on the retinal blood vessel image data;

[0051] performing image processing on the data of the data set as input data of a retinal blood vessel image segmentation network initial model;

[0052] building the retinal blood vessel image segmentation network initial model, the retinal blood vessel image segmentation network initial model comprising a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder comprising a detail branch, a context branch and an edge branch, for extracting detail, semantic and edge features in the retinal blood vessel image; wherein the detail branch, the edge branch, the context branch and the decoder are each divided into n units, the units in the detail branch and the edge branch are each connected to the units of the same layer in the context branch, the units in the context branch are each connected to the units of the corresponding layer in the decoder through a coordinate attention module, and the output end of the context branch is connected to the input end of the decoder through the bottleneck module; the retinal blood vessel image segmentation network initial model is used to output a fundus retinal image semantic segmentation result;

[0053] training and iterating the retinal blood vessel image segmentation network initial model based on the image-processed retinal blood vessel image data, and verifying the retinal blood vessel image segmentation network initial model, to obtain a retinal blood vessel image segmentation network model;

[0054] collecting retinal blood vessel image data to be segmented and performing image processing thereon, inputting the image-processed data to the retinal blood vessel image segmentation network model, and outputting a corresponding fundus retinal image semantic segmentation result by the retinal blood vessel image segmentation network model.

[0055] In this embodiment, the method for image processing the retinal blood vessel image data comprises greyscale processing, normalization, contrast limited adaptive histogram equalization, gamma correction, data enhancement and real edge extraction of blood vessels.

[0056] The greyscale processing of the retinal blood vessel image converts a color image of the retinal image into a greyscale image, and the specific calculation formula is: X gray = 0.299R + 0.578G + 0.114B, wherein R, G and B represent the red, green and blue channels in the color image, and X gray represents the greyscale processing result.

[0057] Normalization: first, standardize the greyscale image, and then adjust the range of the image greyscale to 0-255, and the specific calculation formula is as follows:

[0058]

[0059] wherein μ and δ represent the average value and the standard deviation of the pixel greyscale in X gray , X min and Xmax Represents X norm The minimum and maximum values ​​of the grayscale of the medium pixel.

[0060] Contrast-limited adaptive histogram equalization, for a standardized image, sets a threshold to limit the histogram distribution to enhance image contrast in a restrictive manner, while ensuring that the histogram distribution can be transformed into an approximately uniform distribution; at the same time, bilinear interpolation is used in the pixel mapping process to speed up the histogram equalization process.

[0061] Gamma correction performs non-linear tonal editing on the image gamma curve, identifies the foreground and background parts in the image signal, and increases their ratio to improve the image contrast. Finally, to avoid negative, excessive, or NaN loss values ​​during network training, the pixel values ​​on the image are normalized and mapped to the range of 0-1.

[0062] Data augmentation includes training set data augmentation and test set data augmentation. The training set data augmentation randomly extracts 1000 sub-images of size 128×128 from each image, while the test set data augmentation extracts sub-images of size 128×128 from the top left corner of the input original image, following the principle of "from top to bottom and from left to right" with a step size of 15.

[0063] The extraction of true blood vessel edges first uses the Canny edge detection algorithm to extract blood vessel edges from the label image, and then uses dilation processing on the obtained blood vessel edge image to connect the breakpoints to obtain a smooth and continuous true blood vessel edge image.

[0064] In this embodiment, the upper and lower branches include a patch embedding layer and three downsampling units. Each downsampling unit includes an omnidirectional state space unit, a 2×2 max pooling layer, and a boundary attention-guided fusion module connected in sequence. The omnidirectional state space unit is used to extract global context feature maps at different scales. The 2×2 max pooling layer is used to downsample the feature map size to half of the original size. The boundary attention-guided module is used to fuse the feature map extracted by the downsampling unit and the feature map extracted by the corresponding detail branch unit layer in the blood vessel edge region, and retain the feature map obtained by the downsampling unit in other regions.

[0065] The output of the omnidirectional state space unit is connected to the detail branch and edge branch. The output of the boundary attention guidance module is connected to the decoder via the coordinate attention module. The input data of the model is input to the patch embedding layer. The output of the patch embedding layer is connected to the first downsampling unit. The three downsampling units are connected sequentially from top to bottom. The output of the third downsampling unit is connected to the input of the decoder via the bottleneck module.

[0066] The omnidirectional state space unit first uses a layer normalization operation to standardize the input features, then uses a linear transformation layer (denoted as the first linear transformation) to transform the channel dimension, and then uses a 3x3 depth convolution layer to extract spatial features with fewer parameters. The obtained features are sequentially input into an omnidirectional selective scanning module and a linear transformation layer (denoted as the second linear transformation), and the output is multiplied by the linear transformation of the corresponding pixels of the layer normalization operation output. The product is input into another linear transformation layer (denoted as the third linear transformation) and then added to the input of the layer normalization operation through a residual connection. The omnidirectional selective scanning module selectively scans the input features along the horizontal, vertical, diagonal, and anti-diagonal directions, and then adds all the scanning results.

[0067] The detail branch includes a segmentation head and three detail units; the detail unit includes a 3x3 convolution layer and a pixel attention guidance fusion module, wherein the 3x3 convolution layer is composed of a 3x3 convolution, a batch normalization operation and a ReLu activation layer in sequence, and is used to extract detail features; the pixel attention guidance fusion module selectively aggregates the detail features extracted by the 3x3 convolution layer and the semantic features extracted by the unit layer of the corresponding context branch; the pixel attention guidance fusion module is connected to the output end of the omnidirectional state space unit in the context branch and the input end of the boundary attention guidance fusion module, respectively;

[0068] The three detail units are connected in sequence from top to bottom, and the input data of the model is input into the first detail unit. The segmentation head is connected to the output end of the third detail unit. The segmentation head of the detail branch is used to generate a detail prediction map. The detail branch is beneficial to the accurate extraction of small size branch blood vessels.

[0069] The edge branch includes an edge head and three edge units; the edge unit includes a 3x3 convolution layer and an addition module, wherein the 3x3 convolution layer is used to extract edge features; the addition module is used to add the feature map extracted by the 3x3 convolution layer and the up-sampling result of the output feature map of the omnidirectional state space unit of the context branch; the input end of the addition module is connected to the 3x3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected to the input end of the boundary attention guidance fusion module;

[0070] The three edge units are connected in sequence from top to bottom, and the input data of the model is input into the first edge unit. The edge head is connected to the output end of the third edge unit. The edge head of the edge branch is used to generate an edge prediction map. The edge branch is beneficial to accurately extracting blood vessels from low-contrast fundus retinal images.

[0071] The bottleneck module comprises an all-vector state space unit, and the decoder comprises a segmentation head and n up-sampling units; the up-sampling unit comprises a dynamic graph convolution module and a 3*3 transpose convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3*3 transpose convolution layer is used for up-sampling; the output end of the dynamic graph convolution module is connected with the input end of the 3*3 transpose convolution layer;

[0072] The three up-sampling units are sequentially connected from bottom to top, wherein the first up-sampling unit is connected with the bottleneck module, the output end of the third up-sampling unit is connected with the segmentation head of the decoder, and the segmentation head of the decoder is used to output a blood vessel prediction map; wherein the first up-sampling unit is connected with the third down-sampling unit, the second up-sampling unit is connected with the second down-sampling unit, and the third up-sampling unit is connected with the first down-sampling unit in sequence.

[0073] In the embodiment, the specific calculation formula of the pixel attention guidance fusion module is as follows:

[0074]

[0075] wherein and represent the vectors corresponding to any one pixel in the feature maps of the detail branch and the context branch respectively, and σ represents the probability that the two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[ ] represents a Sigmoid function, and f p and f i represent 1*1 convolution operations on the detail branch and the context branch respectively, and Out Pag is the output of the pixel attention guidance fusion module.

[0076] The specific calculation formula of the boundary attention guidance fusion module is as follows:

[0077]

[0078] wherein represents the vector corresponding to any one pixel in the feature map of the edge branch, and f out represents a 3*3 convolution layer, Out bag represents the output of the boundary attention guidance fusion module, and σ' represents the output of a Sigmoid function.

[0079] The method for data processing based on the coordinate attention module is as follows:

[0080] Two one-dimensional global pooling operations are performed along the width and height directions of the input features, respectively, to obtain two directional perceptual feature maps with dimensions H×1×C and 1×W×C. The two directional perceptual feature maps are then concatenated in spatial dimension, and a 1×1 convolution is used to compress the number of channels to obtain a feature map with dimensions 1×(W+H)×C / r. The compressed feature map is then subjected to batch normalization and ReLU activation functions to re-divide the feature map into feature vectors in two directions. The dimensions of the two feature vectors are restored to H×1×C and C×1×W by 1×1 convolution and Sigmoid activation functions, respectively. The two-dimensional restored feature vectors are then weighted and calculated in both directions with the input feature map of the coordinate attention module, and the feature is output.

[0081] The computation of the dynamic graph convolution module includes the generation of the dynamic graph and the computation of graph convolution. The generation of the dynamic graph includes the graph node matrix. and adjacency matrix The graph convolution calculation is as follows:

[0082]

[0083] in The result of dynamic graph convolution. For parameter matrices;

[0084] The graph node matrix is ​​obtained by inputting the dynamic graph convolution module into the feature F. H×W×C (H, W, and C represent the height, width, and number of channels of the feature map, respectively) are obtained by inputting the transformation matrix;

[0085] The adjacency matrix is ​​first calculated by inputting the dynamic graph convolution module into the feature F. H×W×C The input is reduced to M channels by a 1×1 convolution operation, then activated by a ReLU function, and finally transformed into F by a transformation function. HW×M and F M×HW F HW×M The input features of the dynamic graph convolution module are processed by global mean pooling, 1×1 convolution, sigmoid activation function, and diagonalization, resulting in the following φ: M×M and F M×HW The matrix multiplications are performed sequentially, and the resulting products are input into the Softmax activation function to obtain the normalized adjacency matrix.

[0086] Loss function L during model iterative optimization total The calculation formula is as follows:

[0087] L total =0.4l0+20l1+l2+l3

[0088] wherein l0 is a detail branch loss function, adopting a cross-entropy loss function, and l1 is an edge branch loss function, adopting a weighted binary cross-entropy loss function, and the specific calculation is as follows:

[0089]

[0090] wherein s i,c and are the true value and the prediction result of the i-th pixel of the category c, and β is a weight parameter;

[0091] l2 and l3 belong to the semantic branch loss function, and adopt a cross-entropy loss function and an edge perception loss function respectively, and the specific calculation is as follows:

[0092]

[0093] wherein t is a predefined threshold, t=0.8, b i is the edge head output of the i-th pixel.

[0094] The application introduces the detail and edge branch to guide the blood vessel extraction together, pixel attention guidance fusion module, omnidirectional state space unit, boundary attention guidance fusion module, coordinate attention module and dynamic graph convolution module in the blood vessel extraction network, and comprehensively solves the problems that the existing blood vessel automatic extraction method of fundus retina image is easily interfered by similar external tissues, blurred boundary between blood vessels and surrounding tissues, uneven illumination and blood spot shielding, and serious loss of detail information.

[0095] The above is only the preferred specific embodiment of the application, but the protection scope of the application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A retinal blood vessel image segmentation method based on edge and detail co- topology perception, characterized in that, The method comprises the following steps: obtaining retinal blood vessel image data, and constructing a data set based on the retinal blood vessel image data; performing image processing on the data of the data set as input data of an initial model of a retinal blood vessel image segmentation network; building the initial model of the retinal blood vessel image segmentation network, which comprises a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder comprises a detail branch, a context branch and an edge branch, and is used to extract details, semantics and edge features in the retinal blood vessel image; wherein the detail branch, the edge branch, the context branch and the decoder are each divided into n units, the units in the detail branch and the edge branch are connected with the units in the same layer of the context branch, the units in the context branch are connected with the units in the corresponding layer of the decoder through a coordinate attention module, and the output end of the context branch is connected with the input end of the decoder through the bottleneck module; the initial model of the retinal blood vessel image segmentation network is used to output a semantic segmentation result of a fundus retinal image; wherein the upper and lower branches comprise a patch embedding layer and n down-sampling units; the down-sampling unit comprises a universal state space unit, a 2*2 max pooling layer and a boundary attention guidance fusion module connected in sequence, wherein the universal state space unit is used to extract global context feature maps at different scales, the 2*2 max pooling layer is used to down-sample the feature map size to half of the original size, and the boundary attention guidance fusion module is used to fuse the feature map extracted by the down-sampling unit and the feature map extracted by the unit layer of the corresponding detail branch in the blood vessel edge region, and retain the feature map obtained by the down-sampling unit in other regions; the output of the universal state space unit is connected with the detail branch and the edge branch, and the output end of the boundary attention guidance fusion module is connected with the decoder through a coordinate attention module; the input data of the model is input into the patch embedding layer, the output end of the patch embedding layer is connected with the first down-sampling unit, the n down-sampling units are connected in sequence from top to bottom, and the output end of the nth down-sampling unit is connected with the input end of the decoder through the bottleneck module; training and iterating the initial model of the retinal blood vessel image segmentation network based on the image-processed retinal blood vessel image data, verifying the initial model, and obtaining a retinal blood vessel image segmentation network model; collecting retinal blood vessel image data to be segmented and performing image processing, inputting the image data into the retinal blood vessel image segmentation network model, and outputting a corresponding semantic segmentation result of a fundus retinal image by the retinal blood vessel image segmentation network model.

2. The retinal blood vessel image segmentation method based on edge and detail co-guided topology perception according to claim 1, wherein: the detail branch comprises a segmentation head and n detail units; the detail unit comprises a 3*3 convolution layer and a pixel attention guidance fusion module, wherein the 3*3 convolution layer is used to extract detail features; the pixel attention guidance fusion module selectively aggregates the detail features extracted by the 3*3 convolution layer and the semantic features extracted by the unit layer of the corresponding context branch. The pixel attention guiding module is connected with the output end of the omnidirectional state space unit in the context branch and the input end of the boundary attention guiding fusion module, respectively. The n detail units are connected in sequence from top to bottom, the input data of the model is input to the first detail unit, and the segmentation head is connected with the output end of the nth detail unit.

3. The retinal blood vessel image segmentation method based on edge and detail co-orientation topology perception according to claim 2, characterized in that: The edge branch includes an edge head and n edge units. The edge unit includes a 3*3 convolution layer and an addition module, wherein the 3*3 convolution layer is used to extract edge features. The addition module is used to add the feature map extracted by the 3*3 convolution layer and the result after upsampling of the output feature map of the omnidirectional state space unit of the context branch; the input end of the addition module is connected with the 3*3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected with the input end of the boundary attention guiding fusion module. The n edge units are connected in sequence from top to bottom, the input data of the model is input to the first edge unit, and the edge head is connected with the output end of the nth edge unit.

4. The retinal blood vessel image segmentation method based on edge and detail co- topology-awareness according to claim 3, characterized in that: The bottleneck module includes an omnidirectional state space unit, and the decoder includes a segmentation head and n upsampling units. The upsampling unit includes a dynamic graph convolution module and a 3*3 transpose convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3*3 transpose convolution layer is used for upsampling; the output end of the dynamic graph convolution module is connected with the input end of the 3*3 transpose convolution layer. The n upsampling units are connected in sequence from bottom to top, wherein the first upsampling unit is connected with the bottleneck module, and the output end of the nth upsampling unit is connected with the segmentation head of the decoder; the segmentation head of the decoder is used to output a blood vessel prediction map. The first upsampling unit is connected with the nth downsampling unit, the second upsampling unit is connected with the (n-1)th downsampling unit, and so on until the nth upsampling unit is connected with the first downsampling unit.

5. The edge and detail co-cod topology-aware based retinal blood vessel image segmentation method according to claim 1, wherein: The method for data processing based on the coordinate attention module is as follows: Two one-dimensional global pooling operations are used to perform global pooling along the width and height directions of the input feature respectively, to obtain two direction perception feature maps with dimensions of H*1*C and 1*W*C respectively; The two direction perception feature maps are spliced in the spatial dimension, and a 1*1 convolution is used for channel compression to obtain a feature map with a dimension of 1*(W+H)*C / r; After the compressed feature map is sequentially subjected to a batch normalization operation and a ReLU activation function, the feature map is divided into two direction feature vectors again; The dimensions of the two feature vectors are restored to H*1*C and C*1*W respectively through a 1*1 convolution and a Sigmoid activation function; The two dimension-restored feature vectors are weighted calculated with the coordinate attention module input feature map in two directions, and the feature is output.

6. The edge and detail co-cod topology-aware based retinal blood vessel image segmentation method according to claim 4, characterized in that: The calculation of the dynamic graph convolution module includes generation of a dynamic graph and graph convolution calculation, wherein the generation of the dynamic graph includes a graph node matrix and an adjacency matrix The graph convolution calculation is as follows: Z = ReLU(AF' Θ) wherein is the result of a dynamic graph convolution, is a parameter matrix; The graph node matrix is obtained by inputting the dynamic graph convolution module into the feature F H×W×C (H, W, C are height, width and channel number of the feature map respectively) input transformation matrix The calculation of the adjacency matrix is firstly to input the dynamic graph convolution module feature F H×W×C The input 1×1 convolution operation reduces the channel dimension to M, passes through a ReLU activation function, and then passes through two transformation functions to convert F HW×M and F M×HW , respectively. HW×M The dynamic graph convolution module input feature F M×M and F M×HW are sequentially multiplied according to matrix multiplication, and the obtained product is input into the Softmax activation function to obtain the normalized adjacency matrix.

7. The edge and detail co-cod topology aware retinal blood vessel image segmentation method according to claim 2, characterized in that: The specific calculation formula of the pixel attention guiding fusion module is as follows: wherein and respectively represent the vector corresponding to any one pixel in the detail branch and the context branch feature map, σ represents the probability that the two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[] represents the Sigmoid function, f p () and f i () respectively represent the 1×1 convolution operation on the detail branch and the context branch, Out Pag is the output of the pixel attention guided fusion module.

8. The edge and detail co-cod topology-aware based retinal blood vessel image segmentation method according to claim 7, characterized in that: The specific calculation formula of the boundary attention guiding fusion module is as follows: wherein a vector corresponding to any one pixel in the feature map representing the edge branch, f out [] represents a 3x3 convolution layer, Out bag represents the output of the boundary attention guidance fusion module.

9. The edge and detail co-cod topology aware retinal blood vessel image segmentation method according to claim 7, characterized in that: The loss function L when the model is iteratively optimized total The calculation formula is as follows: L total ＝0.4l0+20l1+l2+l3 where lo is the detail branch loss function, which adopts the cross-entropy loss function, and li is the edge branch loss function, which adopts the weighted binary cross-entropy loss function, and the specific calculation is as follows: where s i,c and are the true value and the prediction result of the i-th pixel of class c, and β is a weight parameter; l2 and l3 belong to the semantic branch loss function, which respectively adopts the cross-entropy loss function and the edge perception loss function, and the specific calculation is as follows: where t is a predefined threshold, t = 0.8, b i is the edge head output of the i-th pixel.

Images