Retinal blood vessel image segmentation method based on edge and detail common guide topology perception

Abstract

The invention belongs to the technical field of medical image semantic segmentation methods, and aims to solve the problems of low blood vessel extraction precision and serious detail information loss of an existing algorithm. The invention provides a retinal blood vessel image segmentation method based on edge and detail common guide topology perception. The method comprises the following steps: constructing a retinal blood vessel image segmentation network initial model; performing training iteration and verification on the initial model of the retinal blood vessel image segmentation network to obtain a retinal blood vessel image segmentation network model; and retinal blood vessel image data needing to be segmented are collected and subjected to image processing, the retinal blood vessel image data are input into the retinal blood vessel image segmentation network model, and the retinal blood vessel image segmentation network model outputs a corresponding fundus retina image semantic segmentation result. The method solves the problems that blood vessel extraction of the fundus retina image is easily interfered by surface similar tissues, fuzzy boundaries of blood vessels and surrounding tissues, non-uniform illumination and blood spot shielding, and detail information is seriously lost.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image semantic segmentation methods, and in particular relates to a retinal vascular image segmentation method based on edge and detail co-conducting topological perception. Background Art

[0002] Analysis of vascular structure from fundus images has been widely used in the diagnosis of various ophthalmic diseases caused by arteriosclerosis, diabetes and stroke. To analyze the morphology of blood vessels, it is necessary to accurately segment the blood vessels from fundus images so as to fully reflect the changes in vessel width, vessel density and vessel branching pattern. Since manual annotation of blood vessels is a difficult and time-consuming task, automatic segmentation of blood vessels from fundus images becomes particularly important.

[0003] Existing algorithms for automatically segmenting blood vessels from fundus images divide the pixels in fundus retinal images into fundus retinal vessels and background. The key to these methods is to obtain various features of fundus retinal vessels, so as to effectively distinguish them from the complex surrounding tissues. Among them, traditional segmentation methods use image intensity, vascular edges and morphological characteristics to design various artificially designed features for extracting fundus retinal vessels. However, these methods rely heavily on artificially designed low-level semantic features and cannot be well applied to complex scenes such as high-resolution remote sensing images. Therefore, they have poor generalization performance and are easily affected by various noise interferences. The fully convolutional neural network based on deep learning can automatically obtain high-level semantic features from remote sensing images through a multi-layer network structure and nonlinear transformation. In addition, the end-to-end, pixel-to-pixel implementation of the fully convolutional neural network can provide pixel-level blood vessel recognition and positioning.

[0004] The fundus retinal blood vessel extraction method based on a fully convolutional network can effectively improve the extraction accuracy and generalization performance compared to traditional methods. However, due to the inherent locality of the convolution operation, the fully convolution-based method shows obvious limitations in modeling clear long-distance pixel relationships. Transformer was originally proposed for natural language processing. It uses a self-attention mechanism to obtain long-distance dependencies between words. It was applied to computer vision tasks in 2020 and named Visual Transformer (ViT). In ViT, patches from different areas of the image are regarded as different words, and the dependencies between different positions in the image are obtained through a self-attention mechanism. With the continuous application of visual transformers in various visual tasks, a large number of fundus retinal blood vessel extraction methods based on visual transformers have emerged recently.

[0005] However, when ViT is used in practice, it often generates huge memory requirements and computational costs, which seriously affects its potential in downstream tasks. In addition, ViTs mainly focus on capturing global context and ignore the preservation of spatial detail context, but spatial details are also crucial for the extraction of small blood vessels. Although many algorithms combine ViT with fully convolutional networks to extract global and detail features at the same time, these methods directly combine the two, which easily causes the detail information to be submerged. More importantly, the blood vessel extraction method based on ViT loses a lot of spatial structural information due to the self-attention mechanism when converting two-dimensional images into one-dimensional sequences, causing the model to rely too much on the appearance of blood vessels, and is therefore easily affected by similar-looking tissues, uneven lighting, and blood spot occlusion. In addition, the boundaries between blood vessels and surrounding tissues are not obvious, which seriously affects the accuracy of the extracted blood vessels; the detail information in the network is seriously lost. Summary of the invention

[0006] In order to solve at least one of the above-mentioned technical problems existing in the prior art, the present invention provides a retinal vascular image segmentation method based on edge and detail co-conducting topological perception.

[0007] The present invention is implemented by the following technical solution: a retinal vascular image segmentation method based on edge and detail co-conduction topological perception, comprising the following steps:

[0008] Acquire retinal vascular image data, and construct a data set based on the retinal vascular image data; perform image processing on the data of the data set as input data of the initial model of the retinal vascular image segmentation network; build the initial model of the retinal vascular image segmentation network, the initial model of the retinal vascular image segmentation network includes a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder includes a detail branch, a context branch and an edge branch, which are used to extract details, semantics and edge features in the retinal vascular image; wherein the detail branch, the edge branch, the context branch and the decoder are divided into n units, the units in the detail branch and the edge branch are connected to the units in the same layer of the context branch, the units in the context branch are connected to the units in the corresponding layer of the decoder through a coordinate attention module, and the output end of the context branch is connected to the input end of the decoder through a bottleneck module; the initial model of the retinal vascular image segmentation network is used to output the semantic segmentation result of the fundus retinal image; based on the image-processed retinal vascular image data, the initial model of the retinal vascular image segmentation network is trained, iterated and verified to obtain the retinal vascular image segmentation network model; collect the retinal vascular image data to be segmented and perform image processing, input it into the retinal vascular image segmentation network model, and the retinal vascular image segmentation network model outputs the corresponding semantic segmentation result of the fundus retinal image.

[0009] Preferably, the upper and lower branches include a patch embedding layer and n downsampling units; the downsampling unit includes an omnidirectional state space unit, a 2×2 maximum pooling layer and a boundary attention guidance fusion module connected in sequence, wherein the omnidirectional state space unit is used to extract global context feature maps at different scales, the 2×2 maximum pooling layer is used to downsample the feature map size to half of the original size, and the boundary attention guidance module is used to fuse the feature map extracted by the downsampling unit and the feature map extracted by the unit layer of the corresponding detail branch in the blood vessel edge area, and retain the feature map obtained by the downsampling unit in other areas; the output of the omnidirectional state space unit is connected to the detail branch and the edge branch, and the output end of the boundary attention guidance module is connected to the decoder via the coordinate attention module; the input data of the model is input into the patch embedding layer, the output end of the patch embedding layer is connected to the first downsampling unit, the n downsampling units are connected in sequence from top to bottom, and the output end of the nth downsampling unit is connected to the input end of the decoder through the bottleneck module.

[0010] Preferably, the detail branch includes a segmentation head and n detail units; the detail unit includes a 3×3 convolutional layer and a pixel attention guidance fusion module, wherein the 3×3 convolutional layer is used to extract detail features; the pixel attention guidance module selectively aggregates the detail features extracted by the 3×3 convolutional layer and the semantic features extracted by the unit layer of the corresponding context branch; the pixel attention guidance module is respectively connected to the output end of the omnidirectional state space unit in the context branch and the input end of the boundary attention guidance fusion module; the n detail units are connected in sequence from top to bottom, the input data of the model is input to the first detail unit, the segmentation head is connected to the output end of the nth detail unit, and the segmentation head of the detail branch is used to generate a detail prediction map.

[0011] Preferably, the edge branch includes an edge head and n edge units; the edge unit includes a 3×3 convolution layer and an addition module, wherein the 3×3 convolution layer is used to extract edge features; the addition module is used to add the feature map extracted by the 3×3 convolution layer and the upsampling result of the output feature map of the omnidirectional state space unit of the context branch; the input end of the addition module is connected to the 3×3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected to the input end of the boundary attention guidance fusion module; the n edge units are connected in sequence from top to bottom, the input data of the model is input to the first edge unit, the edge head is connected to the output end of the nth edge unit, and the edge head of the edge branch is used to generate an edge prediction map.

[0012] Preferably, the bottleneck module includes a full vector state space unit, and the decoder includes a segmentation head and n upsampling units; the upsampling unit includes a dynamic graph convolution module and a 3×3 transposed convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3×3 transposed convolution layer is used for upsampling; the output end of the dynamic graph convolution module is connected to the input end of the 3×3 transposed convolution layer; the n upsampling units are connected in sequence from bottom to top, wherein the first downsampling unit is connected to the bottleneck module, and the output end of the nth upsampling unit is connected to the segmentation head of the decoder, and the segmentation head of the decoder is used to output a blood vessel prediction map; wherein the first upsampling unit is connected to the nth downsampling unit, the second upsampling unit is connected to the n-1th downsampling unit, and so on, until the nth upsampling unit is connected to the first downsampling unit.

[0013] Preferably, the method for data processing based on the coordinate attention module is: using two one-dimensional global pooling operations to perform global pooling along the width and height directions of the input features respectively, to obtain two directional perception feature maps with dimensions of H×1×C and 1×W×C respectively; splicing the two directional perception feature maps in the spatial dimension, and using 1×1 convolution to compress the number of channels to obtain a feature map with a dimension of 1×(W+H)×C / r; after the compressed feature map passes through batch normalization operation and ReLU activation function in turn, the feature map is redivided into feature vectors in two directions; the dimensions of the two feature vectors are restored to H×1×C and C×1×W respectively through 1×1 convolution and Sigmoid activation function; the feature vectors after the two dimensions are restored and the coordinate attention module input feature map perform weighted calculation in two directions, and output features.

[0014] Preferably, the calculation of the dynamic graph convolution module includes the generation of a dynamic graph and graph convolution calculation, wherein the generation of the dynamic graph includes the graph node matrix and the adjacency matrix The graph convolution is calculated as follows:

[0015] Z = ReLU(AF'Θ)

[0016] in is the result of dynamic graph convolution, is the parameter matrix;

[0017] The graph node matrix is ​​fed into the dynamic graph convolution module as input feature F H×W×C (H, W, C are the height, width and number of channels of the feature map respectively) input transformation matrix to get;

[0018] The adjacency matrix is ​​first calculated by inputting the dynamic graph convolution module into the feature F H×W×CThe input 1×1 convolution operation reduces the channel dimension to M, passes a ReLU activation function, and then transforms it to F through the transformation function HW×M and F M×HW , F HW×M The result of the dynamic graph convolution module input feature after global mean pooling, 1×1 convolution operation, Sigmoid activation function and diagonalization operation φ M×M and F M×HW Multiply them one by one according to matrix multiplication, and input the product into the Softmax activation function to obtain the normalized adjacency matrix.

[0019] Preferably, the specific calculation formula of the pixel attention guidance fusion module is as follows:

[0020]

[0021] in and Represent the vector corresponding to any pixel in the detail branch and context branch feature map, σ represents the probability that two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[ ] represents the Sigmoid function, f p ( ) and f i ( ) represent the 1×1 convolution operations on the detail branch and the context branch, respectively. Pag is the output of the pixel attention guided fusion module.

[0022] Preferably, the specific calculation formula of the boundary attention guidance fusion module is as follows:

[0023]

[0024] in Represents the vector corresponding to any pixel in the feature map of the edge branch, f out [ ] represents a 3×3 convolutional layer, Out bag Represents the output of the boundary attention guided fusion module.

[0025] Preferably, the loss function L during iterative optimization of the model is total The calculation formula is as follows:

[0026] L total =0.4l0+20l1+l2+l3

[0027] Among them, l0 is the detail branch loss function, which uses the cross entropy loss function, and l1 is the edge branch loss function, which uses the weighted binary cross entropy loss function. The specific calculation is as follows:

[0028]

[0029] where s i,c and is the true value and prediction result of the i-th pixel of category c, and β is the weight parameter;

[0030] l2 and l3 belong to the semantic branch loss function, which adopts the cross entropy loss function and edge perception loss function respectively. The specific calculation is as follows:

[0031]

[0032] Where t is a predefined threshold, t = 0.8, b i is the edge header output for the ith pixel.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1) The present invention uses additional edge branches and detail branches to guide the learning of vascular features, so that the model learns more details and structural features, and enhances the recognition ability of blood vessels and the edge positioning accuracy;

[0035] 2) The pixel attention guided fusion module in the present invention selectively aggregates the detail features and the semantic features in the context branch and avoids being overwhelmed by the semantic features in the context branch;

[0036] 3) The omnidirectional state space unit in the network of the present invention can extract global vascular features efficiently with low computational complexity;

[0037] 4) The boundary attention guided fusion module of the present invention enables the model to fuse the feature map of the detail branch with the feature map obtained by the context branch only in the blood vessel edge area, while retaining the feature map obtained by the context branch in other areas;

[0038] 5) The coordinate attention module is used in the skip connection of the present invention. This design is conducive to selecting appropriate channel information for fusion and avoiding interference and loss of spatial structure information, so as to better improve the blood vessel extraction accuracy of the model;

[0039] 6) The present invention adopts a dynamic graph convolution module in the decoder, which can effectively identify the vascular target even when the model blood vessels are blocked or other appearance clues are poor due to its powerful topological morphological perception ability;

[0040] Through the above beneficial effects, the present invention comprehensively solves the problems that the existing automatic extraction method of blood vessels in fundus retinal images is easily affected by similar-looking tissues, blurred boundaries between blood vessels and surrounding tissues, uneven illumination and blood spot occlusion, and serious loss of detail information. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a schematic diagram of the overall structure of the fundus retinal image blood vessel extraction network constructed in the method of the present invention;

[0043] Figure 2 It is a schematic diagram of the composition structure of the omnidirectional state space unit in the fundus retinal image blood vessel extraction network constructed in the method of the present invention;

[0044] Figure 3 It is a schematic diagram of the composition structure of the coordinate attention module in the fundus retinal image blood vessel extraction network constructed in the method of the present invention;

[0045] Figure 4 It is a schematic diagram of the composition structure of the dynamic graph convolution module in the fundus retinal image blood vessel extraction network constructed in the method of the present invention. DETAILED DESCRIPTION

[0046] In conjunction with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other implementation methods obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0047] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention. It should be noted that in this specification, relational terms such as first and second are only used to distinguish one entity from several other entities, and do not necessarily require or imply any actual relationship or order between these entities.

[0048] The present invention provides an embodiment:

[0049] like Figures 1 to 4 As shown, a retinal vascular image segmentation method based on edge and detail co-conduction topology perception includes the following steps:

[0050] Acquire retinal vascular image data, and construct a data set based on the retinal vascular image data;

[0051] Perform image processing on the data set and use it as input data for the initial model of the retinal vascular image segmentation network;

[0052] Build an initial model of the retinal vascular image segmentation network, which includes a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder includes a detail branch, a context branch and an edge branch, which are used to extract detail, semantic and edge features in the retinal vascular image; wherein the detail branch, the edge branch, the context branch and the decoder are each divided into n units, the units in the detail branch and the edge branch are connected to the units in the same layer of the context branch, the units in the context branch are connected to the units in the corresponding layer of the decoder through the coordinate attention module, and the output of the context branch is connected to the input of the decoder through the bottleneck module; the initial model of the retinal vascular image segmentation network is used to output the semantic segmentation results of the fundus retinal image;

[0053] Based on the image-processed retinal vascular image data, the initial model of the retinal vascular image segmentation network is trained, iterated, and verified to obtain a retinal vascular image segmentation network model;

[0054] The retinal vascular image data to be segmented is collected and processed, and input into the retinal vascular image segmentation network model, which outputs the corresponding fundus retinal image semantic segmentation result.

[0055] In this embodiment, the method for performing image processing on retinal vascular image data includes grayscale conversion, normalization, contrast-limited adaptive histogram equalization, gamma correction, data enhancement, and real vascular edge extraction.

[0056] Among them, the grayscale retinal vascular image converts the retinal image color image into a grayscale image. The specific calculation formula is: gray =0.299R+0.578G+0.114B, where R, G, B represent the red, green, and blue channels in a color image, and X gray Represents the grayscale result.

[0057] Normalization: First, the grayscale image is standardized, and then the grayscale range of the image is adjusted to 0-255. The specific calculation formula is as follows:

[0058]

[0059] Where μ and δ represent X gray The mean and standard deviation of pixel grayscale in X min and Xmax Represents X norm The minimum and maximum grayscale values ​​of pixels.

[0060] Contrast-limited adaptive histogram equalization, for the standardized image, under the premise of ensuring that the histogram distribution can be transformed into an approximately uniform distribution, sets a threshold to limit the histogram distribution to restrictively enhance the image contrast; at the same time, bilinear interpolation is used in the pixel mapping process to speed up the histogram equalization.

[0061] Gamma correction performs nonlinear tone editing on the image gamma curve, identifies the foreground and background parts of the image signal, and increases the ratio between the two, thereby improving the contrast of the image. Finally, in order to avoid negative, excessive, and nan loss values ​​during network training, the pixel values ​​on the image are normalized and mapped to between 0 and 1.

[0062] Data augmentation includes training set data augmentation and test set data augmentation. For the training set data augmentation, 1000 sub-images of size 128×128 are randomly obtained from each image, while for the test set data augmentation, sub-images of size 128×128 are extracted from the upper left corner of the input original image in accordance with the principle of “from top to bottom and from left to right” with a step size of 15.

[0063] The real edge extraction of blood vessels first uses the Canny edge detection algorithm to extract the blood vessel edges from the label image, and then uses the expansion process to connect the breakpoints of the obtained blood vessel edge map to obtain a smooth and continuous real edge map of the blood vessel.

[0064] In this embodiment, the upper and lower branches include a patch embedding layer and three downsampling units; the downsampling unit includes an omnidirectional state space unit, a 2×2 maximum pooling layer and a boundary attention guidance fusion module connected in sequence, wherein the omnidirectional state space unit is used to extract global context feature maps at different scales, the 2×2 maximum pooling layer is used to downsample the feature map size to half of the original size, and the boundary attention guidance module is used to fuse the feature map extracted by the downsampling unit and the feature map extracted by the unit layer of the corresponding detail branch in the blood vessel edge area, and retain the feature map obtained by the downsampling unit in other areas;

[0065] The output of the omnidirectional state space unit is connected to the detail branch and the edge branch, and the output of the boundary attention guidance module is connected to the decoder via the coordinate attention module. The input data of the model is input into the patch embedding layer, and the output of the patch embedding layer is connected to the first downsampling unit. The three downsampling units are connected in sequence from top to bottom, and the output of the third downsampling unit is connected to the input of the decoder through the bottleneck module.

[0066] The omnidirectional state space unit first uses a layer normalization operation to normalize the input features to enhance the stability of model training, then uses a linear transformation layer (denoted as the first linear transformation) to transform the channel dimension and then uses a 3×3 deep convolution layer to extract spatial features with fewer parameters. The obtained features are then sequentially input into an omnidirectional selective scanning module and a linear transformation layer (denoted as the second linear transformation). The output is multiplied with the corresponding pixel of the linear transformation output of the layer normalization operation. The product passes through another linear transformation layer (denoted as the third linear transformation) and is added to the input of the layer normalization operation through a residual connection. The omnidirectional selective scanning module selectively scans its input features in the forward and reverse directions along the horizontal, vertical, diagonal and anti-diagonal directions, and then adds all the scanning results.

[0067] The detail branch includes a segmentation head and three detail units; the detail unit includes a 3×3 convolutional layer and a pixel attention guidance fusion module, wherein the 3×3 convolutional layer is composed of a 3×3 convolution, a batch normalization operation and a ReLu activation layer in sequence, and is used to extract detail features; the pixel attention guidance module selectively aggregates the detail features extracted by the 3×3 convolutional layer and the semantic features extracted by the unit layer of the corresponding context branch; the pixel attention guidance module is respectively connected to the output end of the omnidirectional state space unit in the context branch and the input end of the boundary attention guidance fusion module;

[0068] The three detail units are connected from top to bottom in sequence. The input data of the model is input to the first detail unit. The segmentation head is connected to the output end of the third detail unit. The segmentation head of the detail branch is used to generate a detail prediction map. The detail branch is conducive to the accurate extraction of small-sized branch vessels.

[0069] The edge branch includes an edge head and three edge units; the edge unit includes a 3×3 convolution layer and an addition module, wherein the 3×3 convolution layer is used to extract edge features; the addition module is used to add the feature map extracted by the 3×3 convolution layer and the up-sampled result of the output feature map of the omnidirectional state space unit of the context branch; the input end of the addition module is connected to the 3×3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected to the input end of the boundary attention guidance fusion module;

[0070] The three edge units are connected from top to bottom in sequence. The input data of the model is input to the first edge unit. The edge head is connected to the output end of the third edge unit. The edge head of the edge branch is used to generate an edge prediction map. The edge branch is conducive to accurately extracting blood vessels from low-contrast fundus retinal images.

[0071] The bottleneck module includes a full vector state space unit, and the decoder includes a segmentation head and n upsampling units; the upsampling unit includes a dynamic graph convolution module and a 3×3 transposed convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3×3 transposed convolution layer is used for upsampling; the output of the dynamic graph convolution module is connected to the input of the 3×3 transposed convolution layer;

[0072] The three upsampling units are connected sequentially from bottom to top, wherein the first downsampling unit is connected to the bottleneck module, and the output end of the third upsampling unit is connected to the segmentation head of the decoder, and the segmentation head of the decoder is used to output the blood vessel prediction map; wherein the first upsampling unit is connected to the third downsampling unit, the second upsampling unit is connected to the second downsampling unit, and so on, until the third upsampling unit is connected to the first downsampling unit.

[0073] In this embodiment, the specific calculation formula of the pixel attention guidance fusion module is as follows:

[0074]

[0075] in and Represent the vector corresponding to any pixel in the detail branch and context branch feature map, σ represents the probability that two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[ ] represents the Sigmoid function, f p ( ) and f i ( ) represent the 1×1 convolution operations on the detail branch and the context branch, respectively. Pag is the output of the pixel attention guided fusion module.

[0076] The specific calculation formula of the boundary attention guidance fusion module is as follows:

[0077]

[0078] in Represents the vector corresponding to any pixel in the feature map of the edge branch, f out [ ] represents a 3×3 convolutional layer, Out bag represents the output of the boundary attention guided fusion module, and σ' represents the output of the Sigmoid function.

[0079] The method for data processing based on the coordinate attention module is:

[0080] Two one-dimensional global pooling operations are used to perform global pooling along the width and height directions of the input features respectively, and two directional perception feature maps with dimensions of H×1×C and 1×W×C are obtained; the two directional perception feature maps are spliced ​​in the spatial dimension, and the number of channels is compressed using 1×1 convolution to obtain a feature map with a dimension of 1×(W+H)×C / r; the compressed feature map is subjected to batch normalization operation and ReLU activation function in turn, and then the feature map is redivided into feature vectors in two directions; the dimensions of the two feature vectors are restored to H×1×C and C×1×W by 1×1 convolution and Sigmoid activation function respectively; the feature vectors after the two dimensions are restored and the coordinate attention module are input into the feature map for weighted calculation in two directions, and the features are output.

[0081] The calculation of the dynamic graph convolution module includes the generation of dynamic graphs and graph convolution calculations, where the generation of dynamic graphs includes the graph node matrix and the adjacency matrix The graph convolution is calculated as follows:

[0082]

[0083] in is the result of dynamic graph convolution, is the parameter matrix;

[0084] The graph node matrix is ​​fed into the dynamic graph convolution module as input to the feature F H×W×C (H, W, C are the height, width and number of channels of the feature map respectively) input transformation matrix to get;

[0085] The adjacency matrix is ​​first calculated by feeding the dynamic graph convolution module into the feature F H×W×C The input 1×1 convolution operation reduces the channel dimension to M, passes a ReLU activation function, and then is converted to F through the deformation function. HW×M and F M×HW , F HW×M The result of the dynamic graph convolution module input feature after global mean pooling, 1×1 convolution operation, Sigmoid activation function and diagonalization operation φ M×M and F M×HW Multiply them one by one according to matrix multiplication, and input the product into the Softmax activation function to obtain the normalized adjacency matrix.

[0086] The loss function L during model iterative optimization total The calculation formula is as follows:

[0087] L total =0.4l0+20l1+l2+l3

[0088] Among them, l0 is the detail branch loss function, which uses the cross entropy loss function, and l1 is the edge branch loss function, which uses the weighted binary cross entropy loss function. The specific calculation is as follows:

[0089]

[0090] where s i,c and is the true value and prediction result of the i-th pixel of category c, and β is the weight parameter;

[0091] l2 and l3 belong to the semantic branch loss function, which adopts the cross entropy loss function and edge perception loss function respectively. The specific calculation is as follows:

[0092]

[0093] Where t is a predefined threshold, t = 0.8, b i is the edge header output for the ith pixel.

[0094] The present invention introduces detail and edge branches to jointly guide blood vessel extraction, a pixel attention guidance fusion module, an omnidirectional state space unit, a boundary attention guidance fusion module, a coordinate attention module and a dynamic graph convolution module into the blood vessel extraction network, which comprehensively solves the problems that the existing automatic blood vessel extraction methods of fundus retinal images are easily affected by similar-looking tissues, blurred boundaries between blood vessels and surrounding tissues, uneven illumination and blood spot occlusion, and serious loss of detail information.

[0095] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A retinal vascular image segmentation method based on edge and detail co-conduction topological perception, characterized in that: The following steps are involved: Acquire retinal vascular image data, and construct a data set based on the retinal vascular image data; Perform image processing on the data set and use it as input data for the initial model of the retinal vascular image segmentation network; An initial model of a retinal vascular image segmentation network is constructed, wherein the initial model comprises a three-branch encoder, a bottleneck module and a decoder; the three-branch encoder comprises a detail branch, a context branch and an edge branch, and is used to extract detail, semantic and edge features in retinal vascular images; wherein the detail branch, the edge branch, the context branch and the decoder are each divided into n units, and the units in the detail branch and the edge branch are each connected to the units in the same layer in the context branch, and the units in the context branch are each connected to the units in the corresponding layer in the decoder through a coordinate attention module, and the output end of the context branch is connected to the input end of the decoder through a bottleneck module; the initial model of the retinal vascular image segmentation network is used to output semantic segmentation results of fundus retinal images; wherein the upper and lower branches comprise a patch embedding layer and n downsampling units; the downsampling unit comprises a sequentially connected The omnidirectional state space unit, 2×2 maximum pooling layer and boundary attention guidance fusion module are used, wherein the omnidirectional state space unit is used to extract global context feature maps at different scales, the 2×2 maximum pooling layer is used to downsample the feature map size to half of the original size, and the boundary attention guidance module is used to fuse the feature map extracted by the downsampling unit and the feature map extracted by the unit layer of the corresponding detail branch in the edge area of ​​the blood vessel, and retain the feature map obtained by the downsampling unit in other areas; the output of the omnidirectional state space unit is connected to the detail branch and the edge branch, and the output end of the boundary attention guidance module is connected to the decoder through the coordinate attention module; the input data of the model is input into the patch embedding layer, the output end of the patch embedding layer is connected to the first downsampling unit, the n downsampling units are connected in sequence from top to bottom, and the output end of the nth downsampling unit is connected to the input end of the decoder through the bottleneck module; Based on the image-processed retinal vascular image data, the initial model of the retinal vascular image segmentation network is trained, iterated, and verified to obtain a retinal vascular image segmentation network model; The retinal vascular image data to be segmented is collected and processed, and input into the retinal vascular image segmentation network model, which outputs the corresponding fundus retinal image semantic segmentation result.

2. The retinal vascular image segmentation method based on edge and detail co-conduction topological perception according to claim 1 is characterized in that: The detail branch includes a segmentation head and n detail units; The detail unit includes a 3×3 convolution layer and a pixel attention guided fusion module, wherein the 3×3 convolution layer is used to extract detail features; The pixel attention guidance module selectively aggregates the detail features extracted by the 3×3 convolutional layer and the semantic features extracted by the unit layer of the corresponding context branch; The pixel attention guidance module is connected to the output of the omnidirectional state space unit in the context branch and the input of the boundary attention guidance fusion module respectively; The n detail units are connected from top to bottom in sequence, the input data of the model is input to the first detail unit, the segmentation head is connected to the output end of the nth detail unit, and the segmentation head of the detail branch is used to generate a detail prediction map.

3. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 2 is characterized in that: The edge branch includes an edge head and n edge units; The edge unit includes a 3×3 convolution layer and an addition module, wherein the 3×3 convolution layer is used to extract edge features; The addition module is used to add the feature map extracted by the 3×3 convolution layer and the up-sampled result of the output feature map of the omnidirectional state space unit of the context branch; the input end of the addition module is connected to the 3×3 convolution layer and the omnidirectional state space unit, and the output end of the addition module is connected to the input end of the boundary attention guidance fusion module; The n edge units are connected in sequence from top to bottom. The input data of the model is input to the first edge unit. The edge head is connected to the output end of the nth edge unit. The edge head of the edge branch is used to generate an edge prediction graph.

4. The retinal vascular image segmentation method based on edge and detail co-conduction topological perception according to claim 3 is characterized in that: The bottleneck module includes a full vector state space unit, and the decoder includes a segmentation head and n upsampling units; The upsampling unit includes a dynamic graph convolution module and a 3×3 transposed convolution layer, wherein the dynamic graph convolution module is used to obtain global topological features at different scales, and the 3×3 transposed convolution layer is used for upsampling; the output end of the dynamic graph convolution module is connected to the input end of the 3×3 transposed convolution layer; The n upsampling units are connected sequentially from bottom to top, wherein the first downsampling unit is connected to the bottleneck module, and the output end of the nth upsampling unit is connected to the segmentation head of the decoder, and the segmentation head of the decoder is used to output a blood vessel prediction map; The first upsampling unit is connected to the nth downsampling unit, the second upsampling unit is connected to the n-1th downsampling unit, and so on, until the nth upsampling unit is connected to the first downsampling unit.

5. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 1, characterized in that: The method for data processing based on the coordinate attention module is: Two one-dimensional global pooling operations are used to perform global pooling along the width and height directions of the input features, respectively, to obtain two directional perception feature maps with dimensions of H×1×C and 1×W×C; The two directional perception feature maps are spliced ​​in the spatial dimension, and the number of channels is compressed using 1×1 convolution to obtain a feature map with a dimension of 1×(W+H)×C / r; After the compressed feature map passes through the batch normalization operation and the ReLU activation function in turn, the feature map is re-divided into feature vectors in two directions; The dimensions of the two feature vectors are restored to H×1×C and C×1×W through 1×1 convolution and Sigmoid activation function respectively; The feature vectors restored in two dimensions and the coordinate attention module are input into the feature map for weighted calculation in two directions, and the features are output.

6. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 4, characterized in that: The calculation of the dynamic graph convolution module includes the generation of dynamic graphs and graph convolution calculations, wherein the generation of dynamic graphs includes the graph node matrix and the adjacency matrix The graph convolution is calculated as follows: Z = ReLU(AF'Θ) in is the result of dynamic graph convolution, is the parameter matrix; The graph node matrix is ​​fed into the dynamic graph convolution module as input feature F H×W×C (H, W, C are the height, width and number of channels of the feature map respectively) input transformation matrix to get; The adjacency matrix is ​​first calculated by inputting the dynamic graph convolution module into the feature F H×W×C The input 1×1 convolution operation reduces the channel dimension to M, passes a ReLU activation function, and then is converted to F through two transformation functions. HW×M and F M×HW , F HW×M The result of the dynamic graph convolution module input feature after global mean pooling, 1×1 convolution operation, Sigmoid activation function and diagonalization operation φ M×M and F M×HW Multiply them one by one according to matrix multiplication, and input the product into the Softmax activation function to obtain the normalized adjacency matrix.

7. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 2, characterized in that: The specific calculation formula of the pixel attention guidance fusion module is as follows: in and Represents the vector corresponding to any pixel in the detail branch and context branch feature maps, σ represents the probability that two pixels belong to blood vessels or surrounding tissues at the same time, Sigmoid[] represents the Sigmoid function, and f p () and f i () represent the 1×1 convolution operations on the detail branch and the context branch, respectively. Pag is the output of the pixel attention guided fusion module.

8. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 7, characterized in that: The specific calculation formula of the boundary attention guidance fusion module is as follows: in Represents the vector corresponding to any pixel in the feature map of the edge branch, f out [] represents a 3×3 convolutional layer, Out bag Represents the output of the boundary attention guided fusion module.

9. The retinal vascular image segmentation method based on edge and detail co-conduction topology perception according to claim 7, characterized in that: The loss function L during iterative optimization of the model total The calculation formula is as follows: L total ＝0.4l0+20l1+l2+l3 Among them, l0 is the detail branch loss function, which uses the cross entropy loss function, and l1 is the edge branch loss function, which uses the weighted binary cross entropy loss function. The specific calculation is as follows: where s i,c and is the true value and prediction result of the i-th pixel of category c, and β is the weight parameter; l2 and l3 belong to the semantic branch loss function, which adopts the cross entropy loss function and edge perception loss function respectively. The specific calculation is as follows: Where t is a predefined threshold, t = 0.8, b i is the edge header output for the ith pixel.

Images