A method for curve structure segmentation based on skeleton and boundary enhancement

Abstract

In view of the problems of low boundary accuracy, poor topological connectivity, and difficulty in segmenting narrow curve branches in curve structure segmentation, the present invention proposes a method for curve structure segmentation based on boundary and skeleton enhancement. The present invention constructs a Skeleton and Edge Boosted Network (SEBN), which is based on an encoder-decoder architecture. The encoder is used to generate features at different scales, and three decoder heads are designed in the decoder: a curve segmentation head, an edge detection head, and a skeleton extraction head. Among them, the skeleton extraction head alleviates the problem of unbalanced branch scales in curve structure segmentation and improves the connectivity of curve segmentation; the edge detection head focuses on edge features to accurately locate the edges of curve objects. Importantly, the present invention integrates these two decoder heads and the curve segmentation head into a Three-Head-fusion-Module (THFM) to achieve three-feature fusion. The model decoder is composed of four cascaded THFMs, and three-feature exchange is performed at different scales of the decoder to achieve high-quality curve structure segmentation. In addition, in the test stage of SEBN, a Sigmentation-Supplementary-Module (SSM) is designed at the end of SEBN, and skeleton prediction is used to supplement segmentation prediction. This method can be used for curve structure segmentation such as medical blood vessel segmentation and road crack detection, and high-quality curve structure segmentation results can be obtained.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and image processing, and relates to a method for curve structure segmentation based on skeleton and boundary enhancement. This method well solves the problems of low boundary accuracy, poor connectivity, and difficult segmentation of narrow curve branches in curve structure segmentation. Background Art

[0002] Curve structure segmentation has extensive applications in medical image vascular segmentation, road extraction, and building crack detection, and has become a research hotspot in the field of computer vision in recent years. Since curve objects such as medical image blood vessels, roads, and cracks are usually irregularly distributed and narrow branches occupy too few pixels, it is very difficult to generate high-quality segmentation results from a single picture input. Early curve structure segmentation methods were usually manually designed based on some heuristic assumptions and can be divided into local and global segmentation methods. Among them, local segmentation methods use the information obtained from small segments or pixel inputs to detect curve objects, but local segmentation methods often lead to problems such as curve breakage and poor connectivity; global segmentation methods achieve curve segmentation by optimizing an energy minimization function based on the connectivity between adjacent fragments, but global segmentation methods have problems such as high computational complexity and sensitivity to noise and interference. In short, all these heuristic methods are manually designed based on the priors observed from a large number of given images and often perform poorly in some complex situations.

[0003] In recent years, deep learning-based methods have achieved rapid development in curve structure segmentation. However, some deep learning networks for curve structure segmentation do not effectively supervise the boundary and connectivity, and often have problems such as low topological connectivity and inaccurate edge detection. In order to improve the topological connectivity and edge accuracy of curve structure segmentation, many deep learning networks have begun to use the skeletons and edges of curve objects to assist curve segmentation. In the feature fusion of curve segmentation and skeleton extraction, skeleton extraction enhances the connectivity of curve segmentation, but due to the model focusing on skeleton extraction, the accuracy of the edge is often low; in the feature fusion of curve segmentation and boundary detection, boundary detection improves the edge accuracy of curve segmentation, but the connectivity of the segmentation is not effectively supervised, resulting in poor connectivity of the segmentation. Therefore, in order to ensure both the accuracy and connectivity of the segmentation, it is necessary to explore the simultaneous integration of edges and skeletons into curve structure segmentation.

[0004] There is currently a deep learning network that simultaneously performs three-task learning of curve structure segmentation, boundary detection, and skeleton extraction [IEEE Transactions on Geoscience and Remote Sensing, VOL.60: 5621414, 2022]. This network uses three different decoder heads for three-task prediction. The main decoder head generates segmentation predictions, and the other two auxiliary decoder heads generate skeleton predictions and boundary predictions. However, after the feature stream generated by the segmentation decoder head reaches the other two decoder heads, the features of the other two decoder heads do not flow back to the segmentation decoder head, and there is no sufficient three-task feature fusion, which limits the performance of segmentation. On the other hand, this network only exchanges shallow features at the end of the decoder and does not fully utilize other scale features extracted in the intermediate layer, which limits the learning ability of the model.

[0005] Therefore, to solve the problems of poor topological connectivity and poor edge accuracy in the current curve structure segmentation method, the present invention proposes a multi-scale feature fusion method that combines curve structure segmentation, skeleton extraction, and boundary detection. This method uses three decoder heads in the decoder to perform segmentation, boundary, and skeleton predictions. Among them, the features of the segmentation decoder head flow to the other two decoder heads, and the features of the other two decoder heads are learned and then flow back to the segmentation decoder head. The three features are fused in the segmentation decoder head to improve the segmentation effect. And this feature exchange is applied to the features of the deep layer, intermediate layer, and shallow layer of the decoder. By comprehensively using semantic information at different levels, the accuracy and connectivity of the segmentation result are improved. This feature exchange strategy helps to enhance the segmentation ability of the model, and this method has achieved high-quality segmentation effects in fields such as medical vascular segmentation and road crack detection. Summary of the Invention

[0006] Aiming at the problems of poor topological connectivity and poor edge accuracy in the existing curve structure segmentation method, the present invention proposes a curve structure segmentation method based on skeleton and edge enhancement. The present invention constructs a Skeleton and Edge Boosted Network (SEBN). This network is based on the encoder-decoder architecture of a convolutional neural network and uses the encoder to capture high-level features and semantic information in the image. Curve structure segmentation, boundary detection, and skeleton extraction are performed at multiple scales of the decoder to improve the boundary accuracy and connectivity of the segmentation. By training, the optimal weight file is obtained. The test image is input into the trained model, and skeleton prediction is used at the output end of the SEBN to supplement the segmentation prediction, further supplementing the segmentation result of the curve object. This method can cope with the challenges of complex images and provide more accurate and more connected curve structure segmentation results.

[0007] The technical solution adopted by the present invention includes the following steps:

[0008] 1. Select five publicly available curve structure datasets, and use an algorithm to generate boundary and skeleton annotations for the existing segmentation annotations;

[0009] 2. After preprocessing and image enhancement of the input image, input it into the SEBN network;

[0010] 3. Build the SEBN network structure. In SEBN, use the encoder to extract semantic features, and in the decoder, use four cascaded Three-Head-Fusion Module (THFM) to perform curve segmentation, skeleton extraction, and boundary detection simultaneously. Insert a Gated Attentive Unit (GAU) between the encoder and the decoder to achieve skip connections. SEBN uses an iterative feedback strategy to iteratively feedback the output prediction map into the GAU to supplement and enhance the segmentation features;

[0011] 4. Define loss functions for curve structure segmentation, boundary detection, and skeleton extraction in SEBN respectively;

[0012] 5. Training and optimization of the network: Use the curve structure dataset with three types of annotations to train the model, optimize the network parameters through the backpropagation algorithm, and save the most suitable weight model by training the network model;

[0013] 6. Testing of the network: Input the test set into the trained model. Design a Sigmentation-Supplementary-Module (SSM) at the output end of SEBN, and use the skeleton prediction to supplement the segmentation result.

[0014] The present invention proposes a method for curve structure segmentation based on skeleton and boundary enhancement, and its advantages are as follows:

[0015] 1. This method can improve the accuracy of curve structure segmentation and topological connectivity: In the feature fusion of segmentation and boundary, the boundary information provides supervision for the segmentation boundary, effectively improving the segmentation accuracy of curve structures. Skeleton extraction pays more attention to topological connectivity. In the feature fusion of segmentation and skeleton, the skeleton information provides topological connectivity information for segmentation, and also supplements the features of narrow curve branches for segmentation features, improving the connectivity of narrow branches.

[0016] 2. Multi-scale feature fusion: The method of the present invention performs three-feature fusion at multiple scales in the decoder. By fully utilizing the deep features at multiple scales, the model can more comprehensively capture the details and context information of curve structures, thereby further improving the accuracy and connectivity of the segmentation results.

[0017] 3. Design an SSM in the test stage, use the skeleton output with higher connectivity to supplement the segmentation result, supplement the narrow curve branches for the segmentation result, and improve the connectivity of the segmentation result. Description of the Drawings

[0018] Figure 1 : Overall flowchart of the method for curve structure segmentation based on skeleton and boundary enhancement;

[0019] Figure 2 : Flowchart of the curve structure segmentation network (SEBN) based on skeleton and boundary enhancement;

[0020] Figure 3 : Skip connection module in SEBN: Schematic diagram of the process of GAU;

[0021] Figure 4 : Components of the decoder in SEBN: Schematic diagram of the process of THFM;

[0022] Figure 5 : Schematic diagram of the process of SSM in the model testing stage. Detailed Implementation Manner

[0023] The technical solution of the present invention will be further specifically described below with reference to the drawings.

[0024] The overall process of the present invention is as Figure 1 shown. This method first needs to use an algorithm to generate the boundary and skeleton annotations of the dataset, then preprocess the input image, input the preprocessed image into the SEBN network for training, and save the optimal weight file after training. In the model testing stage, SEBN obtains the skeleton prediction and segmentation prediction. At the end of SEBN, SSM is used to supplement the segmentation prediction with the skeleton prediction to obtain the final segmentation result map.

[0025] The publicly available curve structure datasets used in the experiments include 4 medical image vascular datasets (DRIVE, STARE, OCT-B3, PRIME-FP20) and 1 road crack dataset (CRACK500). For the annotation of the datasets, the Canny operator and the skeletonization algorithm in the Python library are used to generate the corresponding boundary annotations and single-pixel skeleton annotations. The single-pixel skeleton annotations are dilated to skeletons with a pixel width of 3 to obtain the final skeleton annotations.

[0026] Figure 2Flowchart of the Skeleton and Edge Enhancement based Curve Structure Segmentation Network (SEBN) proposed by the present invention. SEBN uses a deep learning encoder-decoder architecture and mainly consists of three parts: the encoder on the left, the decoder on the right, and the Gated Attentive Unit (GAU) between the encoder and the decoder. Specifically, the encoder uses the first five layers of VGG16 as the backbone, which are sequentially represented by Conv_l (l = 1, 2, 3, 4, 5). GAU realizes the skip connection between the encoder and the decoder. As Figure 3 shown, it combines the iterative feedback input and the input of the encoder of the corresponding layer and uses spatial attention (SA) and channel attention (CA) to supplement and enhance the segmentation features in the decoder (t is the number of model iterations, l is the layer number of the encoder or decoder, is the output of the previous iteration). The decoder consists of four cascaded THFMs. In each THFM, from left to right, there are a skeleton extraction head Kl, a curve segmentation head Sl, and an edge detection head El (l = 1, 2, 3, 4). In addition, an SSM without learnable parameters is designed at the end of SEBN to supplement the segmentation prediction with skeleton prediction during the model testing phase.

[0027] Figure 4 Flowchart of THFM. Specifically: The three decoding heads Kl, Sl, and El in the l-th layer of THFM respectively upsample the variables and to generate feature maps of the same size. The curve segmentation head Sl performs a concatenate operation on the upsampling result and the refined feature output by GAU. After learning through a convolutional module, the learned features flow into the skeleton extraction head Kl and the edge detection head El respectively. In Kl and El, the incoming features are respectively concatenated and learned through two consecutive convolutional modules to obtain the skeleton feature and the edge feature of this layer of THFM. Subsequently and flow back to the curve segmentation head to be concatenated with the segmented features, and after learning through another convolutional module, they are added to the previous features in the curve segmentation head to obtain the optimized segmentation feature of this layer of THFM. Then the skeleton feature segmentation feature and the edge feature of this layer are input into the next layer of THFM.

[0028] The present invention designs SEBN to generate skeleton prediction Segmentation prediction and boundary prediction (when t = 3, l = 4). Annotations are used during training and to regularize these three predictions, and three loss functions are defined:

[0029]

[0030]

[0031]

[0032] Among them, represent the loss functions of skeleton extraction, curve structure segmentation, and boundary detection respectively. The model accumulates the losses of curve segmentation, boundary detection, and skeleton extraction in three iterations and finally performs backpropagation.

[0033] Specific process of SEBN network training:

[0034] In each round of training, for the input images, horizontal flipping, random rotations of 90°, 180°, and 270°, and random cropping are used to enhance the training pictures. Four medical vascular datasets are randomly cropped to a size of 256×256 for training, and the crack dataset is randomly cropped to 360×360 for training. The enhanced images are input into the SEBN network, and the segmentation prediction is obtained after the fourth layer THFM of the decoder. The segmentation prediction is iteratively fed back into the GAU of each layer to iteratively refine the features, and the final segmentation prediction is obtained after three iterations (t = 3). The model is implemented in PyTorch and optimized by Adam, with an initial learning rate of 10 -3 , and the weight decay is 5×10 -4 . It is trained for 500 rounds on the OCT - B3 dataset and 2000 rounds on other datasets, with the batch_size set to 2. After training, the network model is saved.

[0035] Testing of the model: The test set is input into the trained SEBN model. At the last iteration of SEBN, an SSM is designed at the end of SEBN, as Figure 5 shown, where the skeletonization algorithm in the Python library is used for skeletonization. In the SSM, the predicted result of the skeleton is skeletonized into a single - pixel skeleton and then directly added to the segmentation result to finally obtain the segmentation output. The method of the present invention has a relatively high connectivity of the prediction results on five datasets, and the segmentation results are more complete at the narrow curve branches.

[0036] Parts not detailed in the present invention belong to the common general knowledge of those skilled in the art.

Claims

1. A method for curve structure segmentation based on skeleton and boundary enhancement, the method comprising the following steps: Step 1: Select a curve structure dataset, and apply the Canny operator and the skeletonization algorithm to the segmentation annotations in the dataset to generate corresponding boundary annotations and skeleton annotations; Step 2: Build a Skeleton and Edge Boosted Network (SEBN), which is based on the encoder-decoder architecture of a convolutional neural network. The decoder consists of four cascaded Three-Head-Fusion Modules (THFMs), and three-feature fusion is performed at multiple scales of the decoder. SEBN uses a Gated Attentive Unit (GAU) to connect the encoder and the decoder, and uses a three-iteration feedback mechanism to refine the segmentation features; Step 3: Define loss functions for curve structure segmentation, boundary detection, and skeleton extraction in the loss function of the network; Step 4: Use a curve structure image dataset with three types of annotations to train the SEBN network to obtain an optimal weight model; Step 5: Input the test set into the trained SEBN model. At the last iteration of SEBN, design a Sigmentation-Supplementary-Module (SSM) at the end of SEBN, and use skeleton prediction to supplement segmentation prediction.

2. The method for curve structure segmentation based on skeleton and boundary enhancement according to claim 1, characterized in that In Step 1, the selected public curve structure dataset includes a medical vascular dataset and a road crack dataset. Apply the Canny operator and the Skeletonize algorithm in the Python library to the segmentation annotations of the public dataset to generate corresponding boundary annotations and single-pixel skeleton annotations. Dilate the single-pixel skeleton annotations to skeletons with a pixel width of 3 to obtain the final skeleton annotations.

3. The method for curve structure segmentation based on skeleton and boundary enhancement according to claim 1, wherein: In Step 2, SEBN uses an encoder-decoder architecture. The encoder is constructed from the first five layers of VGG16; four cascaded THFMs are used in the decoder to perform three-feature fusion at different scales of the decoder, and upsampling is used to gradually restore the details and spatial information of the image; specifically, in each THFM, the curve segmentation head, the boundary detection head, and the skeleton extraction head each upsample the features of the previous layer THFM to the same size. The features of curve segmentation are concatenated with the features of the corresponding layer GAU, and then passed through two convolutional modules to obtain segmentation features. These segmentation features flow to the boundary detection head and the skeleton extraction head on both sides; the boundary detection head and the skeleton extraction head combine the incoming segmentation features and each pass through two convolutional modules for learning to obtain the boundary features and skeleton features of this layer THFM. These two features then flow back into the curve segmentation head; the two features that flow back are concatenated with the features of curve segmentation, and then passed through a convolutional module and added to the previous segmentation features in the curve segmentation head to obtain the curve segmentation features of this layer THFM; the curve segmentation features, boundary features, and skeleton features of this layer are input into the next layer THFM.

4. The method for curve structure segmentation based on skeleton and boundary enhancement according to claim 1, characterized in that: In step 3, SEBN defines three loss functions for curve structure segmentation, boundary detection, and skeleton extraction. The traditional binary cross-entropy loss is used for the loss functions. These three losses are calculated in three iterations. Finally, all the loss functions are added together and then backpropagation is performed.

5. The method for curve structure segmentation based on skeleton and boundary enhancement according to claim 1, characterized in that: In step 5, the test set is input into the trained SEBN model. SEBN obtains the prediction maps for curve structure segmentation, boundary detection, and skeleton extraction. In the last iteration of the model, an SSM is designed at the end of SEBN. The more complete skeleton prediction map is skeletonized into a single-pixel skeleton, which is then added to the segmentation prediction map to finally obtain a complete and highly connected segmentation output.

Images