Curve segmentation network, segmentation method and image segmentation method using the network
By introducing a selective kernel convolution module and a Multi-Bi-ConvLSTM module into the medical image segmentation network, the problems of detail loss and noise in existing methods when segmenting curved structures are solved, and higher segmentation accuracy and detail capture are achieved.
Patent Information
- Application Number
- CN202110843182.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-26
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-07-26
AI Technical Summary
Existing medical image segmentation methods are prone to losing details or generating noise when segmenting curved structures, leading to false positive segmentation results. Furthermore, existing CNN-based methods are difficult to effectively capture high-level semantic and spatial information.
Selective kernel convolution (SK) modules are used to extract multi-scale features, and Multi-Bi-ConvLSTM is used as a non-linear skip connection to fuse the information learned by the network in the same stage, so that the feature information is gradually propagated from the deep layer to the shallow layer. Selective kernel convolution modules and joint convolutional recurrent neural networks are used to improve the feature extraction capability.
It improves the accuracy of medical image segmentation, especially when segmenting blood vessels and nerve fibers, and can better capture detailed information and high-level semantic information, reducing the false positive segmentation rate.
Smart Images

Figure CN113674284B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and more specifically, to a curve segmentation network for medical image segmentation, a segmentation method and an image segmentation method using the network. Background Art
[0002] The main purpose of curved structure segmentation is to automatically segment blood vessels or nerve fibers from medical images. In the past few decades, many methods for automatic curved structure segmentation have been proposed. Early segmentation methods usually construct manual features based on the shape, appearance and grayscale value of the image to complete the segmentation task. However, it is difficult for manually extracted features to effectively capture high-level semantic information to accurately segment complex and subtle curved structures, and the manually extracted features often contain image noise. In recent years, convolutional neural networks (CNNs) have become the mainstream method for many image segmentation tasks and have achieved good performance. CNN-based segmentation methods can learn more and more high-level semantic features by using many convolutional layers, thereby obtaining highly accurate segmentation results. However, most existing CNN-based segmentation methods can directly predict the segmentation results through the high-level feature maps output by the last layer of CNN, which can easily lose spatial information and details, resulting in rough segmentation boundaries.
[0003] In response to this limitation, the fully convolutional network (FCN) was proposed in 2015 and became a commonly used neural network structure in the field of medical image segmentation. FCN replaces the last few fully connected layers of the convolutional network with convolutional layers, so more spatial details can be retained for segmentation. FCN is an encoder-decoder architecture, but the decoding process in FCN is relatively simple, with only one deconvolution operation.
[0004] In order to expand the architecture of FCN to obtain more accurate segmentation results, the U-Net segmentation network was proposed. U-Net is a symmetrical encoding-decoding structure consisting of a contraction path and an expansion path, and these two paths fuse the features learned in the same stage through jump connections. Due to the good performance of the U-Net segmentation network, it has become a benchmark for medical image segmentation methods. At present, many improved segmentation methods are designed based on the structure of the U-Net segmentation network. For example, Li and Chen et al. introduced ResNets into the U-Net network to obtain more contextual information, and Jin and Oktay et al. embedded the attention mechanism into the U-Net network to reduce unimportant information, thereby improving the efficiency of network information processing. These existing methods have achieved good results on various medical image segmentation tasks. However, they still have some shortcomings, such as the easy loss of details of the curve structure or the generation of noise in the segmentation results, which will lead to many false positive segmentation results. Existing image segmentation technology still has shortcomings. Summary of the invention
[0005] The purpose of the present invention is to provide an image segmentation network based on nonlinear skip connection and a segmentation method thereof in view of the technical problems existing in the prior art. A new curve segmentation network is named Curv-Net. In the network, a Selective Kernel (selective kernel convolution) module is used to realize dynamic selection and fusion of features based on convolution kernels of different sizes, and a Multi-Bi-ConvLSTM (joint convolutional recurrent neural network) is constructed as a nonlinear skip connection, instead of directly using a linear skip connection such as addition to improve the feature extraction capability of the network.
[0006] Specifically, in order to solve the above-mentioned technical problems, the technical solution adopted by the present invention is:
[0007] A curve segmentation network includes multiple stages from shallow to deep layers for learning feature mapping, each stage is provided with an encoding path and a corresponding decoding path; selective kernel convolution modules are embedded in the encoding path and the decoding path; a nonlinear jump connection is connected between the encoding path and the decoding path.
[0008] Furthermore, the nonlinear jump connection is composed of a joint convolutional recurrent neural network, a bidirectional convolutional recurrent neural network or a convolutional recurrent neural network.
[0009] Furthermore, the joint convolutional recurrent neural network includes: a bidirectional convolutional recurrent neural network corresponding to the number of the input feature maps, and multiple bidirectional convolutional recurrent neural networks are connected in series; the bidirectional convolutional recurrent neural network accepts the feature maps learned by the selective kernel convolution module in the encoding path in the same stage, and the fused features output by the bidirectional convolutional recurrent neural network in the previous stage; the first-level bidirectional convolutional recurrent neural network receives the fused features output by the joint convolutional recurrent neural network in the adjacent deeper stage; the last-level bidirectional convolutional recurrent neural network outputs the fused features to the decoding path in the same stage, and the joint convolutional recurrent neural network in the adjacent shallower stage.
[0010] Furthermore, the bidirectional convolutional recurrent neural network comprises a forward path and a backward path respectively constituted by two convolutional recurrent neural networks, which respectively perform forward processing and backward processing on input data and output bidirectional spatial information.
[0011] Furthermore, the selective kernel convolution module filters the input feature map through group convolution with three different kernel sizes, and generates different feature maps through batch normalization and ReLU activation function; the kernel sizes are: 3×3, 5×5 and 7×7 respectively.
[0012] Furthermore, in each stage, the encoding path includes two selective kernel convolution modules with adaptive receptive field sizes and a pooling layer; the decoding path includes two selective kernel convolution modules and a deconvolution layer.
[0013] In order to solve the above problems, the present invention also provides:
[0014] A curve segmentation method applied to the above curve segmentation network comprises the following steps:
[0015] S1. In each stage of learning feature maps, the encoding path uses two selective kernel convolution modules with adaptive receptive field sizes to extract rich multi-scale features of the input image;
[0016] S2. The encoding path uses a pooling layer to perform a maximum pooling operation as a downsampling operator to reduce the scale of the feature map;
[0017] S3. Based on the nonlinear skip connection method of the joint convolutional recurrent neural network, the feature maps learned by the encoding path in the same stage of the network are fused, and the feature information is gradually propagated from the deep stage of the network to the shallow stage, and finally the fused features are output to the decoding path;
[0018] S4. The decoding path uses two selective kernel convolution modules to process the input fusion features to obtain better position information;
[0019] S5. The decoding path uses a deconvolution layer to perform a deconvolution operation to restore the image.
[0020] In order to solve the above problems, the present invention also provides:
[0021] A curve segmentation method for an image uses the above-mentioned curve segmentation network to perform curve segmentation on an input image.
[0022] Compared with the prior art, the present invention has the following beneficial effects:
[0023] The present invention first uses the SK (selective kernel convolution) module to adaptively extract the multi-scale features of the input image, and then designs a Multi-Bi-ConvLSTM (joint convolutional recurrent neural network) as the encoding path and decoding path of the same stage of the nonlinear jump connection segmentation network, so as to fuse the information learned by the network at the same stage. At the same time, through the jump connection, the feature information is gradually transferred from the deep layer of the network to the shallow layer, so that the features captured by Curv-Net (curve segmentation network) can contain more detailed information and high-level semantic information at the same time, thereby improving the segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 It is a structural diagram of the curve segmentation network of the present invention.
[0025] Figure 2 This is a structural diagram of the joint convolutional recurrent neural network of the present invention.
[0026] Figure 3 This is a structural diagram of the selective kernel convolution module of the present invention.
[0027] Figure 4 This is a structural diagram of the convolutional recurrent neural network of the present invention.
[0028] Figure 5 The figure is an example of the blood vessel segmentation result of the image segmentation method of the present invention on the DRIVE dataset.
[0029] Figure 6 This is an example of the blood vessel segmentation results of the image segmentation method of the present invention on the CHASE_DB1 dataset.
[0030] Figure 7 This is an example of the corneal nerve fiber segmentation results of the image segmentation method of the present invention on the CCM-2 dataset.
[0031] Figure 8 Flow chart of the curve segmentation method of the present invention. DETAILED DESCRIPTION
[0032] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0034] Embodiment 1
[0035] See also Figure 1 As shown, the present invention provides a curve segmentation network, including multiple stages from shallow to deep for learning feature mapping, each stage is provided with an encoding path and a corresponding decoding path; selective kernel convolution modules are embedded in both the encoding path and the decoding path; a nonlinear jump connection is connected between the encoding path and the decoding path.
[0036] Specifically, some existing segmentation methods based on U-Net networks usually use a fixed receptive field in each layer of the network, that is, the size of the convolution filter in each layer is the same. However, the fixed-size receptive field cannot capture multi-scale spatial information at the same time, which will reduce the performance of the deep network. In order to solve this problem, the present invention proposes to use an SK (selective kernel convolution) module with an adaptive receptive field in each encoding and decoding stage of the network to extract rich multi-scale features from the input image.
[0037] As attached Figure 3 As shown in the figure, the detailed structure of the SK (selective kernel convolution) module used in the Curv-Net (curve segmentation network) proposed in the present invention. First, the input feature map is mapped using grouped convolutions with different kernel sizes (3×3, 5×5 and 7×7). (C is the number of feature channels, W and H are the width and height of feature X respectively) are filtered, and then further processed by batch normalization and ReLU activation function to generate different feature maps X 3 , X 5 and X 7 Next, in order to adaptively fuse the feature map X 3 , X 5 and X 7 , the weights m of feature fusion need to be learned separately in different branches of the SK module k , weight m k The calculation formula is as follows:
[0038]
[0039] Among them, W k Represents a parameter, f is processed by global average pooling operation and fully connected layer A compact feature is obtained. is the fusion of multiple feature maps, that is
[0040]
[0041] By learning the weight vector m k , the final feature map F output by the SK module is an adaptive fusion of multi-scale features, which can be described by the following formula:
[0042]
[0043] Further, as attached Figure 1 , 2As shown, the nonlinear jump connection is composed of a joint convolutional recurrent neural network, a bidirectional convolutional recurrent neural network or a convolutional recurrent neural network. Among them, the Multi-Bi-ConvLSTM (joint convolutional recurrent neural network) module can be replaced by Bi-ConvLSTM (bidirectional convolutional recurrent neural network) or ConvLSTM (convolutional recurrent neural network), but the effect of improving the accuracy of segmentation is not as good as the Multi-Bi-ConvLSTM module.
[0044] The present application preferably uses Multi-Bi-ConvLSTM (joint convolutional cyclic neural network) to construct nonlinear jump connections between the encoding path and the decoding path, effectively integrating all the information learned at the same stage in the encoding path of the segmentation network, and gradually propagating high-level semantic features from the deep stage of the network to the shallow stage.
[0045] Furthermore, the joint convolutional recurrent neural network includes: a bidirectional convolutional recurrent neural network corresponding to the number of input feature maps, and multiple bidirectional convolutional recurrent neural networks are connected in series; the bidirectional convolutional recurrent neural network accepts the feature maps learned by the selective kernel convolution module in the encoding path in the same stage, and the fused features output by the previous level bidirectional convolutional recurrent neural network; the first level bidirectional convolutional recurrent neural network receives the fused features output by the joint convolutional recurrent neural network in the adjacent deeper stage; the last level bidirectional convolutional recurrent neural network outputs the fused features to the decoding path of the same stage, and the joint convolutional recurrent neural network in the adjacent shallower stage.
[0046] Specifically, as attached Figure 1 As shown, in the Curv-Net (curve segmentation network) proposed in the present invention, the feature map learned in the shallow stage of the network has a higher spatial resolution and less semantic information, while the feature map learned in the deep stage of the network has a lower spatial resolution but contains richer semantic information. Therefore, the present invention designs Multi-Bi-ConvLSTM (joint convolutional recurrent neural network) as a nonlinear jump connection in Curv-Net, which can fully mine and fuse low-level detail information and high-level semantic information during the training process of Curv-Net, thereby effectively improving the accuracy of segmentation.
[0047] Among them, as attached Figure 4 As shown in Figure 2, ConvLSTM (Convolutional Circular Neural Network) is an effective recurrent neural network that uses a storage unit c t , input gate i t , hidden unit H t 、Forget gate t and output gate o tProcessing 2D spatial data. By cyclically using ConvLSTM for data conversion, the memory cells c t and hidden units H t Obtain long-term and short-term information.
[0048] Specifically, the input gate controls the information that enters the current memory unit; the forget gate determines how much information from the previous memory is forgotten; and the output gate controls the information that is output from the current memory unit. t , o t and f t ) access, update, and clear the accumulation of state information in the memory cell. If the input gate is turned on, new input data will be accumulated to the memory cell, and if the forget gate is activated, the past state of the memory cell will be forgotten. Output gate o t Controls the state of the latest storage unit c t Will it be passed to the final state H? t According to the above definition, ConvLSTM can be described by the following formula:
[0049] i t =σ(W i X *Input t +W i H *H t-1 ) (4)
[0050]
[0051] Among them, the W with superscripts (such as W i X , W i H etc.) are learnable weights, Input t represents the input information of ConvLSTM, σ represents the sigmoid function, * and ° represent the convolution function and Hadamard function respectively, and tanh(·) represents the nonlinear activation function.
[0052] Bi-ConvLSTM (bidirectional convolutional recurrent neural network), which obtains forward and backward sequence information at the same time, can effectively improve the prediction of the model, and is a further improvement on the original ConvLSTM. The original ConvLSTM only processes the past sequence from the forward direction, which will lose some valuable information in the reverse direction. To solve this problem, the bidirectional ConvLSTM (Bi-ConvLSTM) was proposed. Bi-ConvLSTM uses two ConvLSTMs as the forward and backward paths to process the input data forward and backward respectively, and then outputs the bidirectional spatial information through the following formula:
[0053]
[0054] in, and Respectively represent the forward and backward propagation functions of ConvLSTM, and denotes two learnable parameters.
[0055] The Multi-Bi-ConvLSTM module designed by the present invention is preferably composed of three Bi-ConvLSTM groups connected in series, wherein the input information of each Bi-ConvLSTM can be described by the following formula:
[0056] Input i =W Yt *Cat[F i , M i ]+B Yt ,i∈{1,2,3} (10)
[0057] Among them, * represents the convolution operation; Cat represents the connection operation; M i is the output of the previous Bi-ConvLSTM (bidirectional convolutional recurrent neural network), which can be calculated by formula (9); F i is the corresponding feature map learned at the same stage in the network encoding path; W Yt and B Yt are the weights and biases in the convolution operation, respectively.
[0058] Furthermore, the bidirectional convolutional recurrent neural network consists of a forward path and a backward path composed of two convolutional recurrent neural networks, which respectively perform forward processing and backward processing on the input data and output bidirectional spatial information.
[0059] Furthermore, the selective kernel convolution module filters the input feature map through grouped convolutions with three different kernel sizes, and generates different feature maps through batch normalization and ReLU activation function; the kernel sizes are 3×3, 5×5 and 7×7 respectively.
[0060] Furthermore, in each stage, the encoding path includes two selective kernel convolution modules with adaptive receptive field size and a pooling layer; the decoding path includes two selective kernel convolution modules and a deconvolution layer.
[0061] Embodiment 2
[0062] In order to solve the above problems, the present invention also provides a curve segmentation method applied to the above curve segmentation network, comprising the following steps:
[0063] S1. In each stage of learning feature maps, the encoding path uses two selective kernel convolution modules with adaptive receptive field sizes to extract rich multi-scale features of the input image;
[0064] S2. The encoding path uses a pooling layer to perform a maximum pooling operation as a downsampling operator to reduce the scale of the feature map.
[0065] S3. Based on the nonlinear skip connection method of the joint convolutional recurrent neural network, the feature maps learned by the encoding path in the same stage of the network are fused, and the feature information is gradually propagated from the deep stage of the network to the shallow stage, and finally the fused features are output to the decoding path;
[0066] S4. The decoding path uses two selective kernel convolution modules to process the input fusion features to obtain better position information;
[0067] S5. The decoding path uses a deconvolution layer to perform a deconvolution operation to restore the image.
[0068] Embodiment 3
[0069] In order to solve the above problems, the present invention also provides a curve segmentation method for an image, which uses the above-mentioned curve segmentation network to perform curve segmentation on an input image.
[0070] The following experiments further illustrate the content of the curve segmentation network described above and its beneficial effects compared with the prior art:
[0071] 1. The effectiveness and reliability of Curv-Net were verified on two color fundus datasets (DRIVE and CHASE_DB1) and one corneal nerve dataset (CCM-2). The details are as follows:
[0072] In order to verify the performance of Curv-Net proposed in the previous description, the present invention tests Curv-Net on two retinal blood vessel datasets (DRIVE and CHASE_DB1) and one corneal nerve fiber dataset (CCM-2). Table 1 gives the information of the datasets used.
[0073] Table 1 Datasets used in the experiment
[0074]
[0075] 1) Retinal blood vessel dataset
[0076] The present invention uses the DRIVE and CHASE_DB1 datasets to test the performance of Curv-Net in the retinal vessel segmentation task. The DRIVE dataset is obtained from the diabetic retinopathy screening program in the Netherlands. The dataset contains 40 color retinal images, and the size of each image is 565×584 pixels. The DRIVE dataset provides training and test subsets, that is, the training set contains 20 training samples and the test set contains 20 test samples. The CHASH_DB1 dataset has a total of 28 color retinal images with a resolution of 999×960 pixels. The CHASH_DB1 dataset uses 20 samples as training samples and the remaining 8 samples as test samples.
[0077] 2) Corneal nerve dataset
[0078] The present invention uses the CCM-2 dataset to evaluate the effectiveness of Curv-Net in the corneal nerve segmentation task. The original images of CCM-2 are provided by the University of Padua, which contains 120 corneal subbasal epithelial images from normal and pathological objects, and the size of each image is 384×384 pixels.
[0079] Experimental settings and evaluation indicators
[0080] 1) Experimental setup
[0081] The present invention uses Python3.6 programming tools to implement Curv-Net under the Keras (Tensorflow-backend) framework. In the experiment, the computer used in the present invention is a computer equipped with NVIDIA GeForce GTX1080 and 16G RAM. During the model training process, the initial learning rate of Curv-Net is set to 0.0001, the weight decay is set to 0.0005, and the parameters of Curv-Net are optimized using adaptive moment estimation (Adam). In order to better train the model, the image size in the DRIVE and CHASE_DB1 data sets is adjusted to 512×512 pixels, and the image size in the CCM-2 data set is adjusted to 384×384 pixels. In order to verify the performance of the proposed Curv-Net network, the present invention performs k-fold (k=5) cross-validation on the CCM-2 data set to test Curv-Net. For the DRIVE and CHASE_DB1 data sets, the present invention uses the official training and test subsets of these two data sets to train and test the performance of Curv-Net.
[0082] 2) Evaluation indicators
[0083] The present invention uses four commonly used evaluation indicators: FDR (false discovery rate), ACC (accuracy), SE (sensitivity), SP (specificity) and AUC (area under the receiver operating characteristic curve (ROC curve)) to evaluate the performance of Curv-Net proposed in the present invention. AUC is calculated using the code provided in the scikit-learn python library, and the calculation formulas of other evaluation indicators are as follows:
[0084]
[0085] Among them, TP, TN, FP and FN represent true positive, true negative, false positive and false negative respectively. The smaller the value of the evaluation index FDR, the better the segmentation result, while the larger the value of other evaluation indicators, the better the segmentation result.
[0086] Experimental results and analysis
[0087] 1) Retinal vessel segmentation results and analysis
[0088] Quantitative analysis: In this section, the proposed Curv-Net is compared with several other currently superior medical image segmentation methods (Roychowdhury, Qiaoliang Li, U-Net, Recurrent U-Net, R2U-Net, Dual Encoding U-Net, BCDU-Net, CE-Net and DDNet). Tables 2 and 3 give the segmentation results of different methods on the DRIVE dataset and CHASE_DB1 dataset, respectively. In Tables 2-3, except for U-Net, the results of all the comparison methods are provided by the papers that proposed these methods.
[0089] By analyzing the segmentation results in Table 2-3, the present invention can summarize the following points: 1) The performance of Roychowdhury, Qiaoliang Li and the original U-Net network is generally lower than that of other methods (Recurrent U-Net, R2U-Net, Dual EncodingU-Net, BCDU-Net, CE-Net, DDNet and Curv-Net). The reason for this phenomenon may be that Roychowdhury and Qiaoliang Li did not use the skip connection method to effectively fuse multi-level features, and although the U-Net network can fuse multi-level features through traditional CNN modules and simple linear skip connections, it is still difficult to capture enough discriminative features when processing curve structure segmentation tasks;
[0090] 2) Since Recurrent U-Net, R2U-Net, Dual Encoding U-Net, BCDU-Net, CE-Net, DDNet and Curv-Net all use some new modules to replace the original modules in the U-Net network or directly modify the U-Net network structure, they obtain better segmentation results than the original U-Net network; 3) The Curv-Net proposed in the present invention is generally better than all the comparison methods. This is because Curv-Net not only uses the SK module to learn more useful multi-scale information, but also uses Multi-BiConvLSTM as a nonlinear jump connection to effectively fuse the features learned by the network in the same stage, and propagate the features from the deep stage of the network to the shallow stage.
[0091] Table 2 Segmentation results of Curv-Net and other comparison methods on the DRIVE dataset
[0092]
[0093] Table 3 Segmentation results of Curv-Net and other comparison methods on the CHASE_DB1 dataset
[0094]
[0095] Qualitative analysis: To further demonstrate the effectiveness of the Curv-Net proposed in this paper, the segmentation results of Curv-Net on different datasets are visualized and compared with the results obtained by the ground truth of the dataset and the baseline method (U-Net). Figure 5-6 As shown in the attached Figure 5-6 In the figure, the area enclosed by the white dotted box is a representative area in each retinal vascular image, and the area enclosed by the dark gray solid box in the upper left corner or lower left corner is an enlarged version of the area in the white dotted box. Figure 5-6 It can be seen that the Curv-Net proposed in the present invention can obtain better segmentation results than U-Net, especially when the blood vessels are very small and have a low contrast with the background (such as the area surrounded by the white dotted box), Curv-Net can segment the blood vessels more accurately.
[0096] 2) Corneal nerve fiber segmentation results and analysis
[0097] Quantitative analysis: In this section, the proposed Curv-Net is compared with several other medical image segmentation methods (DeepRes-Unet, U-Net, U-Net++, AttentionU-Net and CS-Net) on the CCM-2 dataset, and the comparison results are shown in Table 4. As can be seen from Table 4, the SE and FDR values of the segmentation results obtained by Curv-Net are 0.8903 and 0.2547, respectively, which are better than all other comparison methods. In particular, compared with the U-Net network (the Curv-Net proposed in the present invention is an improvement based on the U-Net network structure), the SE value obtained by Curv-Net is about 0.1246 higher than that obtained by U-Net, and the FDR value obtained by Curv-Net is about 0.0818 lower than that obtained by U-Net. This shows that Curv-Net can effectively improve the segmentation performance of corneal nerve fibers by introducing SK and Multi-Bi-ConvLSTM modules into the U-Net network.
[0098] Table 4 Segmentation results of Curv-Net and other comparison methods on the CCM-2 dataset
[0099]
[0100] Qualitative analysis: as attached Figure 7 As shown in the figure, the corneal nerve fiber segmentation results on the CCM-2 dataset are given. Figure 7 The first column in the figure shows two original images, and the last three columns are the ground truth, the U-Net, and the Curv-Net. Figure 7 It can be seen that Curv-Net can obtain better segmentation results, that is, the segmentation results obtained by Curv-Net contain more details of corneal nerve fibers (such as the area shown in the gray solid box) than the segmentation results obtained by U-Net. Figure 5-6 The retinal blood vessel segmentation results shown in the figure are similar. Curv-Net can not only accurately segment very thin corneal nerve fibers with low contrast with the background, but also reduce the segmentation error rate of corneal nerve fibers (as shown in the gray dotted box in the figure). This fully verifies the effectiveness of Curv-Net in targeting the curve detail features when segmenting images.
[0101] Ablation Experiment Results and Analysis
[0102] In order to verify the importance of the SK module and the Multi-Bi-ConvLSTM module in Curv-Net, the present invention tests the performance of the SK module and the Multi-Bi-ConvLSTM module respectively. Specifically, since the basic component of Curv-Net is the U-Net architecture, the present invention uses U-Net as the baseline and records the segmentation results of different networks by gradually introducing different modules into the baseline. 1) The SK module is used to replace the convolution operation in U-Net, and it is recorded as Baseline+SK; 2) In order to explain why the Multi-Bi-ConvLSTM module is used in U-Net instead of directly using Bi-ConvLSTM as a jump connection, the present invention introduces the original Bi-ConvLSTM into the Baseline+SK structure, and records it as Baseline+SK+bi-ConvLSTM; 3) The Multi-Bi-ConvLSTM module proposed by the present invention is introduced into the Baseline+SK structure (this is the final setting of the Curv-Net proposed by the present invention). Tables 5-7 give the results of the ablation experiment.
[0103] 1) Ablation experiment on retinal vascular dataset
[0104] Table 5-6 shows the ablation experimental results on two retinal vascular datasets (DRIVE and CHASE_DB1). From Table 5-6, it can be seen that: 1) Baseline+SK can obtain better segmentation results than Baseline. For example, compared with Baseline, Baseline+SK improves the SE value by 0.1082 and the SP value by 0.0197 on the CHASE_DB1 dataset. This shows that after the introduction of the SK module, Baseline+SK can use convolution kernels of different sizes to adaptively and dynamically select features, thereby improving the blood vessel segmentation performance; 2) Compared with Baseline+SK, Baseline+SK+Bi-ConvLSTM further improves the segmentation performance. For example, on the DRIVE and CHASE_DB1 datasets, Baseline+SK+Bi-ConvLSTM increases the SE value by 0.0187 and 0.0014, respectively, which proves that it is feasible to use Bi-ConvLSTM as a non-linear jump connection; 3) When the Multi-Bi-ConvLSTM module proposed in the present invention is used instead of Bi-ConvLSTM as a non-linear jump connection (this is the final configuration of the Curv-Net proposed in the present invention), the segmentation result obtained is better than Baseline+SK+Bi-ConvLSTM. For example, on the DRIVE and CHASE_DB1 datasets, the SE value increases by 0.0152 and 0.0180, respectively, which proves the effectiveness of the Multi-Bi-ConvLSTM module proposed in the present invention for curve detail feature recognition.
[0105] Table 5 Ablation experiment results on DRIVE dataset
[0106]
[0107] Table 6 Ablation experiment results on CHASE_DB1 dataset
[0108]
[0109] 2) Ablation experiment on corneal nerve fiber dataset
[0110] Table 7 shows the ablation experiment results on the corneal nerve fiber dataset CCM-2. The results in Table 7 show that: 1) The SE and FDR values of the segmentation results of Baseline+SK and Baseline+SK+Bi-ConvLSTM are better than those of Baseline; 2) Curv-Net introduces SK and Multi-Bi-ConvLSTM modules into U-Net, which can obtain better corneal nerve fiber segmentation results, such as SE value of 0.8903 and FDR value of 0.2547. These conclusions are consistent with the conclusions drawn from the segmentation results of the retinal vascular dataset (as shown in Tables 5-6), thus verifying that the Curv-Net proposed in the present invention is effective, that is, the performance of Curv-Net is better than that of Baseline and other configurations (Baseline+SK and Baseline+SK+Bi-ConvLSTM).
[0111] Table 7 Ablation experiment results on CCM-2 dataset
[0112]
[0113] The present invention is further improved based on U-Net, using the SK (selective kernel convolution) module to adaptively extract the multi-scale features of the input image, and then designing a Multi-Bi-ConvLSTM (joint convolutional recurrent neural network) as the encoding path and decoding path of the same stage of the nonlinear jump connection segmentation network, so as to fuse the information learned by the network at the same stage. At the same time, the feature information is gradually transferred from the deep layer of the network to the shallow layer through the jump connection. Compared with the U-Net in the prior art, the features captured by Curv-Net (curve segmentation network) can simultaneously contain more detailed information and high-level semantic information, thereby improving the segmentation performance.
[0114] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A curve segmentation network for medical image segmentation, comprising multiple stages from shallow to deep for learning feature mapping, each stage being provided with an encoding path and a corresponding decoding path; characterized in that, A selective kernel convolution module is embedded in both the encoding path and the decoding path; a nonlinear jump connection is connected between the encoding path and the decoding path; The selective kernel convolution module filters the input feature map through group convolution with three different kernel sizes, and generates different feature maps through batch normalization and ReLU activation function; the kernel sizes are: 3×3, 5×5 and 7×7; The selective kernel convolution module extracts multi-scale features from the input image; The nonlinear skip connection is composed of a joint convolutional recurrent neural network, a bidirectional convolutional recurrent neural network or a convolutional recurrent neural network; The nonlinear jump connection is used to fuse the features learned by the network in the same stage, and propagate the features from the deep stage of the network to the shallow stage, so as to improve the accuracy of image segmentation.
2. The curve segmentation network according to claim 1, characterized in that: The joint convolutional recurrent neural network includes: a bidirectional convolutional recurrent neural network corresponding to the number of the input feature maps, and multiple bidirectional convolutional recurrent neural networks are connected in series; the bidirectional convolutional recurrent neural network accepts the feature maps learned by the selective kernel convolution module in the encoding path in the same stage, and the fused features output by the bidirectional convolutional recurrent neural network in the previous stage; the first-level bidirectional convolutional recurrent neural network receives the fused features output by the joint convolutional recurrent neural network in the adjacent deeper stage; the last-level bidirectional convolutional recurrent neural network outputs the fused features to the decoding path in the same stage, and the joint convolutional recurrent neural network in the adjacent shallower stage.
3. The curve segmentation network according to claim 2, characterized in that: The bidirectional convolutional recurrent neural network is composed of a forward path and a backward path respectively formed by two convolutional recurrent neural networks, which respectively perform forward processing and backward processing on input data and output bidirectional spatial information.
4. The curve segmentation network according to claim 1, characterized in that: In each stage, the encoding path includes two selective kernel convolution modules with adaptive receptive field size and a pooling layer; the decoding path includes two selective kernel convolution modules and a deconvolution layer.
5. A curve segmentation method applied to the curve segmentation network according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1. In each stage of learning feature maps, the encoding path uses two selective kernel convolution modules with adaptive receptive field sizes to extract rich multi-scale features of the input image; S2. The encoding path uses a pooling layer to perform a maximum pooling operation as a downsampling operator to reduce the scale of the feature map; S3. Based on the nonlinear skip connection method of the joint convolutional recurrent neural network, the feature maps learned by the encoding path in the same stage of the network are fused, and the feature information is gradually propagated from the deep stage of the network to the shallow stage, and finally the fused features are output to the decoding path; S4. The decoding path uses two selective kernel convolution modules to process the input fusion features to obtain better position information; S5. The decoding path uses a deconvolution layer to perform a deconvolution operation to restore the image.
6. A curve segmentation method for an image, characterized in that: The curve segmentation network as claimed in any one of claims 1 to 4 is used to perform curve segmentation on the input image.
