A method for segmenting blood vessels and nerves in neurosurgery microsurgery images
By converting microsurgery videos into true color images and using the improved FastFCN network for blood vessel and nerve segmentation, the difficult problem of blood vessel and nerve segmentation in neurosurgery microsurgery is solved, high-precision automatic segmentation is achieved, the burden on doctors is reduced, and surgical efficiency is improved.
Patent Information
- Application Number
- CN202211039882.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-29
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2042-08-29
AI Technical Summary
Existing technologies lack effective segmentation methods for blood vessels and nerves in true color images during neurosurgery microsurgery, which makes segmentation time-consuming and labor-intensive for doctors. In addition, automatic segmentation relies on a large amount of medical image training, which is inefficient.
Neurosurgery microsurgery videos were converted into 24-bit true color images, the dataset was annotated and data augmentation was performed, and the improved FastFCN network model was used for training. The Xception_D backbone network with feature distillation module was added to achieve accurate segmentation of blood vessels and nerves.
It improves the segmentation accuracy of blood vessels and nerves in neurosurgery microsurgery images, reduces doctors' manual segmentation time, provides an efficient AI surgical assistant, reduces doctors' mental stress, and improves surgical speed and safety.
Smart Images
Figure CN115311249B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing and segmentation, and in particular to a method for segmenting blood vessels and nerves in neurosurgery microsurgery images. Background Art
[0002] Currently, medical examinations are becoming safer, more efficient, and more accurate. However, the massive amount of medical image data, beyond diagnostics, is placing an increasing burden on professional physicians. Currently, commonly used image segmentation techniques include manual, semi-automatic, and automatic segmentation. Manual segmentation, performed manually by physicians, produces accurate and clinically significant results. However, it is time-consuming and labor-intensive, and relies on subjective judgment, making it unsuitable for widespread adoption and application. Semi-automatic segmentation requires physicians to manually segment key areas, with the remainder completed by a machine. While this method improves segmentation speed while maintaining a certain level of accuracy, it still requires manual segmentation and is prone to high repetitiveness. With the advent of deep learning, automatic segmentation has become possible. This eliminates the need for manual segmentation by physicians, enabling machines to learn segmentation quickly using a large number of medical images, significantly improving physician convenience. However, the drawback is the large number of medical images required. The explosive growth of deep learning has accelerated the transition from traditional medical image processing to deep learning. The application of deep learning to address the segmentation of blood vessels and nerves in neurosurgery holds significant benefits for both physicians and patients. Medical images of cerebral blood vessels are generally acquired through methods such as computed tomography angiography (CTA), digital subtraction angiography (DSA), and magnetic resonance angiography (MRA). Currently, there is no public technology for extracting true color images used in neurosurgery microsurgery and segmenting blood vessels and nerves. Summary of the Invention
[0003] The purpose of the present invention is to provide a method for segmenting blood vessels and nerves in neurosurgery microsurgery images, so as to achieve the function of accurately segmenting blood vessels and nerves during surgery, reduce the time and energy spent by doctors during surgery, and improve surgical efficiency.
[0004] To achieve the above object, the present invention adopts the following technical solutions:
[0005] A method for segmenting blood vessels and nerves in neurosurgery microsurgery images comprises the following steps:
[0006] a) acquiring a neurosurgery microsurgery video captured by a surgical microscope, and converting the neurosurgery microsurgery video into a 24-bit true color image;
[0007] b) Annotate each image with segmentation labels of blood vessels and nerves to obtain a neurosurgery microsurgery image dataset;
[0008] c) performing data augmentation on the neurosurgery microsurgery image dataset;
[0009] d) dividing the data-enhanced neurosurgery microsurgery image dataset into a training set, a validation set, and a test set;
[0010] e) The backbone network Resnet101 in the basic network FastFCN is replaced with the improved Xception_D to obtain the improved FastFCN network model, where the improved Xception_D is a network obtained by adding feature distillation modules to the input stream, intermediate stream, and output stream of Xception_65 respectively. The feature distillation module includes one 3×3 depthwise separable convolutional layer, two standard 1×1 convolutional layers, two shallow residual blocks, and one connection layer responsible for fusing features from the 1×1 convolutional layer and the shallow residual block. The shallow residual block includes a 3×3 convolutional layer, a skip connection, and an activation unit;
[0011] f) Using the training set to train the improved FastFCN network model, saving the model with the best performance on the validation set as the optimal model, and using the test set to complete the network test of the optimal model. The tested optimal model is used to segment blood vessels and nerves in neurosurgical microsurgery images to obtain a neurosurgical microsurgery image segmentation result map.
[0012] The present invention has the following beneficial effects: After converting neurosurgery microsurgery videos into medical true-color images, the present invention uses the FastFCN network as the base network and makes modifications. A feature distillation module is added to the backbone network to further refine the features, resulting in an improved FastFCN network model. After training and testing, a model is ultimately obtained that can be used for segmenting blood vessels and nerves in neurosurgery microsurgery images, achieving accurate segmentation of blood vessels and nerves in neurosurgery microsurgery images. The segmentation results retain more feature information, the target boundary information is more complete, and the semantic information is clearer, improving the accuracy of semantic segmentation of cerebral vascular images and providing effective support for future intelligent medical treatment.
[0013] The deep learning-based method for segmenting blood vessels and nerves in neurosurgery microsurgery images, presented in this paper, addresses the flexible deformation of brain structures and enables accurate positioning and segmentation. This enables the creation of an AI surgical assistant that is tireless, possesses comprehensive anatomical knowledge, and provides precise positioning during surgery. This approach reduces surgeon stress, speeds up surgery, and reduces surgical trauma and complications. This method aligns with national industrial planning for intelligent healthcare and facilitates the future development of intelligent hospitals. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 A flowchart of a method for segmenting blood vessels and nerves in neurosurgery microsurgery images provided by an embodiment of the present invention;
[0015] Figure 2 This is the network structure diagram of the improved Xception_D in the present invention;
[0016] Figure 3 This is a network structure diagram of the feature distillation module in the present invention;
[0017] Figure 4 This is a network structure diagram of the shallow residual block in the present invention;
[0018] Figure 5 This is the improved FastFCN network model structure diagram in the present invention;
[0019] Figure 6 The training loss curves of the basic network FastFCN and the improved FastFCN network model. DETAILED DESCRIPTION
[0020] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.
[0021] In one embodiment, the present invention provides a method for segmenting blood vessels and nerves in neurosurgery microsurgery images. The method is a deep learning-based image segmentation method and specifically includes the following steps:
[0022] a) Obtain neurosurgery microsurgery videos captured by a surgical microscope and convert them into 24-bit true color images.
[0023] Specifically, the neurosurgery microsurgery video in this step can be captured using a ZEISS OPMI@VARIO 700 surgical microscope in MPG format. Image processing software is used to convert the neurosurgery microsurgery video into a 24-bit true color JPG image. The neurosurgery microsurgery video in this step captures a portion of the entire video captured by the surgical microscope, specifically the period between the release of cerebrospinal fluid and the suturing of the dura mater.
[0024] b) Annotate each image with segmentation labels of blood vessels and nerves to obtain a neurosurgery microsurgery image dataset.
[0025] After obtaining the images corresponding to the neurosurgery microsurgery videos, data annotation software such as LabelMe was used to create segmentation labels for the images. Experts then annotated the blood vessels and nerves in each neurosurgery microsurgery image, resulting in a number of labeled neurosurgery microsurgery images. These images constitute the neurosurgery microsurgery image dataset. The segmentation labels include: cn5 for the trigeminal nerve, cn7 for the facial nerve, cn9 for the glossopharyngeal nerve, cn10 for the vagus nerve, aica for the anterior inferior cerebellar artery, pica for the posterior inferior cerebellar artery, aica+cn7 for the anterior inferior cerebellar artery and facial nerve, pica+cn7 for the posterior inferior cerebellar artery and facial nerve, and pv for the petrosal vein.
[0026] c) Data augmentation is performed on the acquired neurosurgery microsurgery image dataset.
[0027] The images in the neurosurgery microsurgery image dataset were resized to a uniform size of 512×512. Next, each image in the dataset was horizontally flipped with a probability of 0.5. Random cropping was applied to randomly reduce the image to 512×512. Then, the image was processed with a random Gaussian blur with a probability of 0.5. Finally, the image enhancement operation was completed using a normalization strategy.
[0028] d) The data augmented neurosurgery microsurgery image dataset is divided into training set, validation set and test set.
[0029] After converting the neurosurgery microsurgery video into 24-bit true color images, 1993 images are obtained. In step d), the 1993 images are divided into a training set containing 1638 images, a validation set containing 181 images, and a test set containing 174 images.
[0030] e) The backbone network Resnet101 in the basic network FastFCN is replaced with the improved Xception_D to obtain the improved FastFCN network model, where the improved Xception_D is the network obtained by adding feature distillation modules to the input stream, intermediate stream and output stream of Xception_65 respectively. The feature distillation module includes 1 3×3 depth-separable convolution layer, 2 standard 1×1 convolution layers, 2 shallow residual blocks and 1 connection layer responsible for fusing features from the 1×1 convolution layer and the shallow residual block. The shallow residual block includes a 3×3 convolution layer, a skip connection and an activation unit.
[0031] Specifically, the improved network structure of Xception_D is as follows Figure 2 As shown in Figure 1, the improved Xception_D is a network obtained by adding the Feature Distillation Block (FDB) to the input stream, intermediate stream, and output stream of Xception_65. FDB includes a 3×3 depthwise separable convolutional layer, two standard 1×1 convolutional layers, two shallow residual blocks (SRBs), and a connection layer, as shown in Figure 1. Figure 3 As shown in Figure 2, the use of 1×1 convolution for channel reduction in FDB is more effective than in other CNN models. 1×1 convolution greatly reduces the number of parameters, and FDB is located on the main body of Xception_65, taking into account the spatial context, has better refinement characteristics, and can introduce fine-grained residual learning in the network. SRB is the main building block of FDB, which consists of 3×3 convolution layers, skip connections, and activation units, as shown in Figure 2. Figure 4 Compared with ordinary convolution, SRB can benefit residual learning without introducing additional parameters, and can better utilize the ability of residual learning, so that the network can maximize the benefits of residual learning while maintaining sufficient lightweight.
[0032] Specifically, Xception_65 consists of an input stream, an intermediate stream, and an output stream, with a total of 36 convolutional layers, which can be divided into 14 modules. The input stream includes four modules. The first module consists of two standard 3×3 convolutional layers without linear residual connections. The second, third, and fourth modules all consist of two 3×3 depthwise separable convolutional layers and one max pooling layer, with linear residual connections. The input stream of the improved Xception_D is obtained by replacing the first 3×3 depthwise separable convolutional layer in the second, third, and fourth modules with FDB and removing the max pooling layer.
[0033] The intermediate stream of Xception_65 includes 8 modules with the same structure, each of which includes 3 3×3 depth-separable convolutional layers and linear residual connections. Only one module is extracted for deletion and replacement to complete the module improvement, and then the improved intermediate stream of Xception_D is obtained by combining 16 improved modules with the same structure (the other modules of the original intermediate stream of Xception_65 are deleted, and the intermediate stream of the improved Xception_D is composed only of the improved modules). Deletion and replacement of the extracted module means replacing the first 3×3 depth-separable convolutional module in the module with FDB and deleting the last 3×3 depth-separable convolutional layer.
[0034] The output stream of Xception_65 consists of two modules. The first module is composed of two 3×3 depth-separable convolutional layers, one maximum pooling layer and a linear residual connection. The second module is composed of two 3×3 depth-separable convolutional layers and one average pooling layer. There is no linear residual connection. The first 3×3 depth-separable convolutional layer in the first module is replaced by FDB, the maximum pooling layer is canceled, and the average pooling layer in the second module is replaced by a 3×3 depth-separable convolutional layer to obtain the output stream of the improved Xception_D.
[0035] f) The network structure of the improved FastFCN network model is as follows Figure 5 As shown, the training set and validation set obtained in step d) are imported into the network model, and training is performed in the network model according to the parameters shown in Table 1. During the training process, the loss function used is the cross entropy loss function, the "poly" learning strategy is used, and mini-batch stochastic gradient descent is used, and a total of 100 rounds of training are used.
[0036] Table 1 Training parameters
[0037]
[0038]
[0039] Specifically, during training, the images in the training set are imported into the improved FastFCN network model. First, the original image passes through the input stream, intermediate stream, and output stream of Xception_D, and finally a feature map with 2048 channels is extracted. Figure 5Conv1 represents the feature map extracted by the 3×3 standard convolution layer with a stride of 2 in the Xception_D input stream, Conv2 and Conv3 represent the feature maps extracted by the first two 3×3 depth-separable convolution layers with a stride of 2 in the Xception_D input stream, Conv4 represents the feature map extracted by the 3×3 depth-separable convolution layer in the last module of the Xception_D intermediate stream, Conv5 represents the feature map extracted by the 3×3 depth-separable convolution layer in the first module of the Xception_D output stream, and Conv6 represents the feature map extracted by the 3×3 depth-separable convolution layer in the second module of the Xception_D output stream; then, the feature maps extracted by Conv4, Conv5 and Conv6 are imported into the Joint Pyramid Upsampling (JPU) module, which completes the extraction of multi-scale text information on the multi-layer feature map, and the output feature map is 1 / 8 of the original image; finally, the multi-scale context model, namely the Atrous Spatial Pyramid Pooling module (AtrousSpatial Pooling), is used to extract the multi-scale text information on the multi-layer feature map. Pyramid Pooling (ASPP) completes the final prediction.
[0040] After the training is completed, the model with the best performance on the validation set is saved as the optimal model, that is, the model corresponding to the maximum segmentation accuracy of the validation set, and the network test of the optimal model is completed using the test set. After the network test, the neurosurgical microsurgery image segmentation result map can be obtained. The optimal model after the test is used for the segmentation of blood vessels and nerves in neurosurgical microsurgery images, thereby obtaining the neurosurgical microsurgery image segmentation result map. Figure 6 The training loss curves of the basic network FastFCN and the improved FastFCN network model are shown. Figure 6 It can be observed that the improved FastFCN network model has lower loss than the original basic network FastFCN, with a significant decrease in loss and a continued downward trend. Furthermore, the improved FastFCN network model achieved a segmentation accuracy of 77.52%, higher than the original basic network FastFCN.
[0041] This paper converts neurosurgery microsurgery videos into medical true-color images. The FastFCN network is modified and a feature distillation module is added to the backbone network to further refine the features. This results in an improved FastFCN network model. After training and testing, a model for segmenting blood vessels and nerves in neurosurgery microsurgery images is developed, achieving accurate segmentation of blood vessels and nerves in neurosurgery microsurgery images. The segmentation results retain more feature information, resulting in more complete target boundary information and clearer semantic information. This improves the accuracy of semantic segmentation in cerebral vascular images and provides effective support for future intelligent medical treatment.
[0042] The deep learning-based method for segmenting blood vessels and nerves in neurosurgery microsurgery images, presented in this paper, addresses the flexible deformation of brain structures and enables accurate positioning and segmentation. This enables the creation of an AI surgical assistant that is tireless, possesses comprehensive anatomical knowledge, and provides precise positioning during surgery. This approach reduces surgeon stress, speeds up surgery, and reduces surgical trauma and complications. This method aligns with national industrial planning for intelligent healthcare and facilitates the development of future intelligent hospitals.
[0043] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0044] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.
Claims
1. A method for segmenting blood vessels and nerves in neurosurgery microsurgery images, characterized by: The following steps are involved: a) acquiring a neurosurgery microsurgery video captured by a surgical microscope, and converting the neurosurgery microsurgery video into a 24-bit true color image; b) Annotate each image with segmentation labels of blood vessels and nerves to obtain a neurosurgery microsurgery image dataset; c) performing data augmentation on the neurosurgery microsurgery image dataset; d) dividing the data-enhanced neurosurgery microsurgery image dataset into a training set, a validation set, and a test set; e) The backbone network Resnet101 in the basic network FastFCN is replaced with the improved Xception_D to obtain the improved FastFCN network model, where the improved Xception_D is a network obtained by adding feature distillation modules to the input stream, intermediate stream, and output stream of Xception_65 respectively. The feature distillation module includes one 3×3 depthwise separable convolutional layer, two standard 1×1 convolutional layers, two shallow residual blocks, and one connection layer responsible for fusing features from the 1×1 convolutional layer and the shallow residual block. The shallow residual block includes a 3×3 convolutional layer, a skip connection, and an activation unit; f) Using the training set to train the improved FastFCN network model, saving the model with the best performance on the validation set as the optimal model, and using the test set to complete the network test of the optimal model. The tested optimal model is used to segment blood vessels and nerves in neurosurgical microsurgery images to obtain a neurosurgical microsurgery image segmentation result map.
2. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to claim 1, characterized in that: The input stream of Xception_65 consists of 4 modules. The first module consists of 2 standard 3×3 convolutional layers without linear residual connections. The second, third, and fourth modules are all composed of 2 3×3 depthwise separable convolutional layers and 1 maximum pooling layer, and there are linear residual connections. After replacing the first 3×3 depthwise separable convolutional layer in the second, third, and fourth modules with a feature distillation module and removing the maximum pooling layer, the input stream of the improved Xception_D is obtained.
3. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to claim 1, characterized in that: The intermediate stream of Xception_65 includes 8 modules with the same structure. Each module includes 3 3×3 depth-separable convolutional layers and linear residual connections. Only one module is extracted for deletion and replacement to complete the module improvement. The improved intermediate stream of Xception_D is obtained by combining 16 improved modules with the same structure. Deletion and replacement of the module means replacing the first 3×3 depth-separable convolutional layer in the module with a feature distillation module and deleting the last 3×3 depth-separable convolutional layer.
4. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to claim 1, characterized in that: The output stream of Xception_65 includes two modules. The first module includes two 3×3 depth-separable convolutional layers, one maximum pooling layer and a linear residual connection. The second module includes two 3×3 depth-separable convolutional layers and one average pooling layer. After replacing the first 3×3 depth-separable convolutional layer in the first module with a feature distillation module and canceling the maximum pooling layer, and replacing the average pooling layer in the second module with a 3×3 depth-separable convolutional layer, the output stream of the improved Xception_D is obtained.
5. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to any one of claims 1 to 4, characterized in that: In step b), experts use data annotation software to annotate the segmentation labels of blood vessels and nerves in each image. The segmentation labels include: cn5 represents the trigeminal nerve, cn7 represents the facial nerve, cn9 represents the glossopharyngeal nerve, cn10 represents the vagus nerve, aica represents the anterior inferior cerebellar artery, pica represents the posterior inferior cerebellar artery, aica+cn7 represents the anterior inferior cerebellar artery and facial nerve, pica+cn7 represents the posterior inferior cerebellar artery and facial nerve, and pv represents the petrosal vein.
6. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to claim 5, characterized in that: The data annotation software adopts LabelMe software.
7. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to any one of claims 1 to 4, characterized in that: The data enhancement described in step c) includes the following steps: The images in the neurosurgery microsurgery image dataset are resized to a uniform size, each image in the dataset is horizontally flipped with a probability of 0.5; random cropping is applied to randomly crop the images to 512×512; The image is processed with random Gaussian blur with a probability of 0.5; the image data enhancement operation is completed using a standardization strategy.
8. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to any one of claims 1 to 4, characterized in that: Videos from the neurosurgical microsurgery video captured by a surgical microscope during the period from the release of cerebrospinal fluid to the suturing of the dura mater were selected. The videos were converted into 24-bit true color images to obtain 1993 images. The dataset was divided into a training set containing 1638 images, a validation set containing 181 images, and a test set containing 174 images according to the data set partitioning described in step d).
9. The method for segmenting blood vessels and nerves in neurosurgery microsurgery images according to any one of claims 1 to 4, characterized in that: The training parameters for training the improved FastFCN network model using the training set include: 41,000 iterations, a learning rate of 0.05, a momentum of 0.9, a weight decay of 0.00004, a crop size of 512×512, and a batch size of 4.