A pharyngeal organ segmentation method based on hollow residual feature pyramid
By introducing a DRP-FPN network with a hollow residual feature pyramid in the segmentation of nasopharyngeal organs, combined with a variety of network modules, the problems of segmentation accuracy and information loss in traditional networks are solved, and higher segmentation accuracy and boundary clarity are achieved.
Patent Information
- Application Number
- CN202111542686.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-12-16
AI Technical Summary
The boundaries of the nasopharyngeal and throat organs are blurred under the background of complex grayscale information. Traditional networks have low segmentation accuracy, especially the difficulty of segmentation of small-scale objects and unclear boundaries, and the FPN network causes continuous convolution operations. Image information is lost.
Using the DRP-FPN network based on the hollow residual feature pyramid, the residual cavity convolution module is introduced to perceive the receptive fields of different sizes in the FPN, and fuse it with the feature layers of different scales, combining RPN, ROI Align and FCN networks to generate accurate results. Recommended boxes and masks to ensure image information integrity.
It significantly improves the average accuracy, mDice coefficient and mIoU value of image segmentation, ensures boundary clarity and can effectively segment objects at smaller scales, avoiding image information loss.
Smart Images

Figure CN114299080B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision and pattern recognition, and in particular relates to a throat organ segmentation method based on a hollow residual feature pyramid. Background Art
[0002] In recent years, the incidence of nasopharyngeal diseases has been increasing. As the most direct and effective means of observing the mucosa of hollow organs, electronic laryngoscopes have become an important reference for clinicians to diagnose and minimally invasively treat nasopharyngeal diseases. Segmenting nasopharyngeal organs for qualitative and quantitative analysis is an important auxiliary means for doctors to make diagnoses and formulate treatment plans. However, the amount of image data in clinical diagnosis is huge, the time cost of manual diagnosis is high, and it is easy to miss and misdiagnose. Therefore, it is necessary to use computers to assist doctors in diagnosis. For example, nasopharyngeal carcinoma is one of the ten most common malignant tumors in my country. Due to the hidden site of onset and the variable symptoms and signs, early symptoms are often overlooked in endoscopic images. By the time cancer is discovered, the best time for treatment has been missed.
[0003] The nasopharyngeal organs are in a complex and ever-changing grayscale information confusion background, and the organ boundaries are blurred and similar to the background grayscale, which increases the difficulty of organ segmentation. Due to the complexity of the nasopharyngeal organs, multi-organ segmentation based on electronic endoscope images is very difficult. The traditional Unet and DeepLab series networks have the problem of low segmentation accuracy in nasopharyngeal multi-organ segmentation. Mask R-CNN and some improved Mask R-CNN segmentation algorithms show great potential in endoscopic image segmentation, but the FPN network in them inevitably causes image information loss in continuous convolution operations, which is not conducive to the segmentation of objects with unclear boundaries and smaller scales. Summary of the invention
[0004] The purpose of the present invention is to provide a method for throat organ segmentation based on a dilated residual pyramid (DRP-Mask) to overcome the problem in the prior art that image information is inevitably lost in continuous convolution operations, which is not conducive to the segmentation of objects with unclear boundaries and smaller scales.
[0005] To achieve the above purpose, the specific technical solutions of the present invention are as follows:
[0006] A method for segmenting pharyngeal organs based on a hollow residual feature pyramid, the method comprising the following steps:
[0007] Step 1: Use the DRP-FPN network to extract the features of the throat organs. In each lateral connection of FPN, a residual hole convolution module is introduced to perceive receptive fields of different sizes, and then fused with feature layers of different scales to obtain a feature map containing global information.
[0008] Step 2: Use the RPN network to set a predetermined number of candidate boxes for each feature point in the feature map, and determine the possibility of each candidate box containing the target. Generate candidate boxes of different sizes and aspect ratios on the feature map, and then use the non-maximum suppression algorithm to screen out the candidate box with the most likely position information to obtain an accurate suggestion box;
[0009] Step 3: Use the ROI Align method to extract candidate regions from the feature map according to the proposals box, and use bilinear interpolation to generate ROI regions of equal size and fixed size from the candidate regions;
[0010] Step 4: Use the classification network and bounding box regression network to calculate the classification loss and bounding box regression loss for the ROI area to determine the precise location and category of the proposed box; at the same time, use the FCN network to segment the ROI area to generate a mask.
[0011] Furthermore, in step 1, the residual dilated convolution module includes a dilated convolution structure and a residual structure, wherein the dilated convolution structure uses three parallel dilated convolution operations for each lateral connection, the convolution kernel is 3×3, and the dilation rates are 2, 4, and 8, respectively. A 3x3 convolution layer is then used to merge the features obtained by the dilated convolution operation, and finally the residual structure is used to combine the features after the dilated convolution with the lateral connection features to obtain new lateral features.
[0012] Compared with the prior art, the present invention has the following beneficial effects:
[0013] The present invention adopts the DRP-FPN network, introduces a DR module to each lateral connection in the FPN to perceive receptive fields of different sizes, and then fuses them with feature layers of different scales, greatly improving the global information extraction. The hole convolution controls the pixel area that the convolution kernel can affect through the hole rate, and expands the receptive field without increasing the amount of calculation. It effectively improves the average accuracy of image segmentation, the mDice coefficient and the mIoU value, does not cause the loss of image information, has clear boundaries and can achieve the segmentation of smaller-scale objects. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 Schematic diagram of a throat organ segmentation method based on a hollow residual feature pyramid in an embodiment of the present invention;
[0015] Figure 2(a) and (b) are respectively a DRP-FPN framework diagram for multi-scale feature extraction and a structural diagram of the feature extraction stage in an embodiment of the present invention;
[0016] Figure 3 A candidate frame coverage map of a region generation module in an embodiment of the present invention;
[0017] Figure 4 Generate a network structure diagram for the region in the embodiment of the present invention;
[0018] Figure 5 1 is the visualization result of different networks in the embodiments of the present invention. DETAILED DESCRIPTION
[0019] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0020] This paper proposes a two-stage instance segmentation model DRP-Mask. The first stage extracts image features and generates proposals through the proposed DRP module, and the second stage classifies the proposals and generates bounding boxes and masks.
[0021] like Figure 1 As shown, the present invention provides a method for segmenting throat organs based on a dilated residual pyramid (DRP-Mask), comprising the following steps:
[0022] Step 1: Use the DRP-FPN network to extract the features of the throat organs. In the FPN, each lateral connection introduces a residual hole convolution module to perceive receptive fields of different sizes, and then fuses them with feature layers of different scales to greatly improve the extraction of global information.
[0023] The specific operation of fusing multi-scale features is: by fusing the three features of upsampling, downsampling and lateral connection of the corresponding layer, a feature map containing detailed semantic information and spatial information is obtained, and the features of each scale obtained in the upsampling process are merged with the features of the corresponding scale during downsampling, that is, lateral connection, thereby greatly improving the network's ability to learn features.
[0024] The residual dilated convolution module includes a dilated convolution structure and a residual structure, wherein the dilated convolution structure uses three parallel dilated convolution operations for each lateral connection, with a convolution kernel of 3×3 and dilation rates of 2, 4, and 8 respectively, and then uses a 3x3 convolution layer to merge the features obtained by the dilated convolution operation. Finally, the residual structure is used to combine the features after the dilated convolution with the lateral connection features to obtain new lateral features.
[0025] Step 2: Use the RPN network to set a predetermined number of candidate boxes for each feature point in the feature map, and determine the possibility of each prior box containing the target. Generate candidate boxes of different sizes and aspect ratios on the feature map, and then use the non-maximum suppression algorithm to screen out the candidate box with the most likely position information to obtain an accurate suggestion box.
[0026] Step 3: Use the ROI Align method to extract candidate regions from the feature map according to the proposals box, and use bilinear interpolation to generate ROI regions of equal size and fixed size from the candidate regions;
[0027] Step 4: Use the classification network and bounding box regression network to calculate the classification loss and bounding box regression loss for the ROI area to determine the precise location and category of the proposed box; at the same time, use the FCN network to segment the ROI area to generate a mask.
[0028] Based on the above embodiment, the following is a further detailed description: given a nasopharyngeal image, we use the DRP-FPN network to extract rich semantic and spatial features to represent the image. The DRP-FPN network structure is as follows: Figure 2 As shown in (a), the residual hole convolution module (DR) is used to fuse the three features of upsampling, downsampling and lateral connection of the corresponding layer to extract rich features. In order to achieve end-to-end training, the residual hole convolution module DRP proposed in this embodiment outputs four scale features P5, P4, P3, and P2. First, the image is continuously downsampled to obtain features C1, C2, C3, C4, and C5 at five scales. Then, continuous upsampling is performed starting from C5 to obtain features M5, M4, M3, and M2. The features of each scale obtained during the upsampling process are fused with the features of the corresponding scale during downsampling, that is, horizontal connection. As shown Figure 2 As shown in (b), three dilated convolution operations are used in each lateral connection, with a convolution kernel of 3×3 and dilation rates of 2, 4, and 8 respectively. The features obtained by dilated convolution are fused with the lateral connection features to obtain a new lateral feature. The specific expression formulas are (1) and (2), as shown in Figure 2 As shown in (a), the 4-way feature maps P5, P4, P3, and P2 are generated with sizes of 192×192, 96×96, 48×48, and 24×24 respectively.
[0029] P i =Conv(Concatenate(Up(P i-1 ),DR(C i ))) (1)
[0030] DR(C i )=Concatenate(DConv(C i,2),
[0031] DConv(C i ,4),DConv(C i ,8))+C i (2)
[0032] The region generation module is a small neural network that uses a sliding window to scan the feature map. For each sliding window center point, a mapping point is found on the original pixel image, which is called an anchor point. With the anchor point as the center, k rectangular boxes are generated as candidate boxes. The selection of k satisfies formula (3). The generated candidate boxes are as follows: Figure 3 The input of this module is the multi-scale feature map generated by DRP, and the output is the category and coordinate information of the candidate box. Here, the candidate box with the largest IoU with the real box is selected as the positive sample. If it is classified as a foreground, it is considered that there is a high probability that there is an object in this candidate box. For the position regression of the candidate box, the model uses the coordinate offset between the real box and the candidate box for training. The structure is as follows Figure 4 As shown in the figure. Using this region generation module, the anchor with the highest probability of containing the target is retained, and its position and size are fine-tuned. In order to reduce repeated calculations caused by overlapping of some anchors, the non-maximum suppression algorithm (NMS) is used to remove redundant anchors and only retain the anchor with the highest NMS value.
[0033] k=scale×ratios(scale=3,ratios=3) (3)
[0034] After the region generation network, proposals are usually of different sizes, and it is necessary to extract features of equal and fixed size from these proposals of different sizes. ROI Align can achieve this requirement. The algorithm flow is as follows:
[0035] (1) Traverse each candidate region and keep the floating point boundaries unquantized.
[0036] (2) The candidate region is divided into kxk units, and the boundary of each unit is not quantized.
[0037] (3) Four fixed coordinate positions are calculated in each unit, the values of these four positions are calculated using the bilinear interpolation method, and then the maximum pooling operation is performed.
[0038] The classification network and bounding box regression network are used to calculate the classification loss and bounding box regression loss for the ROI area to determine the precise location and category of the proposal box; at the same time, the FCN network is used to segment the ROI area to generate a mask.
[0039] A self-built data set was used to verify the effectiveness of the nasopharyngeal organ segmentation method provided by the present invention. 8,000 training set images with an image resolution of 1060×1075 were obtained by cropping from the 350 collected video data, and 1,000 validation sets and test sets were obtained respectively. During the training process, random flipping, color jittering, noise interference and other enhancement techniques were used on the data set images to overcome the overfitting phenomenon that the neural network is prone to produce. Finally, a total of 16,000 training sets and 2,000 validation sets were obtained. During training, the image input shape was 768×768, and the batchsize was 8 for training. Each epoch was trained for 300 steps and verified for 20 steps, for a total of 100 epochs, the initial learning rate was set to 0.001, the attenuation weight was 0.0001, and the non-maximum suppression IoU threshold was 0.5.
[0040] Figure 5 The visualization results of different networks in the embodiments of the present invention show that the segmented image obtained by adopting the technical solution (DRP-Mask) of the present invention is the clearest and has a higher segmentation accuracy.
Claims
1. A throat organ segmentation method based on hollow residual feature pyramid, characterized in that: The method comprises the following steps: Step 1: Use the DRP-FPN network to extract the features of the throat organs. In each lateral connection of FPN, a residual hole convolution module is introduced to perceive receptive fields of different sizes, and then fused with feature layers of different scales to obtain a feature map containing global information. Step 2: Use the RPN network to set a predetermined prior box for each feature point in the feature map, and judge the possibility of each candidate box containing the target. Generate candidate boxes of different sizes and aspect ratios on the feature map, and then use the non-maximum suppression algorithm to screen out the candidate box with the most likely position information to obtain an accurate suggestion box; The region generation module is a small neural network that uses a sliding window to scan the feature map. For each sliding window center point, the mapping point is found on the original pixel image, which is called the anchor point. With the anchor point as the center, k rectangular boxes are generated as candidate boxes. The selection of k satisfies formula (3): k=scale×ratios; scale=3, ratios=3(3) Step 3: Use the ROIAlign method to extract candidate regions from the feature map according to the proposed box, and use bilinear interpolation to generate ROI regions of equal size and fixed size from the candidate regions; Step 4: Use the classification network and bounding box regression network to calculate the classification loss and bounding box regression loss for the ROI area to determine the precise location and category of the proposed box; at the same time, use the FCN network to segment the ROI area to generate a mask.
2. The method for pharyngeal organ segmentation based on hollow residual feature pyramid according to claim 1, characterized in that: In step 1, the residual dilated convolution module includes a dilated convolution structure and a residual structure, wherein the dilated convolution structure uses three parallel dilated convolution operations for each lateral connection, the convolution kernel is 3×3, and the dilation rates are 2, 4, and 8 respectively, and then a 3x3 convolution layer is used to merge the features obtained by the dilated convolution operation. Finally, the residual structure is used to combine the features after the dilated convolution with the lateral connection features to obtain new lateral features.
Citation Information
Patent Citations
Method and system for automatically segmenting esophagus cancer radiotherapy target area and organs at risk
CN110211139A
Object-level edge detection method based on deep residual network
CN110706242A
Parallel spatial pyramid model image segmentation method based on cavity convolution
CN113538444A
Semantic segmentation method and system applicable to ar
WO2021201422A1
