A multi-stage organ segmentation method and device based on hybrid perception
By adopting a multi-stage processing method of hybrid perception in medical image segmentation, using UNet network and hybrid perception module to extract features, the problems of background information influence and spatial information ignorance in the prior art are solved, and more efficient and accurate organ segmentation is achieved .
Patent Information
- Application Number
- CN202210544766.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-19
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-05-19
AI Technical Summary
When processing 3D CT images, existing medical image segmentation methods are susceptible to background information, resulting in inaccurate segmentation, and ignore the connection between the overall spatial information and neighborhood in the image, resulting in reduced accuracy in organ type judgment and computing resources. and time consumption.
Using a multi-stage organ segmentation method based on hybrid perception, two-stage processing is carried out: the first stage uses the UNet network for rough segmentation to remove redundant background information, and the second stage uses the hybrid perception module to extract local and global features for fine segmentation, combining feature fusion The module improves segmentation accuracy.
More accurate organ image segmentation is achieved, reducing the computational amount and running time, and improving the integrity and accuracy of segmentation.
Smart Images

Figure CN115115577B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of medical imaging technology, and in particular relates to a multi-stage organ segmentation method and device based on hybrid perception. Background Art
[0002] With the development of artificial intelligence, its application in medicine is increasing. Segmentation of medical images based on deep neural networks has become a popular research direction. Common medical images include computed tomography (CT), magnetic resonance imaging, and ultrasound. Automatic prediction and segmentation of human organ images based on medical images is an indispensable part of computer-aided medical diagnosis. It can significantly reduce the workload of doctors, reduce doctors' mistakes, speed up doctors' diagnosis, and increase the accuracy of doctors' judgment of diseases.
[0003] Most existing image segmentation methods use the entire 3D CT image as input and use a cascaded two-stage network to segment the image. Alternatively, the 3D or 2D image of the CT slice components is used as input, and the output result is finally reassembled into the original image to achieve the segmentation effect. The existing methods have the following disadvantages: First, directly using the entire CT image as input, there is a lot of background information, which makes the segmentation result easily affected by the background information, resulting in inaccurate segmentation. Using the two-stage prediction, although the memory usage is reduced and the classification accuracy is increased, a lot of computing resources and time resources are added. Second, using the sliced CT as input ignores the overall spatial information in the image, resulting in a significant reduction in the accuracy of organ type judgment; at the same time, it is also easy to ignore the spatial information between the neighbors in the image, and the organ cannot be regarded as a continuous whole in the 3D space, resulting in the segmentation result being prone to missing layers. Similarly, the redundancy of a large amount of background information will also lead to certain prediction biases in the results. Third, ordinary convolution ignores the global context information and only considers the connection between neighbors. Stacked convolution causes the receptive field to continue to decrease, and deep features lose the perception of global information. Summary of the invention
[0004] In order to solve the above problems existing in the prior art, the present invention provides a multi-stage organ segmentation method and device based on hybrid perception, which utilizes multi-stage processing to remove redundant background information, and utilizes hybrid perception to model the global and neighborhood, so as to further improve the accuracy of segmentation.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions.
[0006] In a first aspect, the present invention provides a multi-stage organ segmentation method based on hybrid perception, comprising the following steps:
[0007] Downsample the input CT image of size S0 to obtain image I1 of size S1;
[0008] Input I1 into the first stage segmentation network to obtain the first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R o Determine the smallest frame that contains all organs and record its size and position;
[0009] From R o The minimum frame is cut in the middle, and the image in the minimum frame is downsampled to size S2 and then input into the second stage segmentation network to obtain a second segmentation result, wherein the second stage segmentation network includes a hybrid perception module for extracting local features and global features;
[0010] After upsampling the second segmentation result to the size of the minimum frame, the second segmentation result is pasted to the position of the minimum frame.
[0011] Furthermore, the first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network with a hybrid perception module added.
[0012] Furthermore, the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module.
[0013] Furthermore, the local feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch is a 3×3×3 convolution block, the second branch includes a 2×2×2 average pooling layer and a 3×3×3 convolution block, and the third branch includes a 4×4×4 average pooling layer and a 3×3×3 convolution block.
[0014] Furthermore, the global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
[0015] In a second aspect, the present invention provides a multi-stage organ segmentation device based on hybrid perception, comprising:
[0016] A downsampling module is used to downsample the input CT image of size S0 to obtain an image I1 of size S1;
[0017] The first segmentation module is used to input I1 into the first stage segmentation network to obtain the first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R o Determine the smallest frame that contains all organs and record its size and position;
[0018] The second segmentation module is used to o The minimum frame is cut in the middle, and the image in the minimum frame is downsampled to size S2 and then input into the second stage segmentation network to obtain a second segmentation result, wherein the second stage segmentation network includes a hybrid perception module for extracting local features and global features;
[0019] The image pasting module is used to upsample the second segmentation result to the size of the minimum frame and then paste it to the position of the minimum frame.
[0020] Furthermore, the first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network with a hybrid perception module added.
[0021] Furthermore, the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module.
[0022] Furthermore, the local feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch is a 3×3×3 convolution block, the second branch includes a 2×2×2 average pooling layer and a 3×3×3 convolution block, and the third branch includes a 4×4×4 average pooling layer and a 3×3×3 convolution block.
[0023] Furthermore, the global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
[0024] Compared with the prior art, the present invention has the following beneficial effects.
[0025] The present invention downsamples an input CT image of size S0 to obtain an image I1 of size S1, inputs I1 into a first-stage segmentation network to obtain a first segmentation result, upsamples the first segmentation result to size S0, clips the smallest frame containing all organs, downsamples the image in the smallest frame to size S2, and then inputs it into a second-stage segmentation network to obtain a second segmentation result, upsamples the second segmentation result to the size of the smallest frame, and then pastes it to the position of the smallest frame, thereby realizing automatic segmentation of organ images. The present invention adopts two-stage image segmentation, and utilizes the rough segmentation of the first stage to remove most of the redundant information, reduce the overall calculation amount, and improve the running speed; utilizes the fine segmentation of the second stage based on hybrid perception (fusion of local features and global features), and can improve the integrity and accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 The present invention is a flowchart of a multi-stage organ segmentation method based on hybrid perception.
[0027] Figure 2 It is an overall flow chart of another embodiment of the present invention.
[0028] Figure 3 A network model structure diagram of the hybrid perception module.
[0029] Figure 4 This is a block diagram of a multi-stage organ segmentation device based on hybrid perception according to an embodiment of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the present invention clearer and more understandable, the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0031] Figure 1 This is a flow chart of a multi-stage organ segmentation method based on hybrid perception according to an embodiment of the present invention, comprising the following steps:
[0032] Step 101, down-sampling the input CT image of size S0 to obtain an image I1 of size S1;
[0033] Step 102: input I1 into the first-stage segmentation network to obtain a first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R o Determine the smallest frame that contains all organs and record its size and position;
[0034] Step 103: Crop the smallest bounding box from R o and downsample the image within the smallest bounding box to size S2, and then input it into the second-stage segmentation network to obtain the second segmentation result. The second-stage segmentation network includes a hybrid perception module for extracting local features and global features;
[0035] Step 104: After upsampling the second segmentation result to the size of the smallest bounding box, paste it to the position of the smallest bounding box.
[0036] This embodiment provides a multi-stage organ segmentation method based on hybrid perception. Before introducing the technical solution, the concepts of hybrid perception and multi-stage segmentation are first explained. When segmenting medical images, it is necessary to classify all the image pixels. Usually, due to the too large size of the image, it cannot be input into the network model at one time; if it must be input into the network model at one time, the effect of single segmentation will be very unsatisfactory. Therefore, it is necessary to segment the image through multiple times or multiple stages. In deep learning, different receptive fields can be obtained through convolutional kernels of different sizes and convolutional layers at different levels. The global information is concerned through large receptive fields, and the local neighborhood information is concerned through small receptive fields. Hybrid perception is to obtain global and neighborhood information by mixing the feature sizes of different receptive fields, and perform image segmentation based on the fusion of global features and local features. This is the meaning of hybrid perception and multi-stage image segmentation in this embodiment.
[0037] In this embodiment, step 101 is mainly used to downsample the input image. The input image of this embodiment includes CT images, but is not limited to CT images, and can also be magnetic resonance imaging, etc. The input original CT image is generally a complete image with a large size, and its size is denoted as S0. It is necessary to downsample it first to obtain an image I1 with size S1. Obviously, S1 < S0, as shown in the two leftmost modules in the first row. Figure 2 as shown by the two leftmost modules in the first row.
[0038] In this embodiment, step 102 is mainly used to perform the first-stage image segmentation. Input the image I1 with size S1 obtained in the previous step into the trained first-stage segmentation network, and output the first segmentation result with size S1. Upsample the first segmentation result to the original image size S0 to obtain R o , determine the smallest bounding box (minimum circumscribed cuboid) containing all organs according to the non-zero pixel points of R o and record the size and position of the smallest bounding box. The shape of the smallest bounding box is a cuboid, and its size and position can be represented by the three-dimensional coordinates of the 4 endpoints on the two diagonals of the cuboid. The first-stage image segmentation can be regarded as a rough segmentation stage, and the first-stage segmentation network can adopt the most commonly used Unet network. The operation implemented in step 102 corresponds to Figure 2From the first stage segmentation network to the rightmost module in the first row.
[0039] In this embodiment, step 103 is mainly used to perform the second stage image segmentation. o The smallest frame is cropped, and the image in the smallest frame is downsampled to size S2 and then input into the second stage segmentation network to obtain the second segmentation result, where S2>S1. Compared with the coarse image segmentation in the first stage, the second stage image segmentation can be regarded as fine image segmentation. In order to achieve fine image segmentation, unlike the first stage segmentation network, the second stage segmentation network adds a hybrid perception module, which uses the hybrid perception module to extract local features and global features at the same time, and fuses the extracted local features and global features. The image is classified based on the fused features, thereby outputting a high-precision segmented image to achieve fine segmentation. The operation implemented in step 103 corresponds to Figure 2 The second stage segmentation network in and the two modules on its right.
[0040] In this embodiment, step 104 is mainly used to resample and paste the second segmentation result. First, resample the second segmentation result to the size of the minimum frame, and then paste it to the position coordinates of the recorded minimum frame. A full-zero matrix can be designed according to the size of S1, and the resampled value of the second segmentation result is pasted to the element position corresponding to the minimum frame in the full-zero matrix to obtain the final segmentation result. The set full-zero matrix is equivalent to full background. Because the first stage of image segmentation removes most of the background by cropping, it is necessary to fill these removed parts with 0 in the segmentation result. The operation implemented in step 104 corresponds to Figure 2 The second stage segmentation network in the two modules on the left.
[0041] This embodiment performs two-stage image segmentation. The rough segmentation in the first stage can remove most of the redundant information, reduce the overall calculation amount and improve the running speed; the fine segmentation in the second stage based on hybrid perception can improve the integrity and accuracy of image segmentation.
[0042] As an optional embodiment, the first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network with a hybrid perception module added.
[0043] This embodiment defines the network structure of the first-stage segmentation network and the second-stage segmentation network. In this embodiment, the segmentation networks of the two stages are based on UNet as the basic network. The difference is that a hybrid perception module is added to the second-stage segmentation network, which is also the need for the second-stage segmentation network to achieve fine segmentation. UNet is the most widely used network in the field of medical image segmentation, and its performance is closely related to the fusion of multi-scale features in the network. UNet was first published on MICCAI in 2015, and then became the basic network for most medical image semantic segmentation tasks, and also inspired a large number of researchers to think about U-type semantic segmentation networks. The two biggest features of the UNet structure are the use of U-type structure and skip connection. Compared with FCN and Deeplab, UNet has performed 4 upsamplings and used skip connections in the same stage, instead of directly supervising and loss backpropagation on high-level semantic features, so that the feature map finally recovered integrates more low-level features, and also enables the integration of features of different scales, so that multi-scale prediction and deep supervision can be performed. The 4 upsamplings also make the segmentation map restore edge information more refined. Therefore, in addition to improving segmentation accuracy, UNet can also reduce network parameters and improve computational efficiency.
[0044] As an optional embodiment, the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module.
[0045] This embodiment provides a technical solution for a hybrid perception module. The hybrid perception module of this embodiment includes three parts, namely a local feature extraction module, a global feature extraction module and a feature fusion module. The input ends of the local feature extraction module and the global feature extraction module are connected in parallel, and are respectively used to extract local features and global features from the input image; the output ends are respectively connected to the input ends of the feature fusion module. The feature fusion module outputs the features after the local features and the global features are fused. Figure 3 A network structure diagram of the hybrid perception module is given. Figure 3 The left half above is the local feature extraction module, the right half is the global feature extraction module, and the convolution block below is the feature fusion module.
[0046] As an optional embodiment, the global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
[0047] This embodiment provides a technical solution for the local feature extraction module of the hybrid perception module. The specific network structure of the local feature extraction module is as follows: Figure 3 As shown in the left half above, the input features are first extracted by a 3×3×3 convolution block, and then divided into three paths, that is, processed by three branches at the same time: the first branch has only one 3×3×3 convolution block, the second and third branches first pass through average pooling layers of different sizes (2×2×2 and 4×4×4), and then resample to the original size by a 3×3×3 convolution block to obtain features of different scales, and finally mix features of different scales through a 3×3×3 convolution to obtain local features at different scales.
[0048] As an optional embodiment, the global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
[0049] This embodiment provides a technical solution for the global feature extraction module of the hybrid perception module. The specific network structure of the global feature extraction module is as follows: Figure 3 As shown in the right half above. The input features are first extracted through a 3×3×3 convolution block; then they are divided into three paths, that is, processed simultaneously by three branches: each branch first passes through an average pooling layer of 1×N2×N3, N1×1×N3, and N1×N2×1, and then passes through a 3×1×1, 1×3×1, and 1×1×3 convolution respectively to extract the features on the X, Y, and Z axes; finally, a 3×3×3 convolution is used to fuse the three-dimensional features together. This fusion method uses separate dimensions to extract features separately, greatly reducing parameters and calculations, and obtaining global features based on different three-dimensional dimensions.
[0050] Figure 4 This is a schematic diagram of the composition of a multi-stage organ segmentation device based on hybrid perception according to an embodiment of the present invention, the device comprising:
[0051] A downsampling module 11 is used to downsample the input CT image of size S0 to obtain an image I1 of size S1;
[0052] The first segmentation module 12 is used to input I1 into the first stage segmentation network to obtain a first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R oDetermine the smallest frame that contains all organs and record its size and position;
[0053] The second segmentation module 13 is used to o The minimum frame is cut in the middle, and the image in the minimum frame is downsampled to size S2 and then input into the second stage segmentation network to obtain a second segmentation result, wherein the second stage segmentation network includes a hybrid perception module for extracting local features and global features;
[0054] The image pasting module 14 is used to upsample the second segmentation result to the size of the minimum frame and then paste it to the position of the minimum frame.
[0055] The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be described in detail here. The same is true for the following embodiments, which will not be described in detail.
[0056] As an optional embodiment, the first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network with a hybrid perception module added.
[0057] As an optional embodiment, the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module.
[0058] As an optional embodiment, the local feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch is a 3×3×3 convolution block, the second branch includes a 2×2×2 average pooling layer and a 3×3×3 convolution block, and the third branch includes a 4×4×4 average pooling layer and a 3×3×3 convolution block.
[0059] As an optional embodiment, the global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
[0060] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A multi-stage organ segmentation method based on hybrid perception, characterized in that: The following steps are involved: Downsample the input CT image of size S0 to obtain image I1 of size S1; Input I1 into the first stage segmentation network to obtain the first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R o Determine the smallest frame that contains all organs and record its size and position; From R o The minimum frame is clipped, and the image in the minimum frame is downsampled to size S2 and then input into the second stage segmentation network to obtain a second segmentation result, wherein the second stage segmentation network includes a hybrid perception module for extracting local features and global features; wherein S2>S1; After upsampling the second segmentation result to the size of the minimum frame, pasting it to the position of the minimum frame; The first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network added with a hybrid perception module; the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module; the input ends of the local feature extraction module and the global feature extraction module are connected in parallel, respectively for extracting local features and global features from the input image, and the output ends are respectively connected to the input end of the feature fusion module, and the feature fusion module outputs features after the local features and the global features are fused; The local feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch is a 3×3×3 convolution block, the second branch includes a 2×2×2 average pooling layer and a 3×3×3 convolution block, and the third branch includes a 4×4×4 average pooling layer and a 3×3×3 convolution block; The global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
2. A multi-stage organ segmentation device based on hybrid perception, characterized in that: include: A downsampling module is used to downsample the input CT image of size S0 to obtain an image I1 of size S1; The first segmentation module is used to input I1 into the first stage segmentation network to obtain the first segmentation result, and upsample the first segmentation result to the original size S0 to obtain R o , in R o Determine the smallest frame that contains all organs and record its size and position; The second segmentation module is used to o The minimum frame is clipped, and the image in the minimum frame is downsampled to size S2 and then input into the second stage segmentation network to obtain a second segmentation result, wherein the second stage segmentation network includes a hybrid perception module for extracting local features and global features; wherein S2>S1; an image pasting module, configured to upsample the second segmentation result to the size of the minimum frame and then paste it into the position of the minimum frame; The first-stage segmentation network is a UNet network, and the second-stage segmentation network is a UNet network added with a hybrid perception module; the hybrid perception module includes a local feature extraction module, a global feature extraction module and a feature fusion module; the input ends of the local feature extraction module and the global feature extraction module are connected in parallel, respectively for extracting local features and global features from the input image, and the output ends are respectively connected to the input end of the feature fusion module, and the feature fusion module outputs features after the local features and the global features are fused; The local feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch is a 3×3×3 convolution block, the second branch includes a 2×2×2 average pooling layer and a 3×3×3 convolution block, and the third branch includes a 4×4×4 average pooling layer and a 3×3×3 convolution block; The global feature extraction module includes: a 3×3×3 convolution block, three branches connected to the output end of the convolution block, and a 3×3×3 convolution block connected to the output ends of the three branches; the first branch includes a 1×N2×N3 average pooling layer and a 3×1×1 convolution block, the second branch includes an N1×1×N3 average pooling layer and a 1×3×1 convolution block, and the third branch includes an N1×N2×1 average pooling layer and a 1×1×3 convolution block.
Citation Information
Patent Citations
Method and Apparatus for Accurate Computational Stereo Vision
AU2017100312A4
Tumor automatic segmentation system and method, and electronic equipment
CN111028242A
Nuclear magnetic image brain gray matter nucleus segmentation method based on convolutional neural network
CN112348779A
Real-time semantic segmentation method based on double-branch deep convolutional neural network
CN113421269A
Medical image multi-organ segmentation method and system
CN113808146A
