Segmentation method of choroidal atrophy in fundus medical images
By introducing cascaded multi-scale information interaction fusion module and strip pooling processing in U-Net network, the problem of insufficient feature extraction ability in choroid atrophy segmentation in fundus medical images is solved, the segmentation quality and efficiency are improved, and the Performance metrics.
Patent Information
- Application Number
- CN202011553484.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-24
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-12-24
AI Technical Summary
The existing medical image segmentation methods have problems such as insufficient extraction ability and low segmentation efficiency for different shape features during fundus medical imaging, which cannot meet the early diagnosis needs.
Build a cascading multi-scale information interaction and fusion network based on U-Net network. By setting up a cascading multi-scale information interaction and fusion module between the encoding and decoding networks, multiple scale feature maps are obtained for splicing and fusion, and strip pooling is adopted. Processing, improve feature extraction capability and segmentation efficiency.
It improves the feature extraction ability of complex structural goals with diverse shapes and scales, improves the segmentation quality and segmentation efficiency of choroid atrophy in fundus medical images, and significantly improves the Dice coefficient and intersecting ratio.
Smart Images

Figure CN112634234B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a segmentation method for choroidal atrophy in fundus medical images. Background Art
[0002] Medical imaging refers to the technology and process of obtaining images of internal tissues and organs of the human body or a part of the human body in a non-invasive manner for the purpose of medical treatment or medical research. According to the different implementation steps, medical imaging includes medical imaging technology and medical processing technology. With the help of medical imaging technology, medical personnel can more clearly understand the conditions of specific tissues and organs in the human body and thus provide more accurate and reasonable diagnosis and treatment plans. Medical image segmentation is a key technology in modern medical image processing. It is the basis for subsequent operations such as three-dimensional reconstruction and quantitative analysis of normal and diseased tissues.
[0003] Deep learning image segmentation methods accurately segment multiple segmentation targets in multiple scene images of multiple public data sets. They have the advantages of high accuracy and high generalization. In view of the above advantages of deep learning image segmentation methods, deep learning image semantic segmentation methods such as FCN (Full Convolutional Network) and U-Net segmentation network have been widely used. Full convolutional network FCN is applied to image segmentation due to its excellent feature extraction ability. U-Net uses skip connection and encoder-decoder structure to greatly improve the performance of medical image segmentation. Most other segmentation networks are improved on the basis of FCN and U-Net networks, for example, PSP-Net (Pyramid Scene Parsing Network), CE-Net (Context Encoder Network) and CPF-Net (Context Pyramid Fusion Network). The pyramid pooling module proposed by PSPNet can use pooling operations of different sizes on feature maps to obtain multi-scale features. CE-Net uses multiple convolution branches with different receptive fields to improve the model's ability to obtain multi-scale information. CPF-Net proposes scale-aware pyramid fusion, which also uses dilated convolutions with different dilation rates, and then fuses features of different scales in pairs.
[0004] Complications caused by pathological myopia are the main causes of visual impairment and blindness, and choroidal atrophy is one of the earliest pathological changes of pathological myopia. Automatic segmentation of choroidal atrophy is of great significance for early diagnosis. However, choroidal atrophy segmentation in fundus images is a difficult task due to the following reasons: the shape and area of choroidal atrophy are diverse, and the occurrence area, shape and area of choroidal atrophy in different degrees of pathological myopia are very different. Early pathological myopia is often accompanied by optic disc choroidal atrophy similar to high myopia. As the degree of pathological myopia deepens, diffuse chorioretinal atrophy, plaque chorioretinal atrophy and macular atrophy will occur, and the occurrence area, shape and area vary greatly.
[0005] Existing medical image segmentation methods, such as PSP-Net, CE-Net, and CPF-Net, mostly use dilated convolution or pyramid pooling modules. The segmentation network uses dilated convolution, which can obtain a large-resolution feature map, but also increases memory usage and computation. The pyramid pooling module uses parallel multi-scale pooling, and finally upsamples the features of each branch and merges them with the original feature map to obtain multi-scale features. However, since the pyramid pooling module uses a square pooling template, it is difficult to improve the segmentation performance of slender objects. Therefore, the existing medical image segmentation methods have problems such as insufficient ability to extract features of different shapes and low segmentation efficiency, and cannot meet the segmentation requirements of choroidal atrophy in fundus medical images. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a method for segmenting choroidal atrophy in fundus medical images, which is beneficial to improving its ability to extract features of different shapes and can improve the segmentation efficiency of choroidal atrophy in medical images.
[0007] In order to solve the above technical problems, the technical solutions provided by the present invention are as follows:
[0008] A method for segmenting choroidal atrophy in fundus medical images comprises the following steps:
[0009] S1) Construct a cascaded multi-scale information interaction fusion network based on U-Net network;
[0010] The U-Net network includes an encoding network and a decoding network, the encoding network includes multiple feature encoding modules, the decoding network includes multiple feature decoding modules, the encoding network adopts a residual network, and the feature decoding module is used to splice the upsampled feature map and the feature map output by the corresponding feature encoding module;
[0011] When constructing the cascaded multi-scale information interaction fusion network, a cascaded multi-scale information interaction fusion module is set between the encoding network and the decoding network, and the cascaded multi-scale information interaction fusion module is used to obtain multiple scale feature maps from the feature map output by the encoding network and splice and fuse the obtained multiple scale feature maps to form a multi-scale fusion feature map, and perform strip pooling processing on the feature map output by the encoding network to obtain a strip feature map, and splice the multi-scale fusion feature map, the strip feature map and the feature map output by the encoding network and output them to the encoding network;
[0012] S2) Inputting the fundus medical image to be processed into the cascaded multi-scale information interaction fusion network to perform choroidal atrophy segmentation.
[0013] In one embodiment, in the step S1), the cascaded multi-scale information interaction fusion module obtains multiple scale feature maps from the feature map output by the encoding network, and splices and fuses the obtained multiple scale feature maps to form a multi-scale fusion feature map. The method is: the feature map output by the encoding network is pooled multiple times with different sizes, each time a scale feature map is obtained by pooling, and each scale feature map is converted until the size specification of each scale feature map remains the same, and then all the scale feature maps with the same size specification are spliced and fused to form a multi-scale fusion feature map.
[0014] In one implementation manner, when all scale feature maps with the same size specifications are spliced and fused, all scale feature maps with the same size specifications are spliced according to the channel dimension, and after splicing, convolution, batch normalization and ReLu activation function processing are performed in sequence to achieve fusion.
[0015] In one of the embodiments, in the step S1), the method of performing strip pooling processing on the feature map output by the encoding network to obtain a strip feature map is: performing horizontal strip pooling on the feature map output by the encoding network to obtain a horizontal strip feature map, and performing vertical strip pooling on the feature map output by the encoding network to obtain a vertical strip feature map, and then adding the horizontal strip feature map and the vertical strip feature map and performing ReLu activation function processing to obtain the strip feature map.
[0016] In one of the embodiments, before adding the horizontal strip feature map and the vertical strip feature map, the horizontal strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling, and the vertical strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling.
[0017] In one of the embodiments, in the step S1), the feature decoding module is used to splice the upsampled feature map and the feature map output by the corresponding feature encoding module, and then sequentially perform convolution, batch normalization and ReLu activation function processing on the spliced feature map.
[0018] In one embodiment, the residual network adopts a ResNet34 network.
[0019] The present invention has the following beneficial effects: the segmentation method of choroidal atrophy in fundus medical images of the present invention can fully extract the features of complex structural targets with diverse shapes and scales, thereby improving its ability to extract features of different shapes, and is more suitable for the segmentation of choroidal atrophy in fundus medical images, thereby improving the segmentation quality and efficiency of choroidal atrophy in fundus medical images. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a structural schematic diagram of a method for segmenting choroidal atrophy in fundus medical images of the present invention;
[0021] Figure 2 yes Figure 1 Schematic diagram of the structure of the feature decoding module;
[0022] Figure 3 yes Figure 1 Schematic diagram of the structure of the medium cascade multi-scale information interaction fusion module-CMSI module;
[0023] Figure 4 yes Figure 3 A schematic diagram of the structure of the first processing branch in the CMSI module;
[0024] Figure 5 yes Figure 3 A schematic diagram of the structure of the second processing branch in the CMSI module;
[0025] Figure 6 This is a comparison of the segmentation results of choroidal atrophy in fundus medical images using different segmentation networks; DETAILED DESCRIPTION
[0026] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0027] like Figure 1-Figure 2 As shown, Figure 1 middle represents upsampling, Figure 2 middle This embodiment discloses a method for segmenting choroidal atrophy in fundus medical images, characterized in that it includes the following steps:
[0028] S1) Construct a cascaded multi-scale information interaction fusion network based on U-Net network;
[0029] The U-Net network includes an encoding network and a decoding network, the encoding network includes a plurality of feature encoding modules, the decoding network includes a plurality of feature decoding modules, the feature encoding modules correspond to the feature decoding modules, the encoding network adopts a residual network (ResNet) to effectively obtain the feature map of the input image, avoid gradient disappearance, and improve the convergence rate of the network; the feature decoding module is used to splice the up-sampled feature map and the feature map output by the corresponding feature encoding module to effectively reconstruct the high-resolution feature map;
[0030] When constructing the cascaded multi-scale information interaction fusion network, a cascaded multi-scale information interaction fusion module-CMSI module is set between the encoding network and the decoding network. The CMSI module is used to obtain multiple scale feature maps from the feature map output by the encoding network and splice and fuse the obtained multiple scale feature maps to form a multi-scale fusion feature map, and perform strip pooling on the feature map output by the encoding network to obtain a strip feature map, and splice the multi-scale fusion feature map, the strip feature map and the feature map output from the encoding network and output them to the encoding network;
[0031] That is, if Figure 3 As shown, the CMSI module includes a first processing branch and a second processing branch. The first processing branch is used to obtain multi-scale features from the feature map output by the encoding network and to splice and fuse the multi-scale features to form a multi-scale fused feature map. This method uses cascaded pooling and convolution to interactively fuse information at each scale through feature splicing to capture rich multi-scale features; the second processing branch is used to perform strip pooling processing on the feature map output by the encoding network to obtain a strip feature map, so as to obtain more feature information and establish connections between long-distance features.
[0032] S2) Inputting the fundus medical image to be processed into the cascaded multi-scale information interaction fusion network to segment the choroidal atrophy.
[0033] In the above structure, the cascaded multi-scale information interaction fusion module-CMSI module can efficiently capture multi-scale information through multi-scale and multi-shape pooling and increase the receptive field of convolution.
[0034] In one embodiment, if Figure 4As shown, in step S1), the cascaded multi-scale information interaction fusion module (CMSI module) obtains multiple scale feature maps from the feature map output by the encoding network, and the method of splicing and fusing the obtained multiple scale feature maps to form a multi-scale fusion feature map is as follows: the feature map output by the encoding network is subjected to multiple pooling processes of different sizes, each time a scale feature map is obtained by pooling, and each scale feature map is then converted until the size specification of each scale feature map remains the same, and then all the scale feature maps with the same size specification are spliced and fused to form a multi-scale fusion feature map.
[0035] In one implementation manner, when all scale feature maps with the same size specifications are spliced and fused, all scale feature maps with the same size specifications are spliced according to the channel dimension, and after splicing, convolution, batch normalization and ReLu activation function processing are performed in sequence to achieve fusion.
[0036] For example, when the CMSI module obtains multiple scale feature maps from the feature map output by the encoding network, and concatenates and fuses the obtained multiple scale feature maps to form a multi-scale fusion feature map: the CMSI module first performs adaptive pooling of different sizes on the feature map output by the encoding network to obtain feature maps of four scales, with sizes of 10×10, 6×6, 3×3, and 1×1, and then performs several pooling, 3×3 convolution, batch normalization, and ReLu activation on the feature map of each scale until the size of the feature map of each scale becomes 1×1, and the number of channels is 1 / 4 of the original feature map. Among them, the convolution kernel size used for the feature map of size 1×1 is also 1×1. In the fusion process of feature maps of different scales, they are concatenated according to the channel dimension, and then undergo 3×3 convolution, batch normalization, and ReLu activation.
[0037] In one embodiment, if Figure 5 As shown, in step S1), the method of performing strip pooling processing on the feature map output by the coding network to obtain the strip feature map is: performing horizontal strip pooling on the feature map output by the coding network to obtain the horizontal strip feature map, and performing vertical strip pooling on the feature map output by the coding network to obtain the vertical strip feature map, and then adding the horizontal strip feature map and the vertical strip feature map and performing ReLu activation function processing to obtain the final strip feature map. By combining the above-mentioned horizontal strip pooling and vertical strip pooling, more feature information can be captured, and information loss is not likely to occur, which is conducive to improving the final image segmentation accuracy.
[0038] Furthermore, before adding the horizontal strip feature map and the vertical strip feature map, the horizontal strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling processing, and the vertical strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling processing.
[0039] For example, when the CMSI module performs strip pooling on the feature map output by the encoding network to obtain a strip feature map: the feature map output by the encoding network is subjected to horizontal strip pooling and vertical strip pooling respectively, and two strip feature maps are obtained respectively, namely, the horizontal strip feature map and the vertical strip feature map. The two strip feature maps are respectively subjected to one-dimensional convolution, batch normalization and upsampling, and then the two are added together. Finally, the ReLu activation function is used to effectively establish the connection between long-distance features. Among them, the convolution kernel sizes of the two one-dimensional convolutions are 1×3 and 3×1, and the number of output channels is 1 / 4 of the original feature map. Finally, the CMSI module concatenates the multi-scale fusion feature map obtained by the first processing branch, the strip feature map obtained by the second processing branch, and the input feature map of the CMSI module, and then performs a 1×1 convolution to obtain the output feature map of the CMSI module. The size and number of channels of the output feature map are the same as its input feature map.
[0040] In one embodiment, in step S1), the feature decoding module is used to splice the upsampled feature map and the feature map output by the corresponding feature encoding module, and then sequentially perform convolution, batch normalization and ReLu activation function processing on the spliced feature map, and finally output the feature map after the above processing. It can be understood that the size and number of channels of the feature map output by the feature decoding module are the same as those of the feature map output by the corresponding feature encoding module.
[0041] In one implementation, the residual network (ResNet) uses the ResNet34 network to better ensure the convergence rate and computational efficiency. For example, ResNet34 includes 5 stages. The first stage includes a 7×7 convolution with a step size of 2, batch normalization, and Relu activation. The next 4 stages contain 3, 4, 6, and 3 residual modules, respectively. Each stage starts with downsampling once, and the number of channels doubles. ResNet34 downsamples 5 times in total, and the number of output channels is 64, 64, 128, 256, and 512, respectively.
[0042] Furthermore, the ResNet34 network is a network with its average pooling layer and fully connected layer removed to simplify calculations and improve computational efficiency.
[0043] In one of the implementations, a bilinear interpolation sampling method is used when sampling the output feature map of the feature decoding module.
[0044] In order to verify the effectiveness and practicality of the segmentation method of this embodiment, the present invention uses 400 fundus color images to train and test the designed network, and verifies the method by comparing the segmentation effects of multiple networks and ablation experiments:
[0045] 400 fundus color images were used as the dataset, including 161 normal images, 26 high myopia images, and 213 pathological myopia images. To reduce training time and video memory usage, we downsampled the images to 448×448. To increase the diversity of training data and prevent model overfitting, we used random left-right and up-down flipping, random brightness, random contrast, and random Gaussian blur as online data augmentation. In both the comparative experiment and the ablation experiment, four-fold cross validation was used;
[0046] In order to objectively evaluate the performance of the segmentation method of this embodiment, five evaluation indicators, including Dice coefficient, intersection over union (IoU), accuracy (Acc), sensitivity (Sens) and specificity (Spec), were used.
[0047] The cascaded multi-scale information interaction fusion network of this embodiment is denoted as CMSI-Net, the U-Net network using ResNet34 as the encoding network is used as the basic network of CMSI-Net, denoted as Baseline, and the second processing branch (strip pooling processing branch) in the CMSI module of this embodiment is added to Baseline, denoted as "Baseline+CMSI branch 2". The cascaded multi-scale information interaction fusion network of this embodiment is denoted as CMSI-Net and compared with the existing U-Net, Attention-UNet, CE-Net (Context Encoder Network), and CPF-Net (Context Pyramid Fusion Network). The comparison results are shown in Table 1, which is a comparison result table of CMSI-Net of this embodiment and other segmentation networks;
[0048] Table 1 Comparison results of the CMSI-Net network of this embodiment with other segmentation networks
[0049]
[0050]
[0051] As can be seen from Table 1, in the comparative experiment, the CMSI-Net proposed in this embodiment achieved better performance indicators than other segmentation networks, and its Dice coefficient, intersection-over-union ratio and sensitivity were significantly improved. Its Dice coefficient was 4.16% higher than that of U-Net, which also shows that the CMSI module proposed in this embodiment can well make up for the shortcomings of the U-Net network's insufficient ability to obtain contextual information. In addition, the performance of adding branch 2 in the CMSI module to the Baseline alone, that is, "Baseline+CMSI branch 2", can also be improved compared with other networks.
[0052] Also, see Figure 6 , Figure 6 The comparison of segmentation visualization results of different segmentation networks is shown. Figure 6 It can be seen that the CMSI-Net proposed in this embodiment has good adaptability to the differences in shape and size of the atrophic area, better suppresses the false positive and false negative phenomena, and obtains the best segmentation effect for atrophic lesions of different shapes and sizes, and its ability to extract multi-morphological and multi-scale lesion features is enhanced.
[0053] In summary, the segmentation method of choroidal atrophy in fundus medical images of this embodiment is improved on the existing U-Net segmentation network model. By introducing a cascaded multi-scale information interaction fusion module, the problems of insufficient multi-scale feature extraction capability of the existing segmentation network and incomplete feature extraction information of different shapes are overcome, and the ability of the segmentation network to effectively extract and fuse global multi-scale context information is improved. It can fully extract the features of complex structure targets with diverse shapes and scales, and improve its ability to extract features of different shapes. It is more suitable for the segmentation of choroidal atrophy in fundus medical images, improves the segmentation quality of choroidal atrophy in fundus medical images, and has small computational and memory requirements, thereby improving segmentation efficiency.
[0054] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A segmentation method for choroidal atrophy in fundus medical images. It is characterized in that The following steps are involved: S1) constructing a cascaded multi-scale information interaction fusion network based on a U-Net network, wherein the cascaded multi-scale information interaction fusion network is suitable for the segmentation of choroidal atrophy in fundus medical images; The U-Net network includes an encoding network and a decoding network, the encoding network includes multiple feature encoding modules, the decoding network includes multiple feature decoding modules, the encoding network adopts a residual network, and the feature decoding module is used to splice the upsampled feature map and the feature map output by the corresponding feature encoding module; When constructing the cascaded multi-scale information interaction fusion network, a cascaded multi-scale information interaction fusion module is set between the encoding network and the decoding network, and the cascaded multi-scale information interaction fusion module is used to obtain multiple scale feature maps from the feature map output by the encoding network and splice and fuse the obtained multiple scale feature maps to form a multi-scale fusion feature map, and perform strip pooling processing on the feature map output by the encoding network to obtain a strip feature map, and splice the multi-scale fusion feature map, the strip feature map and the feature map output by the encoding network and output them to the encoding network; The cascaded multi-scale information interaction fusion module concatenates the multi-scale fusion feature map, the strip feature map and the input feature map of the cascaded multi-scale information interaction fusion module and performs a 1×1 convolution to obtain the output feature map of the cascaded multi-scale information interaction fusion module; the input feature map of the cascaded multi-scale information interaction fusion module is the feature map output by the encoding network; S2) Inputting the fundus medical image to be processed into the cascaded multi-scale information interaction fusion network to segment choroidal atrophy.
2. The method for segmenting choroidal atrophy in fundus medical images according to claim 1, It is characterized in that In the step S1), the cascaded multi-scale information interactive fusion module obtains multiple scale feature maps from the feature map output by the encoding network, and splices and fuses the obtained multiple scale feature maps to form a multi-scale fusion feature map. The method is: the feature map output by the encoding network is subjected to multiple pooling processes of different sizes, each time a scale feature map is obtained by pooling, and each scale feature map is then converted until the size specifications of each scale feature map remain the same, and then all the scale feature maps with the same size specifications are spliced and fused to form a multi-scale fusion feature map.
3. The method for segmenting choroidal atrophy in fundus medical images according to claim 2, It is characterized in that When all scale feature maps with the same size specifications are spliced and fused, all scale feature maps with the same size specifications are spliced according to the channel dimension. After splicing, convolution, batch normalization and ReLu activation function processing are performed in sequence to achieve fusion.
4. The method for segmenting choroidal atrophy in fundus medical images according to claim 1, It is characterized in that In the step S1), the method for obtaining a strip feature map by performing strip pooling processing on the feature map output by the encoding network is as follows: performing horizontal strip pooling on the feature map output by the encoding network to obtain a horizontal strip feature map, and performing vertical strip pooling on the feature map output by the encoding network to obtain a vertical strip feature map, and then adding the horizontal strip feature map and the vertical strip feature map and performing ReLu activation function processing to obtain the strip feature map.
5. The method for segmenting choroidal atrophy in fundus medical images according to claim 4, It is characterized in that Before adding the horizontal strip feature map and the vertical strip feature map, the horizontal strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling, and the vertical strip feature map is sequentially subjected to one-dimensional convolution, batch normalization and upsampling.
6. The method for segmenting choroidal atrophy in fundus medical images according to claim 1, It is characterized in that In the step S1), the feature decoding module is used to splice the upsampled feature map and the feature map output by the corresponding feature encoding module, and then sequentially perform convolution, batch normalization and ReLu activation function processing on the spliced feature map.
7. The method for segmenting choroidal atrophy in fundus medical images according to claim 1, It is characterized in that The residual network adopts the ResNet34 network.
Citation Information
Patent Citations
A retinal blood vessel image segmentation method based on a multi-scale feature convolutional neural network
CN108986124A
Context pyramid fusion network and image segmentation method
CN110689083A
Urban road extraction method and device based on remote sensing images
CN110781756A
Retinal fundus vessel segmentation method based on deep multi-scale attention convolutional neural network
CN112102283A
Image processor
WO2020187424A1
