End-to-end chromosome instance segmentation based on edge supervision network

Abstract

The present application aims at the over-segmentation and under-segmentation problems caused by cross and overlapping segmentation of end-to-end chromosome instance segmentation, and proposes a chromosome instance segmentation framework ESNet. Taking Mask R-CNN as the baseline, a positioning enhancement strategy is constructed to solve the problem of missed detection of cross and overlapping chromosomes; a spatial attention module is constructed to provide a larger receptive field for the semantic segmentation branch, thereby bringing more context information and long-range dependency information; an edge supervision segmentation branch is constructed to integrate more chromosome positioning and edge information for chromosome semantic segmentation, so as to comprehensively improve the segmentation performance of the network for cross and overlapping chromosomes. Taking the mean Average Precision (AP) of the MSCOCO dataset as the evaluation index, ESNet obtains a segmentation accuracy of 80.7AP, 98.5AP 50 , 94.1AP 75 , which is 4.4%, 2% and 3.6% higher than the baseline network respectively, and can better complete the end-to-end chromosome instance segmentation task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image segmentation, and solves the problems of chromosome overlapping and intersection segmentation in a chromosome metaphase image, and specifically proposes an edge supervision network to solve the problems. BACKGROUND

[0002] It is well known that there are 46 chromosomes in human healthy cells, of which 44 are autosomes and 2 are sex chromosomes (XY for males and XX for females), carrying genetic information. Chromosomal morphological abnormalities and numerical abnormalities can cause various diseases, such as severe birth defects in newborns. Karyotype analysis takes metaphase chromosomes as the research object, and uses chromosome banding technology to color, segment, identify and number chromosomes, and has become one of the gold standards for diagnosing chromosomal diseases in newborns. It can be seen that karyotype analysis first needs to separate each independent chromosome, which is an important basis for subsequent identification and numbering operations. However, the obtained chromosome images usually have complex situations such as individual intersection, overlapping and impurity interference, therefore, accurate chromosome image segmentation has become an indispensable step in karyotype analysis. Unlike natural image segmentation, chromosome image segmentation has the following two challenges: 1) Chromosome non-rigidity. Chromosomes are a non-rigid material that exists in a curved state in human cell nuclei, causing the complexity of chromosome multi-pose, which brings difficulties to chromosome segmentation tasks. 2) Multi-cluster. Since chromosomes are distributed in a free state in the cell nucleus, this leads to the clustering of two or more chromosomes, causing chromosome contact and overlapping.

[0003] From the development history, chromosome image segmentation is mainly divided into traditional chromosome image segmentation based on threshold value, geometric analysis, etc. and chromosome image segmentation based on deep learning. Grisan et al. proposed a method of combining spatial transformation threshold value and geometric analysis with decision tree to segment and separate Q-band pro-metaphase chromosome images. On a dataset of 6683 high and low fluorescence region intersection chromosome images, 94% of the chromosomes can be correctly segmented. Yilmaz et al. proposed a method of using threshold segmentation and watershed to segment individual chromosomes and chromosome clusters, calculating the change of boundary curvature to obtain the cut point, and finally calculating the best geodesic line between the cut points to segment the contact and overlapping chromosomes. The traditional chromosome segmentation method based on threshold segmentation and geometric analysis relies on the setting of artificial parameters, and is greatly dependent on the quality of the chromosome image, sensitive to the chromosome dataset, and has poor robustness.

[0004] With the development of deep learning, deep learning technology has made good progress in image segmentation. More and more scholars have introduced this technology into chromosome segmentation, which can be divided into two categories: semantic segmentation network-based and instance segmentation network-based chromosome segmentation. Compared with the traditional image segmentation algorithm, the deep learning method has stronger robustness and learnability, and can achieve better results in chromosome segmentation.

[0005] Hu introduced U-Net into the field of chromosome image semantic segmentation. Considering the small size of the input image and the need to reduce computation and model storage, the author cropped the standard U-Net to reduce the size of the model, and completed the segmentation of the artificially synthesized two-by-two overlapping chromosome FISH image dataset, finally obtaining an average intersection of union (IoU) of 94.7% and an accuracy of 99.22%. Saleh et al. improved the network structure used by Hu et al. to further improve the performance of U-Net in chromosome segmentation tasks, and applied test time augmentation algorithm to chromosome segmentation task, which had certain effect on improving the experimental results compared with the former. For semantic segmentation tasks, the overlapping and crossing parts of chromosomes need to be labeled separately, otherwise individual chromosomes cannot be segmented.

[0006] Compared with semantic segmentation, instance segmentation task integrates classification, detection and segmentation into one, which is more convenient and competent in handling chromosome segmentation. Li Kang et al. used self-built dataset to apply Mask R-CNN to chromosome instance segmentation, adjusted different proportions of manual annotation and synthesized chromosome pictures, and finally obtained an average segmentation accuracy of 59.998. Lin Chengchuang et al. based on multi-task architecture, fused image classification, target detection, instance segmentation and chromosome counting into four branches based on instance segmentation network PANet to establish AS-PANet model, and completed the chromosome segmentation task on the self-built chromosome cluster image dataset after threshold segmentation, finally obtained an average segmentation accuracy of 90.63% and a segmentation accuracy of 85%, which improved 2.85% and 3.75% respectively compared with the baseline model Mask R-CNN.

[0007] In summary, the traditional method based on threshold and geometric analysis is highly dependent on the quality of chromosome image, and the parameters need to be manually adjusted, which has poor robustness. The chromosome segmentation method based on semantic segmentation has limitations in data labeling. For the segmentation of the whole chromosome image, it cannot achieve end-to-end effect. The chromosome segmentation method based on instance segmentation cannot achieve good processing effect in the segmentation of overlapping and crossing chromosomes, and the segmentation accuracy needs to be improved. SUMMARY

[0008] The application improves the segmentation effect of the end-to-end chromosome instance segmentation network on the cross and overlapping chromosomes, and proposes an edge supervision segmentation method, so that the network integrates more chromosome edges and positioning information to improve the segmentation accuracy of the network on the cross and overlapping chromosomes.

[0009] The technical idea of the application is that the application takes Mask R-CNN as the basis, introduces a positioning enhancement strategy to improve the detection of the chromosome target, introduces a spatial attention module and edge supervision segmentation to provide more context information and chromosome positioning and edge information for the network, fully utilizes the context information and edge features of the chromosome, and improves the segmentation effect of the network on the overlapping chromosomes.

[0010] The implementation scheme includes the following:

[0011] (1) Preprocess the chromosome image:

[0012] (1a) Scale the chromosome image to 800x1088x3;

[0013] (1b) Generate corresponding positioning, category and segmentation label vectors, as shown in Figure 1 .

[0014] (2) Construct a chromosome positioning enhancement strategy:

[0015] (2a) Take Focal loss as the loss function of chromosome classification;

[0016] (2b) Take Soft-NMS as the post-processing method of the chromosome positioning frame.

[0017] (3) Construct a spatial attention module (SAM):

[0018] (3a) The input feature map is subjected to convolution operation to obtain Key, Value and Query features;

[0019] (3b) Perform matrix multiplication operation on the Key and Query features, and then perform normalization operation through Softmax to obtain Weigt feature;

[0020] (3c) Perform matrix multiplication operation on the Weight and Value features to obtain new features;

[0021] (3d) Perform element corresponding matrix addition operation on the features obtained in step 3c and the input features to obtain the final output features.

[0022] (4) Construct a feature fusion module (FFM):

[0023] (4a) the chromosome edge feature is passed through two layers of Res modules;

[0024] (4b) the chromosome segmentation feature is subjected to a feature splicing operation with the chromosome edge feature to obtain a final chromosome segmentation feature.

[0025] (5) an edge supervised network (ESNet) network is constructed:

[0026] (5a) the ESNet network is constructed based on Mask R-CNN by constructing a positioning enhancement strategy, a spatial attention module, a feature fusion module and a chromosome edge segmentation branch;

[0027] (6) the ESNet network is trained:

[0028] The SGD optimizer is used to optimize the objective loss function, the batch size is set to 1, the learning rate is 0.0025, the maximum training round is set to 40 rounds, and the learning rate is decayed by 1 / 10 at the 27th and 34th rounds, respectively.

[0029] Compared with the prior art, the present application has the following advantages:

[0030] 1. The present application realizes higher chromosome segmentation accuracy;

[0031] 2. The positioning enhancement strategy of the present application can better cope with the missed detection problem of crossing and overlapping chromosomes;

[0032] 3. The edge supervised segmentation branch proposed in the present application can better cope with the segmentation of crossing and overlapping chromosomes; BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a chromosome metaphase picture and a true label

[0034] Figure 2 is an ESNet network structure.

[0035] Figure 3 is a spatial attention module structure diagram.

[0036] Figure 4 is a feature fusion module structure diagram. DETAILED EMBODIMENT

[0037] The present application will be further described in detail below in combination with the drawings and specific embodiments

[0038] Step 1: Preprocessing of chromosome images:

[0039] (1a) Rescale the chromosome image to 800x1088x3;

[0040] (1b) Generate the corresponding localization, class, segmentation label vector.

[0041] Step 2, build chromosome localization enhancement strategy:

[0042] (2a) Take Focal loss as the loss function of chromosome classification to improve the network's learning of difficult samples;

[0043] (2b) Take Soft-NMS as the post-processing method of chromosome localization box to improve the network's detection of crossing and overlapping chromosomes.

[0044] Step 3, build a spatial attention module, as shown in Figure 2

[0045] (3a) The input feature map is respectively passed through 3 convolution kernels with a size of 1x1, a stride of 1, and padding of 0 to obtain Key, Value, and Query features;

[0046] (3b) Perform matrix multiplication on Key and Query features, and then normalize them by Softmax to obtain Weight features;

[0047] (3c) Perform matrix multiplication on Weight and Value features to obtain new features;

[0048] (3d) Perform element-wise matrix addition on the features obtained in (3c) and the input features, and then perform maximum pooling to reduce the resolution of the output feature map to obtain the final output feature. The spatial attention module parameter table is shown in Table 1.

[0049] Step 4, build a feature fusion module, as shown in Figure 3

[0050] (4a) Pass the chromosome edge feature through two layers of Res modules to further extract and enhance edge information, where the Res module consists of two layers of convolution operations with a kernel size of 3x3, a stride of 1, and padding of 1, and skip connection;

[0051] (4b) Perform feature splicing and fusion operation on the chromosome segmentation feature and the chromosome edge feature to obtain the final chromosome segmentation feature. The feature fusion module parameter table is shown in Table 2.

[0052] Step 5, build a chromosome edge supervision branch:

[0053] ​​(5a) On the basis of the base network, the edge segmentation branch is constructed to output the chromosome edge segmentation mask, and the obtained chromosome edge information is fused into the chromosome segmentation feature through the feature fusion module.

[0054] Step 6, the ESNet network is constructed as shown in Figure 1

[0055] (6a) Based on Mask R-CNN, the ESNet network is built by constructing a positioning enhancement strategy, a spatial attention module, a feature fusion module and a chromosome edge segmentation branch, and the segmentation branch parameter table is as shown in Table 3;

[0056] (6b) The positioning performance of the network on the chromosome is improved by constructing the positioning enhancement strategy, so as to improve the downstream semantic segmentation task performance;

[0057] (6c) By constructing the spatial attention module, the network can obtain a larger receptive field and obtain more upper and lower information and long-range dependency information;

[0058] (6d) By constructing the chromosome edge supervision branch, more positioning and edge information is integrated into the chromosome segmentation, so that the network can better cope with the segmentation of the cross and overlapping chromosomes.

[0059] Step 7, the ESNet network is trained:

[0060] The SGD optimizer is used to optimize the target loss function, the batch size is set to 1, the learning rate is 0.0025, the maximum training round is set to 40 rounds, and the learning rate is decayed by 1 / 10 at the 27th and 34th rounds, so that the network can be trained more fully.

[0061] A series of experiments are carried out on the chromosome instance segmentation dataset, 80% of which is selected as the training set and the rest is selected as the test set to evaluate the generalization ability of the network. The mean average precision (AP) of the MSCOCO dataset is used as an evaluation index to evaluate the instance segmentation performance of the ESNet.

[0062] The experimental environment of the present application is configured as follows: the computer processor is Intel(R) Xeon(R) W-2175 CPU@2.50GHz, 64GB running memory, NVIDIA GeForce RTX 2080Ti GPU, and PyTorch framework.

[0063] ​In conclusion, the application proposes a chromosome instance segmentation model ESNet for end-to-end to realize high-precision segmentation of chromosomes. The network improves the instance segmentation of chromosomes, especially the instance segmentation of chromosomes under the condition of intersection and overlap, by constructing a positioning enhancement strategy, a spatial attention module and a chromosome edge supervision segmentation branch. Finally, ESNet obtains a segmentation accuracy of 80.7AP, 98.5AP 50 , 94.1AP 75 , which is 4.4%, 2% and 3.6% higher than the baseline network respectively, and can better complete the end-to-end chromosome instance segmentation task.

[0064] Table 1 Spatial attention module parameter table

[0065]

[0066] Table 2 Feature fusion module parameter table

[0067]

[0068] Table 3 ESNet chromosome segmentation branch parameter table

[0069]

Claims

1. A chromosome instance segmentation method based on edge supervision network, characterized in that, Comprise: (1) preprocessing of chromosome image: (1a) scaling the chromosome image to 800x1088x3; (1b) generating corresponding positioning, class and segmentation label vector; (2) constructing chromosome positioning enhancement strategy: (2a) taking Focal loss as the loss function of chromosome classification; (2b) taking Soft-NMS as the post-processing method of chromosome positioning frame; (3) constructing spatial attention module: (3a) the input feature map is subjected to convolution operation to obtain Key, Value and Query features; (3b) the Key and Query features are subjected to matrix multiplication operation, and then normalized by Softmax to obtain Weight matrix; (3c) the Weight and Value features are subjected to matrix multiplication operation to obtain new features; (3d) the features obtained in (3c) are subjected to element corresponding matrix addition operation with the input features, and then subjected to one maximum pooling to reduce the resolution of the output feature map, to obtain the final output feature; (4) constructing feature fusion module: (4a) the chromosome edge features are subjected to two layers of Res module; (4b) the chromosome segmentation features and the chromosome edge features subjected to two layers of Res module are subjected to feature splicing operation to obtain the final chromosome segmentation features; (5) constructing chromosome edge supervision branch: (5a) constructing edge segmentation branch based on the basic network to output chromosome edge segmentation mask, and at the same time, through the feature fusion module, the obtained chromosome edge information is fused into the chromosome segmentation features; (6) constructing ESNet network: (6a) based on Mask R-CNN, through chromosome positioning enhancement strategy, spatial attention module, feature fusion module and chromosome edge supervision branch, ESNet network is built; (7) training the ESNet network.

2. The method of claim 1, wherein, The convolution operation in step (3a) is a convolution with 3 convolution kernels of size 1x1, stride 1 and padding 0.

3. The method of claim 1, wherein, The Res module in step (4a) is composed of two layers of convolution operation with convolution kernel size of 3x3, stride of 1 and padding of 1, and skip connection.

4. The method of claim 1, wherein, In step (7), the network training uniformly adopts SGD optimizer to optimize the target loss function, the batch size is set to 1, the learning rate is 0.0025, the maximum training round is set to 40 rounds, and the learning rate is decayed to 1 / 10 of the original at the 27th and 34th rounds.

Images