Chromosome metaphase division phase image segmentation method

Abstract

The invention discloses a segmentation method of a chromosome metaphase split phase image, which belongs to the field of image processing, and comprises the following steps: determining a corresponding threshold value through the area ratio of a minimum foreground to a background, and removing the image background by adopting a final image segmentation threshold value; stretching the gray level of each chromosome to enhance the image contrast, performing data enhancement on the images in the training set to increase the number of the images in the training set, regarding the chromosomes and noise in the images as the same class of targets, generating network detection by candidate regions, and giving confidence and an initial rectangular frame; a plurality of sub-network modules identify the category, fine coordinates and contour of a target, a segmentation sub-network segments the identification result to obtain the contour of chromosomes and noise, the background and noise are accurately removed, the banding of the chromosomes is enhanced, the quality of the image is improved, the identification of the chromosomes and the segmentation of crossed, adhered and overlapped chromosomes are automatically completed, and the identification efficiency is improved. And the efficiency of existing karyotype analysis is improved.

Description

Technical Field

[0001] The invention relates to the field of image processing, in particular to a segmentation method of a chromosome metaphase image. Background Art

[0002] Chromosome karyotype analysis has important application value in prenatal diagnosis of genetic diseases. Traditional karyotype analysis is mainly done by doctors manually cutting and identifying chromosome metaphase images. Since normal human chromosomes have 46 and are densely distributed in a small area, it is very easy for chromosomes to cross, stick together, and block each other, and the efficiency of manual segmentation is low.

[0003] Image segmentation methods have emerged in the prior art. However, since chromosome images often have complex and changeable chromosome morphology, mixed noise, insufficient contrast and other problems, the existing image segmentation methods are inaccurate when processing chromosome images, resulting in reduced accuracy of chromosome karyotype analysis. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, one of the objects of the present invention is to provide a method for segmenting chromosome metaphase images, which can independently complete the identification of various types of chromosomes, has the advantages of high accuracy and fast processing speed, and can complete the chromosome identification and segmentation tasks in batches without human intervention, thereby improving the efficiency of existing karyotype analysis.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A method for segmenting a chromosome metaphase image comprises the following steps:

[0007] S1: Collecting chromosome metaphase images;

[0008] S2: Use multiple thresholds to segment the collected image to generate a binary image, calculate the area ratio r of the foreground and background in the binary image, select the threshold corresponding to the smallest r as the final image segmentation threshold, and use the final image segmentation threshold to remove the image background;

[0009] S3: Generate a chromosome set S based on the binary graph generated in step S2 C , S C The gray level of each chromosome is stretched to enhance the image contrast;

[0010] S4: manually annotate the images, divide the annotated data into two sets: training and testing, and perform data augmentation on the images in the training set to increase the number of images in the training set;

[0011] S5: The chromosomes and noise in the image are regarded as the same type of targets, and the candidate region generation network is used for detection, giving the confidence level and the initial rectangular box. Multiple sub-network modules identify the category, fine coordinates and contour of the target, and the segmentation sub-network segments the recognition results to obtain the contours of the chromosomes and noise.

[0012] Furthermore, step S2 specifically includes:

[0013] S21: Set threshold boundary th l , th r , generate multiple thresholds within the boundary threshold and establish a threshold set S th ;

[0014] S22: converting the color image acquired in step S1 into a grayscale image;

[0015] S23: Select threshold th i Segment the grayscale image and generate a binary image;

[0016] S24: Determine whether the binary image is a blank image, if so, select the next image;

[0017] S25: When the binary image is not a blank image, and the number of connected domains in the binary image is greater than 300, the next threshold is selected to re-segment the image; if the number of connected domains in the binary image is less than 300, the larger one is selected from the ratio of the foreground area to the background area and the ratio of the background area to the foreground area, that is:

[0018]

[0019] In the formula, |I foreground |: the number of foreground pixels, i.e. the area of ​​the foreground; |I background |: the number of background pixels, i.e. the area of ​​the background;

[0020] Store r into the set S r After that, select the next threshold to re-segment the image;

[0021] S26: Threshold set S th After all thresholds in are involved in segmentation, select S r The minimum value r min The corresponding threshold is used as the final image segmentation threshold to remove the image background.

[0022] Furthermore, in step S21, a threshold is generated every 0.1 seconds to establish a threshold set S th .

[0023] Furthermore, step S3 specifically includes:

[0024] S31: Generate a chromosome set S based on the binary image generated by background segmentation C ;

[0025] S32: From S C Select a chromosome from , and calculate the grayscale mean μ and standard deviation σ;

[0026] S33: Calculation

[0027] S34: The gray levels in the range are stretched to [0,1];

[0028] S35: Select the next chromosome and repeat steps S31 to S34 until all chromosomes are processed.

[0029] Furthermore, in step S4, data enhancement of the images in the training set includes two methods: image-level processing and chromosome-level processing. The image-level processing targets the entire metaphase image and includes at least one of rotation, upside-down flipping, and left-right flipping. The chromosome-level processing changes the positions and angles of all chromosomes.

[0030] Furthermore, the chromosome-level processing specifically includes the following steps:

[0031] S41: Generate a blank image;

[0032] S42: extracting chromosomes in sequence according to the binary image generated in step S2;

[0033] S43: The current chromosome is rotated and flipped completely randomly;

[0034] S44: placing the processed chromosome into a blank image;

[0035] S45: Repeat steps S41 to S44 until all chromosomes are processed.

[0036] Furthermore, step S4 also includes calculating the mean and variance of the training set images after increasing the number of images in the training set, and normalizing all data.

[0037] Furthermore, step S5 also includes using a non-maximum suppression algorithm to alleviate the problem of repeated detection of some chromosomes and noise. When the confidence is greater than 0.05 and the intersection and union ratio is greater than 0.7, the two detection results are considered to be the same target and the detection results are merged.

[0038] Furthermore, in step S5, the network for generating the candidate region includes a convolutional layer, a maximum pooling layer, and multiple sub-modules, wherein the sub-modules are composed of a batch normalization layer, a ReLU layer, a 1×1 convolutional layer, a batch normalization layer, a ReLU layer, and a 3×3 convolutional layer connected in series, and the multiple sub-modules are densely connected, and the input of each module is connected to the input and output of the previous module.

[0039] Furthermore, in step S5, multiple sub-network modules identify the category, fine coordinates and contour of the target as follows: a classification sub-network and a detection sub-network are used as a group of detection units, the classification sub-network identifies whether the target is a chromosome or noise, and the detection sub-network predicts a fine rectangular frame of the target. The three groups of detection units form a cascade structure, and the chromosomes are processed in sequence according to the intersection-union ratio from small to large.

[0040] Compared with the prior art, the segmentation method of the chromosome metaphase image of the present invention determines the corresponding threshold value through the minimum foreground-background area ratio, and uses the final image segmentation threshold value to remove the image background; the gray level of each chromosome is stretched to enhance the image contrast, data enhancement is performed on the images in the training set to increase the number of images in the training set, the chromosomes and noise in the image are regarded as the same type of targets, and the candidate region generation network detection is performed to give the confidence and the initial rectangular frame; multiple sub-network modules identify the category, fine coordinates and contour of the target, and the segmentation sub-network segments the recognition result to obtain the contour of the chromosome and the noise, accurately remove the background and noise, enhance the chromosome banding, improve the image quality, automatically complete the chromosome recognition, and the segmentation of the crossed, adhered and overlapped chromosomes, thereby improving the efficiency of the existing karyotype analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the method for segmenting a chromosome metaphase image of the present invention;

[0042] Figure 2 Remove the flow chart for the background;

[0043] Figure 3 This is the original image of the chromosome metaphase;

[0044] Figure 4 This is the image after background removal;

[0045] Figure 5 This is the image after contrast enhancement;

[0046] Figure 6 This is the diagram after chromosome segmentation. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may be another intermediate component through which it is fixed. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may be another intermediate component at the same time. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be another intermediate component at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0050] Figure 1 The flowchart of the method for segmenting a chromosome metaphase image of the present invention is as follows. The method for segmenting a chromosome metaphase image of the present invention comprises the following steps:

[0051] S1: Collecting chromosome metaphase images;

[0052] S2: Use multiple thresholds to segment the collected image to generate a binary image, calculate the area ratio r of the foreground and background in the binary image, select the threshold corresponding to the smallest r as the final image segmentation threshold, and use the final image segmentation threshold to remove the image background;

[0053] S3: Generate a chromosome set S based on the binary graph generated in step S2 C , S C The gray level of each chromosome is stretched to enhance the image contrast;

[0054] S4: manually annotate the images, divide the annotated data into two sets: training and testing, and perform data augmentation on the images in the training set to increase the number of images in the training set;

[0055] S5: The chromosomes and noise in the image are regarded as the same type of targets, and the candidate region generation network is used for detection, giving the confidence level and the initial rectangular box. Multiple sub-network modules identify the category, fine coordinates and contour of the target, and the segmentation sub-network segments the recognition results to obtain the contours of the chromosomes and noise.

[0056] Specifically, in step S1, the metaphase image of chromosomes is collected as a three-channel image (color image). The collected metaphase image of chromosomes has a gray background with uneven brightness, which greatly interferes with subsequent chromosome segmentation and identification, so the background needs to be removed.

[0057] Specifically, step S2 includes:

[0058] S21: Set threshold boundary th l , th r , generate multiple thresholds within the boundary threshold and establish a threshold set S th ;

[0059] S22: converting the collected color image into a grayscale image;

[0060] S23: Select threshold th i Segment the grayscale image and generate a binary image;

[0061] S24: Determine whether the binary image is a blank image, if so, select the next image;

[0062] S25: When the binary image is not a blank image, and the number of connected domains in the binary image is greater than 300, the next threshold is selected to re-segment the image; if the number of connected domains in the binary image is less than 300, the larger one is selected from the ratio of the foreground area to the background area and the ratio of the background area to the foreground area, that is:

[0063]

[0064] In the formula, |I foreground |: the number of foreground pixels, i.e. the area of ​​the foreground; |I background |: The number of background pixels, that is, the area of ​​the background; the meaning of this formula is to select the larger one from the ratio of the foreground area to the background area and the ratio of the background area to the foreground area, and store it in the set S r In the above example, r is stored in the set S. r After that, select the next threshold to re-segment the image;

[0065] S26: Threshold set S th After all thresholds in are involved in segmentation, select S r The minimum value r min The corresponding threshold is used as the final image segmentation threshold to remove the image background.

[0066] Specifically, in step S21, a threshold is generated every 0.1, and a threshold set S is established. th In step S22, the size is resampled to 1200×1600.

[0067] Specifically, step S3 includes:

[0068] S31: Generate a chromosome set S based on the binary image generated by background segmentation C ;

[0069] S32: From S C Select a chromosome from , and calculate the grayscale mean μ and standard deviation σ;

[0070] S33: Calculation

[0071] S34: The gray levels in the range are stretched to [0,1];

[0072] S35: Select the next chromosome and repeat steps S31 to S34 until all chromosomes are processed.

[0073] Step S3 is a contrast enhancement algorithm based on grayscale stretching, which is used to improve the clarity of chromosome banding to facilitate identification of chromosome categories.

[0074] Step S4 is specifically as follows:

[0075] After the metaphase image is background removed and contrast enhanced, it is handed over to the physician to manually annotate the outline and category of each chromosome. The annotated objects include the outline and category of chromosomes and noise. When annotating, if two chromosomes overlap, the overlapping area is shared by the two chromosomes. The annotated data is divided into two sets: training and testing. The former is used to train the instance segmentation algorithm, and the latter is used to test the algorithm performance and debug the algorithm's hyperparameters.

[0076] Due to the low efficiency of manual labeling, there are not enough images in the training set. To this end, this patent uses data enhancement methods to increase the number of images in the training set, which is divided into two methods: image-level processing and chromosome-level processing. The image level is to process the entire metaphase image as the target. The specific processing methods include rotation, upside down flipping, left-right flipping, and a combination of the three. The chromosome level is to make a large change in the position and angle of all chromosomes to form a new image. The specific steps are:

[0077] S41: Generate a blank image;

[0078] S42: extracting chromosomes in sequence according to the binary image generated in step S2;

[0079] S43: The current chromosome is rotated and flipped completely randomly;

[0080] S44: placing the processed chromosome into a blank image;

[0081] S45: Repeat steps S41 to S44 until all chromosomes are processed.

[0082] Specifically, in step S41, the size of the blank image is 1200×1600.

[0083] After obtaining enough training data through the above method, the mean and variance of the training set images are calculated, and all data are normalized.

[0084] Step S5 is specifically as follows:

[0085] The segmentation method of metaphase images is based on the deep learning algorithm Cascade Masked-RCNN to identify the difference between chromosomes and noise and separate chromosomes that are adhered, crossed, and overlapped. The processing of this method is divided into two stages. In the first stage, chromosomes and noise are regarded as the same type of targets and detected by the candidate region generation network (RPN), giving confidence and initial rectangular boxes; in the second stage, multiple sub-network modules perform more detailed processing to identify the category, fine coordinates, and contours of the first-stage results.

[0086] In the first stage, the candidate region generation network is implemented based on the classification algorithm DenseNet-121. The first two layers are a 7×7 convolution layer and a 3×3 maximum pooling layer, followed by a submodule called DenseBlock. Each module consists of a batch normalization layer, a ReLU layer, a 1×1 convolution layer, a batch normalization layer, a ReLU layer, and a 3×3 convolution layer in series. The modules are densely connected, and the input of each module is connected to the input and output of the previous module. This connection method can effectively alleviate the problems of gradient dissipation and network degradation in deep networks. DenseNet-121 contains a total of 60 DenseBlocks, and there is a Transition Layer between the 6th-7th, 18th-19th, and 42nd-43rd modules to compress the output of each module. After the output of the last DenseBlock is globally averaged and pooled, it is handed over to two fully connected layers, which are used to predict whether the target is a chromosome or noise, and give an initial rectangular box.

[0087] In the second stage, a classification subnetwork and a detection subnetwork are used as a group of detection units. The former is used to identify whether the target is a chromosome or noise, and the latter predicts its fine rectangular frame. The three groups of units form a cascade structure, and the chromosomes are processed in order from small to large in terms of intersection-over-union ratio. The reason is that the number ratio between positive samples and negative samples during training is inconsistent with that during testing, which leads to mismatched thresholds of the intersection-over-union ratio between the two, causing interference to the detection. Therefore, this method of gradually increasing the intersection-over-union ratio threshold is adopted to cover all possible intersection-over-union ratio thresholds as much as possible to improve the detection performance. The output of the last group of detection units is the recognition and detection results of chromosomes and noise by Cascade Masked-RCNN, and then the segmentation subnetwork segments the detection results to obtain the contours of chromosomes and noise.

[0088] During the segmentation process, the non-maximum suppression algorithm is used to alleviate the problem of repeated detection of some chromosomes and noise, further improving the accuracy. When the confidence is greater than 0.05 and the intersection-over-union ratio is greater than 0.7, the two detection results are considered to be the same target and the detection results are merged.

[0089] The Cascade Masked-RCNN training times are 30, the initial learning rate is 0.0001, the learning rate is reduced by 10 times every 10 training times, and the optimization method is the stochastic gradient descent algorithm.

[0090] The present invention discloses a method for segmenting chromosome metaphase images, which can automatically identify chromosomes and segment crossed, adhered, and overlapping chromosomes, without manual intervention in the entire process; the image processing speed is fast, and a single image can be processed in only 2 seconds, which is significantly better than the manual method and can effectively improve the efficiency of karyotype analysis; the background and noise can be accurately removed, the chromosome banding can be enhanced, and the image quality can be improved, so that physicians can perform karyotype analysis more conveniently.

[0091] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A method for segmenting a chromosome metaphase image, characterized in that: The following steps are involved: S1: Collecting chromosome metaphase images; S2: Use multiple thresholds to segment the collected image to generate a binary image, calculate the area ratio r of the foreground and background in the binary image, select the threshold corresponding to the smallest r as the final image segmentation threshold, and use the final image segmentation threshold to remove the image background; S3: Generate a chromosome set S based on the binary graph generated in step S2 C , S C The gray level of each chromosome is stretched to enhance the image contrast; S4: manually annotate the images, divide the annotated data into two sets: training and testing, and perform data augmentation on the images in the training set to increase the number of images in the training set; S5: The chromosomes and noise in the image are regarded as the same type of targets, and the candidate region generation network is used for detection, giving the confidence level and the initial rectangular box. Multiple sub-network modules identify the category, fine coordinates and contour of the target, and the segmentation sub-network segments the recognition results to obtain the contours of the chromosomes and noise.

2. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: Step S2 specifically includes: S21: Set threshold boundary th l , th r , generate multiple thresholds within the boundary threshold and establish a threshold set S th ; S22: converting the color image acquired in step S1 into a grayscale image; S23: Select threshold th i Segment the grayscale image and generate a binary image; S24: Determine whether the binary image is a blank image, if so, select the next image; S25: When the binary image is not a blank image, and the number of connected domains in the binary image is greater than 300, the next threshold is selected to re-segment the image; if the number of connected domains in the binary image is less than 300, the larger one is selected from the ratio of the foreground area to the background area and the ratio of the background area to the foreground area, that is: In the formula, |I foreground |: the number of foreground pixels, i.e. the area of ​​the foreground; |I background |: the number of background pixels, i.e. the area of ​​the background; Store r into the set S r After that, select the next threshold to re-segment the image; S26: Threshold set S th After all thresholds in are involved in segmentation, select S r The minimum value r min The corresponding threshold is used as the final image segmentation threshold to remove the image background.

3. The method for segmenting a chromosome metaphase image according to claim 2, characterized in that: In step S21, a threshold is generated every 0.1 seconds to establish a threshold set S th .

4. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: Step S3 specifically includes: S31: Generate a chromosome set S based on the binary image generated by background segmentation C ; S32: From S C Select a chromosome from , and calculate the grayscale mean μ and standard deviation σ; S33: Calculation S34: The gray levels in the range are stretched to [0,1]; S35: Select the next chromosome and repeat steps S31 to S34 until all chromosomes are processed.

5. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: In step S4, data enhancement of the images in the training set includes two methods: image-level processing and chromosome-level processing. The image-level processing targets the entire metaphase image and includes at least one of rotation, upside-down flipping, and left-right flipping. The chromosome-level processing changes the positions and angles of all chromosomes.

6. The method for segmenting a chromosome metaphase image according to claim 5, characterized in that: The chromosome level processing specifically comprises the following steps: S41: Generate a blank image; S42: extracting chromosomes in sequence according to the binary image generated in step S2; S43: The current chromosome is rotated and flipped completely randomly; S44: placing the processed chromosome into a blank image; S45: Repeat steps S41 to S44 until all chromosomes are processed.

7. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: Step S4 also includes calculating the mean and variance of the training set images after increasing the number of images in the training set, and normalizing all data.

8. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: Step S5 also includes using a non-maximum suppression algorithm to alleviate the problem of repeated detection of some chromosomes and noise. When the confidence is greater than 0.05 and the intersection-and-union ratio is greater than 0.7, the two detection results are considered to be the same target and the detection results are merged.

9. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: In step S5, the network for generating the candidate region includes a convolution layer, a maximum pooling layer, and multiple sub-modules. The sub-modules are composed of a batch normalization layer, a ReLU layer, a 1×1 convolution layer, a batch normalization layer, a ReLU layer, and a 3×3 convolution layer connected in series. The multiple sub-modules are densely connected, and the input of each module is connected to the input and output of the previous module.

10. The method for segmenting a chromosome metaphase image according to claim 1, characterized in that: In step S5, multiple sub-network modules identify the category, fine coordinates and outline of the target. Specifically, a classification sub-network and a detection sub-network are used as a group of detection units. The classification sub-network identifies whether the target is a chromosome or noise. The detection sub-network predicts the fine rectangular frame of the target. The three groups of detection units form a cascade structure, and the chromosomes are processed in order from small to large intersection-to-union ratio.

Images