Chromosome instance segmentation method based on skeleton guidance

Abstract

The invention relates to a chromosome instance segmentation method based on skeleton guidance. The method comprises the following steps: acquiring an input image; the input image is a chromosome microscopic image; extracting a feature map from the input image through a skeleton network; respectively inputting the feature map into a segmentation head, a skeleton head and a regression head of a skeleton network, and respectively and correspondingly predicting a segmentation mask, a skeleton and a frame transformation parameter; and obtaining a rotating bounding box and a mask thereof according to the outputs of the segmentation head, the skeleton head and the regression head. The method has the beneficial effects that an object is represented by using a high-resolution pixel-level mask and a rotating bounding box guided by a skeleton, and the instance segmentation effect of overlapped chromosomes is superior to that of an existing frame-based method; moreover, the score of the frame proposal is obtained through multiplication of the segmentation graph and the skeleton graph, the dense distribution problem can be solved, and the bad frame proposal near the instance boundary is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of instance segmentation, and more specifically, to a chromosome instance segmentation method based on skeleton guidance. Background Art

[0002] There are many causes of birth defects and congenital diseases, but abnormalities in the number or structure of chromosomes (i.e., chromosomal diseases) are one of the main causes. These abnormalities may lead to spontaneous abortion, birth defects (such as congenital disabilities), and genetic diseases (such as Down syndrome). Normally, each person has 23 pairs of chromosomes, including 22 pairs of autosomes and 1 pair of sex chromosomes (XY for males and XX for females).

[0003] When the number of chromosomes is abnormal, it can cause a series of health problems. For example, if there is an extra chromosome 21, it will cause Down syndrome, and if a woman is missing an X chromosome, she will suffer from Turner syndrome. Traditional karyotyping methods require multiple steps to complete, including cell culture, preparation of chromosome samples, and observation under a microscope. The whole process is very time-consuming and usually takes several weeks to get results.

[0004] However, observing and arranging chromosomes under an optical microscope is undoubtedly the most challenging step. Geneticists use optical microscopes to obtain metaphase images (such as Figure 1 Finally, the identified chromosomes are arranged according to the karyotype diagram (as shown in Figure 2 As shown in Figure 2). This method not only relies on the experience and skills of technicians, but also leads to inconsistent results due to differences in operation between different technicians. Moreover, long-term reliance on technicians for microscope observation can easily lead to visual fatigue and affect the accuracy of detection. Therefore, it is urgent to propose a reliable and rapid chromosome karyotype analysis method.

[0005] Chromosome instance segmentation is a very challenging task due to the non-rigid and complex special structure of chromosomes, such as irregular shapes, small but diverse sizes, and dense distribution in arbitrary directions (even crossover and overlap).

[0006] Image processing technology is increasingly used in the medical field. Therefore, many instance segmentation methods have emerged in recent years, which can be roughly summarized into three categories: key point detection, grid regression and box proposal, such as Figure 4 As shown in (1)-(3). Figure 4(1) Keypoint methods first detect instance-independent keypoints and then group them into corresponding instances in the post-processing stage. For example, PolarMask++ uses centroids and ray lengths (sampled contour points) in polar coordinates to represent instances; Figure 4 (2) Grid regression methods, such as SOLO and SOLOv2, predict instance masks directly based on the image grid without any post-processing; Figure 4 (3) in [3]: Box proposal methods, usually follow a two-stage framework such as Mask-RCNN, which segments instances based on horizontal bounding box proposals.

[0007] The single-stage method usually uses Figure 4 (1) Key points or Figure 4 The grid cells in (2) are used to represent instances, which is too coarse for instance recognition. Two-stage methods such as Figure 4 (3) Mask-RCNN in contains many redundant anchors and may not be able to recognize objects within cluster regions with horizontal boxes.

[0008] Although the above three methods have achieved remarkable results in their respective tasks, they may fail to handle the unique characteristics of objects in chromosome images due to their inappropriate instance representations (i.e., horizontal boxes, grids, and keypoints). Figure 4 As shown in , these representations cannot accurately identify instances within cluster regions. Single-stage methods usually generate poor masks and cannot handle chromosome overlap due to the use of coarse information such as centroids and grids to separate objects. In contrast, two-stage methods can achieve more refined pixel-level masks and have the potential to isolate overlapping instances by bounding boxes. However, they also suffer from limitations such as complex model pipelines and dense anchor strategies. In particular, dense anchor strategies lead to a large number of redundant anchors and involve additional hyperparameters such as anchor basis size and aspect ratio, which must be carefully tuned for specific tasks to guarantee good performance. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the prior art and propose a chromosome instance segmentation method based on skeleton guidance.

[0010] In a first aspect, a skeleton-guided chromosome instance segmentation method is provided, comprising:

[0011] Step 1, obtaining an input image; the input image is a chromosome microscopic image;

[0012] Step 2: Extract feature maps from the input image through the skeleton network;

[0013] Step 3: Input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters;

[0014] Step 4: Obtain a rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head, and the regression head.

[0015] Preferably, in step 2, a feature map is extracted from the input image through multiple A2B blocks.

[0016] Preferably, step 3 comprises:

[0017] Step 3.1: The segmentation head predicts the segmentation mask of each chromosome instance from the feature map and obtains the segmentation map. ; The segmentation mask is used to mark the boundaries of each chromosome;

[0018] Step 3.2: The skeleton head predicts the centerline and skeleton of the chromosome instance from the feature map to obtain the skeleton map ;

[0019] Step 3.3: The regression head predicts the box transformation parameters related to the chromosome instance from the feature map and obtains the regression increment of the anchor box. .

[0020] Preferably, step 4 comprises:

[0021] Step 4.1: Perform Argmax calculation to obtain the allocation mask;

[0022] Step 4.2: Segmentation map and skeleton diagram Perform pixel-by-pixel multiplication to obtain the box score;

[0023] Step 4.3: Set Anchor Point ;

[0024] Step 4.4: Based on the box score and anchor point The regression increment of the anchor box is obtained by the box decoder; and rotation NMS is performed to reduce the overlapping anchor boxes;

[0025] Step 4.5: Determine mask pixels according to the box scores, and reallocate the mask pixels to each anchor box.

[0026] In a second aspect, a chromosome instance segmentation system based on skeleton guidance is provided, which is used to execute any method described in the first aspect, including:

[0027] A first acquisition module is used to acquire an input image; the input image is a chromosome microscopic image;

[0028] An extraction module for extracting feature maps from input images through a skeleton network;

[0029] A prediction module, used to input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters;

[0030] The second acquisition module is used to acquire the rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head and the regression head.

[0031] According to a third aspect, a computer storage medium is provided, wherein a computer program is stored in the computer storage medium; when the computer program is executed on a computer, the computer executes any method described in the first aspect.

[0032] In a fourth aspect, an electronic device is provided, including:

[0033] Memory, used to store computer programs;

[0034] A processor is used to execute the computer program to implement any method as described in the first aspect.

[0035] The beneficial effects of the present invention are:

[0036] 1. The present invention realizes accurate instance segmentation of chromosome microscopic images through skeleton network and multi-task learning mechanism, which can effectively handle complex morphologies such as chromosome rotation, bending, and cross-overlapping, significantly improves segmentation accuracy and robustness, and provides efficient and reliable technical support for biomedical research.

[0037] 2. The present invention uses high-resolution pixel-level masks and skeleton-guided rotated bounding boxes to represent objects, and has better instance segmentation effect on overlapping chromosomes than existing box-based methods. In addition, the present invention obtains the score of the box proposal by multiplying the segmentation map and the skeleton map, which can solve the dense distribution problem and reduce the poor box proposals near the instance boundary.

[0038] 3. The present invention designs a more efficient skeleton network that can better capture and utilize the difference features between different types of chromosomes, and the differences between different types of chromosomes. In addition, the skeleton network of the present invention can better handle the bending and deformation of chromosomes, ensuring the stability and reliability of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a chromosome micrograph;

[0040] Figure 2 is the chromosome karyotype diagram;

[0041] Figure 3 It is a local zoom diagram of the chromosome microscopic image;

[0042] Figure 4 Schematic diagram of various instance segmentation strategies;

[0043] Figure 5 A framework diagram of a skeleton-guided chromosome instance segmentation method provided by an embodiment of the present invention;

[0044] Figure 6 A2B block structure diagram provided for an embodiment of the present invention;

[0045] Figure 7 Box and mask proposal diagram for the test phase provided by an embodiment of the present invention;

[0046] Figure 8 The segmentation diagram and skeleton diagram of the chromosome segmentation case provided by the embodiment of the present invention;

[0047] Fig. 9 A comparison diagram of the results of chromosome instance segmentation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0049] Embodiment 1:

[0050] To solve the problems of the prior art, Example 1 of the present application provides a skeleton-guided chromosome instance segmentation method, which directly uses pixel-by-pixel masks and rotation boxes to separate instances in one stage, and the rotation box is predicted from skeleton-guided sparse anchor points.

[0051] Specifically, Figure 5 As shown, the method includes:

[0052] Step 1: Obtain an input image; the input image is a chromosome microscopic image.

[0053] For clarity, Figure 3 As shown, this application only displays the local zoomed area of ​​the chromosome micrograph, and the numbers in the boxes represent the class labels.

[0054] Step 2: Extract feature maps from the input image through the skeleton network.

[0055] Step 3: Input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters.

[0056] Step 3 includes:

[0057] Step 3.1: The segmentation head predicts the segmentation mask of each chromosome instance from the feature map and obtains the segmentation map. ; The segmentation mask is used to mark the boundaries of each chromosome.

[0058] In step 3.1, the features extracted from the skeleton network are input into the segmentation head, which is a subnetwork designed in this application specifically for predicting target masks. The main task of the segmentation head is to predict the segmentation mask of each chromosome instance from the high-resolution feature map extracted from the skeleton network. These masks are used to accurately mark the boundaries of each chromosome in the image. The segmentation head focuses on generating classification scores for each pixel, indicating the probability that each pixel belongs to the chromosome. This step is the key to achieving fine chromosome instance segmentation. Segmentation masks usually refer to masks used for semantic segmentation, which identify the pixels in the image that belong to each category. In semantic segmentation, each pixel is marked as belonging to a specific category rather than being marked as an independent instance.

[0059] Specifically, the processing of the split header includes:

[0060] Step 3.1.1, feature map processing: Input the feature map generated by the skeleton network, and after multiple layers of convolution operations (including ASPP module), extract multi-scale features to capture chromosome instances of different sizes.

[0061] Step 3.1.2, classification output: The last layer of convolution maps the feature map to the classification of each category, and the output shape is (1, n_classes, H, W). The first parameter (1) indicates that one image is input at a time (batch size is 1). The second parameter (n_classes) indicates that this dimension represents the number of categories. In the chromosome segmentation task, n_classes is 25 (including 24 types of chromosomes and background areas). The third parameter (H) indicates the height of the output image. The fourth parameter (W) indicates the width of the output image. Similar to the height H, W represents the width of the output feature map.

[0062] Step 3.2: The skeleton head predicts the centerline and skeleton of the chromosome instance from the feature map to obtain the skeleton map .

[0063] The main task of the skeleton head is to predict the centerline and skeleton of the chromosome (such as Figure 4The main function of the skeleton head is to generate a "skeleton" segmentation map of the chromosome, which is used to locate and detect features in the corresponding region. It performs convolution on the input feature map to highlight the important structures in the image.

[0064] The output of the skeleton head is a tensor of shape (1, 1, H, W), representing a binarized or scaled probability map, where each pixel value represents the confidence in a specific object region. It uses a convolutional layer to reduce the number of channels of the input feature map to 1 to obtain a one-channel output, which helps in subsequent anchor box generation and detection region decision making.

[0065] Step 3.3: The regression head predicts the box transformation parameters related to the chromosome instance from the feature map and obtains the regression increment of the anchor box. .

[0066] The main task of the regression head is to predict the location parameters related to the chromosome instance, such as the coordinates, size, and rotation angle of the box. These parameters are very important for accurately locating the chromosome and performing quantitative analysis. This is also a very important step in chromosome instance segmentation, which adjusts the anchor box to better align it with the actual target. The output of this head is a shape of (1, num_anchors_per_loc 5, H, W), num_anchors_per_loc is the number of anchor boxes, and the 5 output dimensions correspond to each anchor box's (dx, dy, dw, dh, dtheta), where dx is the offset of the center of the anchor box on the horizontal (x) axis, which indicates the adjustment of the horizontal position of the center of the anchor box in the image. A positive value means moving to the right, and a negative value means moving to the left; dy is the offset of the center of the anchor box on the vertical (y) axis, which indicates the adjustment of the vertical position of the center of the anchor box in the image. A positive value means moving downward, and a negative value means moving upward; dw is the adjustment of the width of the anchor box, usually expressed as the ratio of the width of the anchor box to its original preset width. A positive value usually means that the anchor box becomes wider, and a negative value means that the anchor box becomes narrower; dh is the adjustment of the height of the anchor box, which indicates the ratio of the height of the anchor box to its original preset height. A positive value means that the anchor box is increased, and a negative value means that the anchor box is lowered; dtheta is the change in the angle of rotation of the anchor box, usually expressed in radians or degrees. A positive value indicates a clockwise rotation, and a negative value indicates a counterclockwise rotation. Through these regression increments, the regression head network can learn the transformation relationship from each anchor box to the target box, which is very important for accurately locating chromosomes. For generating the final bounding box, this application follows the following formula:

[0067] px = dx aw + ax;

[0068] py = dy ah + ay;

[0069] pw = torch.exp(dw) aw;

[0070] ph = torch.exp(dh) ah;

[0071] ptheta = dtheta torch.pi + atheta.

[0072] Among them, ax, ay, aw, ah, atheta are the original parameters of the anchor box, dx, dy, dw, dh, dtheta are the regression increments (Box offsets) output by the model, specifically, ax is the x-coordinate of the center point of the anchor box, ay is the y-coordinate of the center point of the anchor box, aw is the width of the anchor box, ah is the height of the anchor box, atheta is the rotation angle of the anchor box, dx is the offset of the x-coordinate of the center point, dy is the offset of the y-coordinate of the center point, dw is the logarithmic scaling factor of the width, dh is the logarithmic scaling factor of the height, and dtheta is the offset of the rotation angle. px is the horizontal coordinate of the center of the prediction box, py is the vertical coordinate of the center of the prediction box, pw is the width of the prediction box, ph is the height of the prediction box, ptheta is the rotation angle of the prediction box, torch.exp is a function in the open source deep learning framework PyTorch, which is used to calculate the exponent of each element of the input tensor, pi is the mathematical constant Π, which can be accessed through the torch.pi function in PyTorch, exp is an exponential function, exp(x) means e^x, where e is the base of the natural logarithm.

[0073] Step 4: Obtain a rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head, and the regression head.

[0074] Once the model is trained, the application can obtain the rotated bounding box and its mask from the three head predictions based on the box and mask proposals, such as Figure 7 As shown, step 4 includes:

[0075] Step 4.1: Perform Argmax calculation to obtain the instance mask (i.e. the marked area).

[0076] Among them, the instance mask refers to a binary mask that marks each independent target instance in the image. Each instance mask only identifies the area of ​​a specific object and is usually used for instance segmentation tasks.

[0077] Both instance masks and the aforementioned segmentation masks involve region identification and object detection in image processing, and are intended to help computers understand the content of an image. The difference is that instance masks focus on distinguishing instances and are suitable for instance segmentation, while segmentation masks focus on class identification and are suitable for semantic segmentation.

[0078] Step 4.2: Segmentation map and skeleton diagram Perform pixel-by-pixel multiplication to get the box score.

[0079] In step 4.2, the box score is defined as:

[0080] Box scores usually refer to the confidence scores associated with a specific detection box, reflecting the probability of the existence of the object in the box. These scores are obtained by dividing the segmentation map ( ) and skeleton diagram ( ) is calculated by pixel-by-pixel multiplication. The calculation formula can be expressed as: = , represents pixel-by-pixel multiplication. In addition, Figure 7 The Score Map is a probability map with the same resolution as the input image that is directly output by the model through the segmentation head. The Score Map has nothing to do with the Box Score: the generation of the Score Map depends only on the image features, not the box score.

[0081] The functions of box scores include:

[0082] 1. Confidence evaluation: The box score is used to measure whether an object exists in a detection box. A high score indicates that the object is likely to exist in the box, while a low score may indicate that the object is missing.

[0083] 2. Improved accuracy: By referencing the segmentation mask and skeleton information, the box score can effectively reduce the confidence of the bounding box near the instance boundary. This processing method can reduce false detections, especially when the object boundaries are blurred or overlapping.

[0084] 3. Optimize subsequent processing: The box score can be further used in subsequent non-maximum suppression (NMS) to help select the most likely detection box, thereby optimizing the final detection result. In the case of local overlap, high-scoring boxes are more likely to be retained.

[0085] 4. Sorting and selection: When making the final box selection and display, the box score can be used as a sorting basis to help the system decide which detection boxes are optimal and which should be discarded.

[0086] The box score is calculated by pixel-by-pixel multiplication and is designed to improve the accuracy and reliability of object detection. It is a key step in achieving effective object recognition.

[0087] Note that this multiplication reduces the confidence of the boxes near the instance boundaries because the skeleton map has higher values ​​(maximum value is 1.0) on the skeleton points of the object and lower values ​​on other pixels.

[0088] Step 4.3: Set Anchor Point .

[0089] Step 4.4: Based on the box score and anchor point The regression increment of the anchor box is obtained by the box decoder; and rotation NMS is performed to reduce the overlapping anchor boxes.

[0090] Step 4.5: Determine mask pixels according to the box scores, and reallocate the mask pixels to each anchor box.

[0091] In step 4.5, the definition of the mask pixel is:

[0092] Mask pixels are used to refer to pixels related to objects or instances in an image. These pixels are usually marked as target areas. In instance segmentation and semantic segmentation tasks, mask pixels are the key to determining region division.

[0093] In step 4.2, the segmentation map and the skeleton map have been multiplied pixel by pixel to generate the box score. This step helps to emphasize the pixels in the area of ​​interest.

[0094] Specifically, the present application applies a threshold to the generated score map, selects high-scoring areas, and further marks them as mask pixels.

[0095] The functions of mask pixels include:

[0096] 1. Target recognition and segmentation: Mask pixels help effectively separate foreground targets from the background for subsequent recognition and analysis.

[0097] 2. Positioning and tracking: Once the mask pixels are obtained, the object can be accurately located and advanced functions such as target tracking can be implemented.

[0098] 3. Enhanced model performance: Accurate acquisition of mask pixels helps improve the accuracy and robustness of subsequent models (such as detection models, classification models, etc.).

[0099] In step 4.5, only overlapping regions and regions with the same classification type are assigned to boxes.

[0100] Furthermore, the box and mask proposals in step 4 are only needed during the testing phase, which avoids the use of box proposals during the training phase and significantly simplifies the implementation.

[0101] Figure 7 Taking chromosomes 1 and 2 as an example, the work performed in the test phase of this application is described. The last part of the figure shows that the probability of the first overlapping chromosome being identified as chromosome 1 is 0.98, and similarly, the probability of the second overlapping chromosome being identified as chromosome 2 is 0.76.

[0102] Embodiment 2:

[0103] Based on Example 1, Example 2 of the present application provides a more specific skeleton-guided chromosome instance segmentation method, including:

[0104] Step 1: Obtain an input image; the input image is a chromosome microscopic image.

[0105] Step 2: Extract feature maps from the input image through the skeleton network.

[0106] When designing the skeleton network for chromosome micrograph feature extraction, this application conducted in-depth analysis and comparison and finally decided not to use other feature extraction networks, but to design a more efficient skeleton network.

[0107] Specifically, the skeleton network of the present application extracts a high-resolution feature map from the input chromosome microscopic image by connecting multiple A2B blocks. Figure 6 The structure of the A2B block is shown. Assume that the input feature map of this block is , the output feature map is , where Din refers to the depth of the input image and Dout refers to the depth of the output image. The mapping relationship between them can be expressed as follows:

[0108] ; .

[0109] in and LN denote convolutional layers and layer normalization, respectively. Yes Input The normalized feature map of . Indicates that and The outputs of the parallel Atrous self-attention modules are connected. The self-attention module is defined as:

[0110] .

[0111] in, represents the weight parameter, represents the attention map, represents the key vector, Represents the query vector. represents pixel-by-pixel multiplication, and , and are the key, query, and value of the self-attention module respectively. , and Unlike the Transformer in

[15] , this application uses dilated convolution to perform these operations because it is more efficient for high-resolution inputs. Formally, these operations can be expressed as:

[0112] ;

[0113] ;

[0114] .

[0115] in, represents dilated convolution, is the dilation rate of the dilated convolution. is an optional transformation function, such as a convolutional layer that downsamples in the channel direction to reduce computational cost. is the attention map, which is calculated as follows:

[0116] .

[0117] in is the pixel index, represents the space of graphs, Represents the temporary pixel index variable during summation, used to calculate the normalization term in the denominator. Represents the attention map Perform an exponential operation to ensure that the result is a positive number, which is necessary in the subsequent Softmax operation. This formula is used to calculate the attention weight , which is obtained by exponentially operating the attention map M and applying the Softmax operation on the spatial dimension. These weights are used to emphasize the important parts of the input feature map so that the network can process information more efficiently. According to the above formula, softmax is performed on the space instead of on the channel. Since the A2B module contains multiple parallel ASA modules with different expansion rates, it is able to extract highly discriminative features with multi-scale information.

[0118] This decision was based on the following key reasons:

[0119] 1. Large intra-class differences and small inter-class differences: Chromosome microscopic images show large morphological and structural differences between chromosomes of the same class, but small differences between chromosomes of different classes. The network often has difficulty effectively distinguishing these subtle differences, and our specially designed skeleton network can better capture and utilize these features, thereby improving the accuracy of classification and recognition.

[0120] 2. Non-rigid structure and bendability: Chromosomes have a non-rigid structure and are prone to bending and deformation in microscopic images. Other feature extraction networks perform poorly when dealing with such non-rigid structures, which can easily lead to distortion and errors in feature extraction. Our specially designed network can better handle the bending and deformation of chromosomes by introducing an optimization algorithm for non-rigid structures, ensuring the stability and reliability of feature extraction.

[0121] 3. Specific task requirements: Feature extraction of chromosome microscopic images is often used in medical diagnosis and research, which requires extremely high accuracy and reliability. Other backbone networks may not be able to meet these high-standard task requirements, but our specially designed network can be optimized according to specific task requirements to ensure that high-standard performance requirements are met.

[0122] In summary, due to the large intra-class differences and small inter-class differences of chromosome microscopic images, as well as the bendable nature of non-rigid structures, this application selected a specially designed skeleton network. This decision ensures the accuracy and efficiency of feature extraction and meets the high-standard task requirements of chromosome microscopic image analysis.

[0123] Step 3: Input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters.

[0124] Step 4: Obtain a rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head, and the regression head.

[0125] In the test phase, in order to suppress the dense distribution problem, this application multiplies the skeleton map by the segmentation map to obtain the final score of each box prediction. Figure 8 The effectiveness of this operation was verified by Figure 8 (a) and Figure 8 The multiplication operation in (b) in Fig. 4 is used to obtain the score of the box proposal, which can indeed solve the dense distribution problem and reduce the bad box proposals near the instance boundary. Figure 8 The prediction results of (a) are: 12|1.0; 5|0.91; 5|1.0; 19|1.0; 4|0.66; 19|1.0; 18|1.0; 4|0.54; 4|0.9; 4|1.0. Figure 8 (a) and Figure 8 The predicted results of the multiplication operation in (b) are: 12|0.88; 5|0.92; 19|0.84; 19|0.87; 4|0.89; 18|0.91. The number on the left of “|” represents the chromosome category number, and the number on the right of “|” is the predicted probability.

[0126] In addition, in order to verify the feasibility of the method of this application, this application conducted experimental analysis on the existing public dataset, the AutoKary2022 chromosome dataset. AutoKary2022 is currently the only publicly available chromosome instance segmentation dataset, so this application conducted relevant experiments on it. The dataset collects 612 high-resolution metaphase images of 3200×2200 from 50 patients, with detailed annotations of each chromosome outline and category.

[0127] like Fig. 9 As shown, the predicted bounding boxes are not shown for clarity. Fig. 9 (a) in the figure is the original chromosome image that was manually annotated. Fig. 9 (b) in the figure is the prediction result of Mask-RCNN chromosome instance segmentation. Fig. 9 (c) in the figure is the prediction result of SCNet chromosome instance segmentation. Fig. 9 (d) in the figure is the prediction result of SOLOv2 chromosome instance segmentation. Fig. 9 (e) in the figure is the prediction result of PolarMask++ chromosome instance segmentation. Fig. 9 (f) in the figure is the predicted result of chromosome instance segmentation of the method of the present application. The red and black arrows indicate false negatives and misclassifications, respectively. By comparing the present application, it is found that the existing methods have more misidentifications than the method of the present application. In particular, for the instance segmentation of overlapping chromosomes, our method is significantly better than the existing methods.

[0128] In addition, the instance segmentation results of the method of the present application are compared with those of the existing methods, and the method of the present application achieves the best segmentation mask. The present application attributes the above phenomenon to the fact that the existing box-based methods (i.e., QueryInst, Mask-RCNN and its variants) perform segmentation based on h-bbox proposals, which may hinder the model from distinguishing instances in cluster areas. Although single-stage methods such as SOLOv2, PolarMask++ and SparseInst have been developed to circumvent box proposals, they still have coarse segmentation problems. This is mainly because they use grid cells, sampled contour points or activation maps to represent instances, which are not fine-grained enough for chromosome objects in microscopic images, especially for chromosomes with elongated and curved shapes. In contrast, the instance segmentation method of the present application is the best method for chromosome instance segmentation, which uses high-resolution pixel-level masks and skeleton-guided rotation bounding boxes to represent objects. For example, the segmentation results of Mask-RCNN are 20|0.95; 6|0.75; 3|0.55; 5|0.99. The segmentation results of SOLOv2 are 20|0.5; 6|0.33; 4|0.7. The segmentation results of PolarMask++ are 20|0.47; 6|0.3; 3|0.19; 5|0.49. The segmentation results of the method of this application are 20|0.85; 6|0.85; 5|0.89.

[0129] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiment 1 can be referenced to each other and will not be described in detail in this application.

[0130] Embodiment 3:

[0131] Based on Examples 1 and 2, Example 3 of the present application provides a chromosome instance segmentation system based on skeleton guidance, including:

[0132] A first acquisition module is used to acquire an input image; the input image is a chromosome microscopic image;

[0133] An extraction module for extracting feature maps from input images through a skeleton network;

[0134] A prediction module, used to input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters;

[0135] The second acquisition module is used to acquire the rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head and the regression head.

[0136] Specifically, the system provided in this embodiment is a system corresponding to the method provided in Embodiments 1 and 2. Therefore, the parts in this embodiment that are the same or similar to Embodiments 1 and 2 can be referenced to each other and will not be repeated in this application.

Claims

1. A skeleton-guided chromosome instance segmentation method, characterized in that: include: Step 1, obtaining an input image; the input image is a chromosome microscopic image; Step 2: Extract feature maps from the input image through the skeleton network; Step 3: Input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters; Step 4: Obtain a rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head, and the regression head.

2. The skeleton-guided chromosome instance segmentation method according to claim 1, characterized in that: In step 2, feature maps are extracted from the input image through multiple A2B blocks.

3. The skeleton-guided chromosome instance segmentation method according to claim 2, characterized in that: Step 3 includes: Step 3.1: The segmentation head predicts the segmentation mask of each chromosome instance from the feature map and obtains the segmentation map. ; The segmentation mask is used to mark the boundaries of each chromosome; Step 3.2: The skeleton head predicts the centerline and skeleton of the chromosome instance from the feature map to obtain the skeleton map ; Step 3.3: The regression head predicts the box transformation parameters related to the chromosome instance from the feature map and obtains the regression increment of the anchor box. .

4. The skeleton-guided chromosome instance segmentation method according to claim 3, characterized in that: In step 3.3, the frame transformation parameters include the coordinates, size and rotation angle of the frame.

5. The skeleton-guided chromosome instance segmentation method according to claim 4, characterized in that: Step 4 includes: Step 4.1: Perform Argmax calculation to obtain the allocation mask; Step 4.2: Segmentation map and skeleton diagram Perform pixel-by-pixel multiplication to obtain the box score; Step 4.3: Set Anchor Point ; Step 4.4: Based on the box score and anchor point The regression increment of the anchor box is obtained by the box decoder; and rotation NMS is performed to reduce the overlapping anchor boxes; Step 4.5: Determine mask pixels according to the box scores, and reallocate the mask pixels to each anchor box.

6. A skeleton-guided chromosome instance segmentation system, characterized in that: The method for executing any one of claims 1 to 5 comprises: A first acquisition module is used to acquire an input image; the input image is a chromosome microscopic image; An extraction module for extracting feature maps from input images through a skeleton network; A prediction module, used to input the feature map into the segmentation head, skeleton head and regression head of the skeleton network, respectively, corresponding to the predicted segmentation mask, skeleton and box transformation parameters; The second acquisition module is used to acquire the rotated bounding box and its mask according to the outputs of the segmentation head, the skeleton head and the regression head.

7. A computer storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed on a computer, the computer executes any one of the methods described in claims 1 to 5.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method according to any one of claims 1 to 5.

Images