A chromosome segmentation method, system, device and medium based on chromosome cluster

Abstract

A chromosome segmentation method, system, device and medium based on chromosome clusters belong to the field of biomedicine and artificial intelligence technology in bioinformatics. The segmentation method comprises: labeling the original chromosome image, sequentially constructing a chromosome cluster recognition data set for different tasks, and training a chromosome cluster recognition network; constructing a chromosome cluster classification data set, and training a chromosome cluster classification network; constructing a chromosome cluster segmentation data set, and training a chromosome cluster segmentation network; inputting the original chromosome image, and sequentially using the trained chromosome cluster recognition network, chromosome cluster classification network and chromosome cluster segmentation network for reasoning to obtain a chromosome segmentation result; the present invention can automatically and accurately identify and segment chromosomes from an observed image; using a chromosome cluster-based architecture to reduce the probability of chromosome missed detection, and the network has high downward compatibility with image resolution.

Description

Technical Field

[0001] The present invention relates to the field of biomedicine and artificial intelligence technology in bioinformatics, and in particular to a chromosome segmentation method, system, device and medium based on chromosome clusters. Background Art

[0002] In biomedical research, chromosome segmentation is a key step in identifying and analyzing chromosome structure and function. Traditional chromosome segmentation methods often directly segment based on image preprocessing, threshold setting, region growing and other steps. Due to the different shapes of chromosomes such as tilt and bend, as well as the influence of noise and the overlap of multiple chromosomes, the accuracy of direct chromosome segmentation is not high.

[0003] In recent years, deep learning has made breakthrough progress in the field of image processing, especially convolutional neural networks have shown excellent performance in tasks such as image classification, object detection and semantic segmentation. Traditional chromosome segmentation methods have difficulty coping with challenges such as image noise, chromosome overlap and complex background, resulting in blurred chromosome boundaries, loss of details, and even possible misidentification of non-chromosome regions. These problems affect the accuracy of segmentation, thus hindering the effectiveness of automatic chromosome analysis.

[0004] There are two main techniques in the field of chromosome segmentation: one is the morphology-based method, which can segment normally in most cases, but has limited performance in the case of chromosome overlap; the other is the detection-based method, which performs well in complex segmentation tasks but has poor stability when the data environment changes. In short, the morphology-based method is more reliable when dealing with non-overlapping chromosomes, but is less effective when dealing with chromosome overlaps; while the detection-based method is more effective in dealing with complex situations, but may not be stable enough under different data environments.

[0005] In summary, the problems existing in the prior art are:

[0006] (1) Difficulty in handling chromosome overlap: Traditional morphology-based methods are usually unable to effectively handle chromosome overlap, especially when the chromosome morphology changes greatly, is tilted or curved. This method often leads to blurred chromosome boundaries or misclassification. Although detection-based methods can effectively handle complex environments, they still have the problem of low segmentation accuracy under challenges such as chromosome overlap and morphological changes.

[0007] (2) Background noise, low contrast, and uneven lighting often occur in chromosome images, which can affect the accuracy of the segmentation network.

[0008] Difficulty in solving the above technical problems:

[0009] Traditional methods do not work well when dealing with overlapping chromosomes. Although deep learning methods perform well, they rely on a large amount of labeled data and have limited generalization capabilities. Directly using instance segmentation networks for chromosome segmentation cannot solve the overlapping problem well and is also greatly affected by noise background.

[0010] The patent application document with publication number CN117078668B discloses a chromosome segmentation method based on large-size images. Through graph cutting, classification network and segmentation network, chromosomes can be automatically and accurately identified and segmented from large-size images. This method uses a dense convolutional neural network, in which a trainable tensor regression layer is used instead of a traditional fully connected layer, thereby significantly improving the training speed and reducing the consumption of computing resources and time. However, since this method requires graph cutting and classification during the reasoning process, and relies on classification to determine the segmented subgraphs, some boundary information is lost, especially when processing complex images, there is a certain loss of accuracy.

[0011] The patent application document with publication number CN113781505A discloses a chromosome segmentation method, a chromosome analyzer and a storage medium. By performing image processing on the original image, obtaining a mask image and extracting a skeleton image, the intersection points and contour key points of the cross chromosomes are determined, thereby achieving accurate segmentation of the cross chromosomes. However, this method may be affected by image noise and incomplete contour extraction when processing highly complex or dense chromosome images, resulting in a decrease in segmentation accuracy. In addition, the skeleton extraction and key point determination process may place high demands on computing resources, especially in high-resolution images, which may result in long processing time and high computing costs. Summary of the invention

[0012] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a chromosome segmentation method, system, device and medium based on chromosome clusters. Combining the advantages of U-Net network in denoising and the advantages of Mask R-CNN network in instance segmentation, a chromosome cluster recognition network based on U-Net network can effectively remove background noise and identify all chromosome clusters, and then use the chromosome cluster classification network to distinguish chromosome clusters containing multiple chromosomes; for chromosome clusters containing multiple chromosomes, a chromosome cluster segmentation network combining Mask R-CNN network, pooling pyramid module PPM and rotation detection frame is used to perform accurate instance segmentation of chromosome clusters, which can reduce the risk of chromosome missed detection, has better segmentation effect for touching or overlapping chromosomes, and has high downward compatibility with image resolution.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is:

[0014] A chromosome segmentation method based on chromosome clusters comprises the following steps:

[0015] Step 1: Annotate the original chromosome image, and build a chromosome cluster recognition data set for different tasks, and train the chromosome cluster recognition network; build a chromosome cluster classification data set, and train the chromosome cluster classification network; build a chromosome cluster segmentation data set, and train the chromosome cluster segmentation network;

[0016] The chromosome cluster segmentation network is based on the Mask R-CNN network, including feature extraction, region proposal network RPN, RoI alignment and branch network; wherein, the feature extraction includes the backbone network BackBone, four pooling pyramid module PPM branches, a concatenation layer Concat and a convolution layer Conv, and the backbone network BackBone selects the ResNet-50 network; the region proposal network RPN includes an RPN module; the RoI alignment includes an RoI alignment module; the branch network includes a category Class, a rotation detection box Rotated Bounding Box and a segmentation mask Mask;

[0017] First, the original image of the chromosome cluster is input, and the feature extraction uses the backbone network BackBone to extract the original image features. Then, the pooling pyramid module PPM is used to pool the original image features at four different scales, and the pooling result is concatenated with the original image features. Then, the number of channels is restored through the convolution layer Conv. The original image of the chromosome cluster is transformed into a feature image after feature extraction.

[0018] The feature image is input into the region proposal network RPN, which generates a large number of candidate regions on the feature image, predicts whether the generated candidate regions contain the target and the corresponding precise position, obtains multiple candidate regions Proposals through non-maximum suppression, and extracts fixed-size regional features from the feature image through RoI alignment operation; finally, the extracted fixed-size regional features are input into the branch network, and for each candidate region, the branch network performs the following tasks in parallel: predict the category of the target, generate a rotated bounding box, and generate a pixel-level binary segmentation mask to obtain the chromosome cluster segmentation result;

[0019] The input and output formula of the pooling pyramid module PPM is:

[0020] in, Represents the input feature map, whose height is H, width is W, and number of channels is C; Indicates output; Indicates the use of k×k pooling window for the input feature map Perform pooling; Indicates that the pooled feature map is upsampled to H×W; Indicates concatenating feature maps in the channel dimension; Indicates that the fused feature map is processed using a 1×1 convolution layer, reducing the number of channels to C;

[0021] The Rotated Bounding Box introduces an additional rotation angle parameter to change the representation of the target box from the traditional (x, y, w, h) to (x, y, w, h, θ), where (x, y) is the center coordinate of the box, w and h are the width and height of the box respectively, and θ is the rotation angle;

[0022] Step 2: Input the original chromosome image, and use the chromosome cluster recognition network, chromosome cluster classification network, and chromosome cluster segmentation network trained in step 1 to perform inference in turn to obtain the chromosome segmentation result.

[0023] The step 1 specifically comprises the following steps:

[0024] Step 1.1: Use the public chromosome segmentation network to pre-label the original chromosome image and obtain the pre-labeled contour information;

[0025] Step 1.2: Use the labelme annotation tool to manually review the pre-annotated contour information obtained in step 1.1, mark the contours of all chromosome instances, obtain the annotated contour information, and construct a chromosome cluster recognition dataset;

[0026] Step 1.3, identifying the chromosome cluster image of the chromosome original image in step 1.1 through the chromosome cluster recognition network, and outputting the contour information of the chromosome cluster image; using the contour information of the chromosome cluster image and the chromosome cluster recognition data set constructed in step 1.2 to train the chromosome cluster recognition network;

[0027] Step 1.4: For the contour information accurately marked in step 1.2, by finding and determining the number of chromosome instances in the chromosome cluster image, a chromosome cluster classification data set is constructed, one type is a chromosome cluster image containing a single chromosome instance, and the other type is a chromosome cluster image containing multiple chromosome instances;

[0028] Step 1.5, classify the chromosome cluster classification data set obtained in step 1.4 through the chromosome cluster classification network, and classify the chromosome cluster images with multiple chromosome instances into positive classes, and classify the chromosome cluster images with a single chromosome instance into negative classes by judging whether there are multiple chromosome instances in the data set. Then, train the chromosome cluster classification network.

[0029] Step 1.6: for the original chromosome image of step 1.1 and the contour information accurately annotated in step 1.2, the chromosome cluster image containing multiple chromosome instances and the chromosome contour annotation information are extracted to construct a chromosome cluster segmentation data set;

[0030] Step 1.7: Use the chromosome cluster segmentation network to perform instance segmentation on the chromosome cluster images with multiple chromosome instances in the chromosome cluster segmentation data set constructed in step 1.6, obtain chromosome instances, and train the chromosome cluster segmentation network.

[0031] The step 2 specifically includes the following steps:

[0032] Step 2.1, identifying all chromosome cluster images and their position information in the chromosome original image through the trained chromosome cluster recognition network obtained in step 1.3;

[0033] Step 2.2, classify the chromosome cluster image identified in step 2.1 by the chromosome cluster classification network trained by step 1.5, and determine whether there are multiple chromosome instances therein, and classify the chromosome cluster image with multiple chromosome instances into a positive class, and classify the chromosome cluster image with a single chromosome instance into a negative class;

[0034] Step 2.3, for the chromosome cluster image and its position information of the single chromosome instance determined in step 2.2, obtain the chromosome instance and its position information therein; perform instance segmentation on the chromosome cluster image determined in step 2.2 with multiple chromosome instances determined by the trained chromosome cluster segmentation network obtained in step 1.7, obtain the remaining chromosome instances, and obtain the position information of the remaining chromosome instances based on the position information of the chromosome cluster image determined in step 2.2 with multiple chromosome instances, and finally obtain all the chromosome instances and their position information;

[0035] Step 2.4: Based on all the chromosome instances and their position information obtained in step 2.3, the position information of the chromosome instances in the chromosome original image in step 2.1 is regressed to obtain a segmentation mask relative to the chromosome original image in step 2.1, and finally the segmentation result of the chromosome original image in step 2.1 is obtained.

[0036] In the step 1.3, the chromosome cluster recognition network includes chromosome pixel classification and chromosome cluster recognition, wherein the chromosome pixel classification is a U-Net network, and the chromosome cluster recognition is composed of a Resize operation, a Softmax layer, and a chromosome cluster contour extraction operation FindContours;

[0037] The original chromosome image is input, and the U-Net network extracts multi-level features through the encoder. Subsequently, the multi-level features are further refined through the bridge layer, and the spatial resolution is gradually restored in the decoder. In this process, the low-level features in the encoder are fused with the high-level features of the corresponding layer of the decoder through jump connections; finally, the U-Net network outputs a pixel category image, and each pixel corresponds to a classification label; then, the pixel category image is converted into a pixel category image of the same size as the input chromosome original image through the Resize operation, and then the probability of each pixel category is obtained through the Softmax layer, and finally the chromosome cluster contour extraction operation FindContours is used to obtain the chromosome outer boundary point set from the pixel category image; the chromosome cluster recognition result is obtained.

[0038] In the step 1.5, the chromosome cluster classification network is based on the ResNet-50 network, which consists of feature extraction and classification, wherein the feature extraction includes a 7×7 convolution layer, a batch normalization layer BN, a ReLU activation function layer, a maximum pooling layer Maxpool and four groups of stacked SE blocks; each SE block includes a residual connection Residual, a global pooling layer Global Pooling, the first fully connected layer FC, a Relu activation function, the second fully connected layer FC, a Sigmoid activation function and a Scale operation; the classification includes a global average pooling layer Avg and a fully connected layer FC;

[0039] The input chromosome instance image first passes through a 7×7 convolutional layer, a batch normalization layer BN, a ReLU activation function layer, and a maximum pooling layer Maxpool in sequence to reduce the image dimension; then, the image passes through four groups of stacked SE blocks in sequence to obtain the chromosome instance feature image. The four groups of stacked SE blocks are composed of 3, 4, 6, and 3 SE blocks, respectively. In each SEblock, the residual structure extracts the global information of the feature through global pooling, and then passes through the first fully connected layer FC, the ReLU activation function, the second fully connected layer, the Sigmoid activation function, and the Scale operation in sequence to dynamically adjust the channel weights of the feature and strengthen the key information; finally, the chromosome instance feature image passes through the global average pooling layer Avg and the fully connected layer to generate a probability distribution for classification; the chromosome cluster classification result is obtained.

[0040] The present invention also provides a chromosome segmentation system based on chromosome clusters, comprising:

[0041] The network training module is used to annotate the original chromosome image, and for different tasks, sequentially construct a chromosome cluster recognition data set to train the chromosome cluster recognition network; construct a chromosome cluster classification data set to train the chromosome cluster classification network; construct a chromosome cluster segmentation data set to train the chromosome cluster segmentation network;

[0042] The network inference module is used to input the original chromosome image, and use the trained chromosome cluster recognition network, chromosome cluster classification network and chromosome cluster segmentation network to perform inference in turn to obtain the chromosome segmentation result.

[0043] The present invention also provides a chromosome segmentation device based on chromosome clusters, comprising:

[0044] Memory: a computer program storing the above-mentioned chromosome segmentation method based on chromosome clusters, which is a computer-readable device;

[0045] Processor: used to implement the chromosome segmentation method based on chromosome clusters when executing the computer program.

[0046] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the chromosome segmentation method based on chromosome clusters.

[0047] In summary, the advantages and positive effects of the present invention are:

[0048] 1. The present invention can significantly improve the segmentation accuracy of chromosomes by using the chromosome cluster recognition network based on the U-Net network after training obtained by step 1.3. The powerful semantic segmentation ability of the U-Net network enables it to accurately extract the pixels of the chromosomes, while effectively removing noise to ensure high-quality segmentation results. In addition, the symmetrical encoder-decoder structure of the U-Net network can capture contextual information and detail features in the image, thereby improving the overall performance of the network. This innovation not only improves the accuracy of chromosome segmentation, but also enhances the adaptability of the network to images of different resolutions, ensuring that stable and accurate segmentation results can be obtained under various input image conditions. Therefore, the present invention solves the problem of boundary information loss in the application document with the prior art publication number CN117078668B. At the same time, the problem of affecting the accuracy of the segmentation network under background noise, low contrast, uneven illumination, etc. is effectively alleviated.

[0049] 2. The present invention significantly improves the accuracy of chromosome cluster classification by using the chromosome cluster classification network based on the trained ResNet-50 network and SE attention mechanism module obtained in step 1.5. As the backbone network, ResNet-50 solves the gradient vanishing problem in the deep network through deep residual connections, and effectively extracts high-level features in the chromosome image. After the SE attention mechanism module is introduced, the network can adaptively adjust the importance of each channel feature, thereby enhancing the key chromosome features and suppressing irrelevant or redundant features. Specifically, the SE attention mechanism module enables the network to distinguish more discriminative features between different chromosome clusters through global average pooling and channel weighting mechanisms, thereby improving classification accuracy.

[0050] 3. The present invention significantly improves the accuracy and robustness of the chromosome segmentation task by using the chromosome cluster segmentation network combined with the Mask R-CNN network, the pooling pyramid module PPM and the rotation detection frame obtained after training in step 1.7. First, the Mask R-CNN network can accurately locate the chromosome clusters based on its powerful target detection and segmentation capabilities, and generate pixel-level masks for each chromosome cluster. Then, combined with the pooling pyramid module PPM, through multi-scale pooling operations, the network can enhance the perception of chromosome clusters of different scales, effectively process multi-scale information in the image, thereby improving the network's segmentation accuracy for large and small chromosome clusters. In addition, the introduction of the rotation detection frame further improves the network's detection ability for rotated or tilted chromosome clusters. Therefore, the present invention solves the problem that the prior art application document with the publication number CN113781505A may be affected by image noise when processing highly complex or dense chromosome images, resulting in incomplete contour extraction. At the same time, the chromosome cluster segmentation network focuses on segmenting chromosomes that are in contact or overlapped, greatly alleviating the problem of difficult processing of chromosome overlap.

[0051] In summary, the present invention realizes accurate extraction of chromosome pixels by utilizing the trained chromosome cluster recognition network obtained in step 1.3, and simultaneously performs denoising processing to effectively remove noise in the image and obtain all chromosome clusters; and utilizes the trained chromosome cluster classification network obtained in step 1.5 to obtain chromosome clusters containing multiple chromosomes; next, the trained chromosome cluster segmentation network obtained in step 1.7 is used to finely segment overlapping or closely arranged chromosome clusters, thereby solving the problems of mutual occlusion and complex morphology between chromosome clusters. The present invention not only significantly improves the accuracy of chromosome segmentation and reduces the risk of missed detection, but also has high image resolution compatibility and can adapt to chromosome images acquired under different devices and conditions. By refining the segmentation strategy, the robustness and adaptability of the segmentation are further improved, so that it can achieve stable and efficient chromosome segmentation in various complex scenarios;

[0052] The present invention adopts deep learning technology and utilizes chromosome cluster recognition network, classification network and segmentation network to automatically and accurately identify and segment chromosomes from observed images; uses a chromosome cluster-based architecture to reduce the probability of chromosome missed detection, and the network has high downward compatibility with image resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a schematic diagram of the principle of the chromosome segmentation method based on chromosome clusters provided in an embodiment of the present invention.

[0054] Figure 2 It is a structural schematic diagram of a chromosome cluster recognition network in a chromosome cluster-based chromosome segmentation method provided in an embodiment of the present invention.

[0055] Figure 3 It is a structural schematic diagram of a chromosome cluster classification network in a chromosome cluster-based chromosome segmentation method provided in an embodiment of the present invention.

[0056] Figure 4 It is a structural schematic diagram of a chromosome cluster segmentation network in a chromosome cluster-based chromosome segmentation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The embodiment of the present invention provides a chromosome segmentation method based on chromosome clusters. Aiming at the problem of chromosome segmentation, combined with the actual chromosome segmentation scenario, a two-stage segmentation architecture is proposed, which effectively solves the challenges of chromosome overlap, complex morphology and image noise. By using the chromosome cluster recognition network to accurately extract chromosome pixels and complete denoising at the same time, all chromosome instances are obtained, and then the chromosome cluster classification network is used to separate the chromosome clusters containing multiple chromosome instances that need to be further segmented, and finally the chromosome cluster segmentation network is used to finely segment the overlapping or closely arranged chromosomes. The present invention not only improves the accuracy of segmentation and reduces the probability of missed detection, but also has high image resolution compatibility and can adapt to chromosome images under different devices and conditions.

[0059] like Figure 1 As shown, a chromosome segmentation method based on chromosome clusters provided by an embodiment of the present invention includes a training process and an inference process. The training process optimizes network parameters by learning the features and rules in the labeled data, and the inference process uses the trained network to predict and apply on new data; specifically, it includes:

[0060] Step 1, training process: annotate the original chromosome image, and build a chromosome cluster recognition data set for different tasks, and train the chromosome cluster recognition network; build a chromosome cluster classification data set, and train the chromosome cluster classification network; build a chromosome cluster segmentation data set, and train the chromosome cluster segmentation network;

[0061] The step 1 training process includes the following steps:

[0062] Step 1.1: Use the public chromosome segmentation network to pre-label the original chromosome image and obtain the pre-labeled contour information;

[0063] Step 1.2: Use the labelme annotation tool to manually review the pre-annotated contour information obtained in step 1.1, mark the contours of all chromosome instances, obtain the annotated contour information, and construct a chromosome cluster recognition dataset;

[0064] Step 1.3, identifying the chromosome cluster image of the chromosome original image in step 1.1 through the chromosome cluster recognition network, and outputting the contour information of the chromosome cluster image; using the contour information of the chromosome cluster image and the chromosome cluster recognition data set constructed in step 1.2 to train the chromosome cluster recognition network;

[0065] The chromosome cluster recognition network includes chromosome pixel classification and chromosome cluster recognition, wherein the chromosome pixel classification is a U-Net network, and the chromosome cluster recognition is composed of a Resize operation, a Softmax layer, and a chromosome cluster contour extraction operation FindContours;

[0066] The original chromosome image is input, and the U-Net network extracts multi-level features through the encoder. Subsequently, the multi-level features are further refined through the bridge layer, and the spatial resolution is gradually restored in the decoder. In this process, the low-level features in the encoder are fused with the high-level features of the corresponding layer of the decoder through jump connections; finally, the U-Net network outputs a pixel category image, and each pixel corresponds to a classification label; then, the pixel category image is converted into a pixel category image of the same size as the input chromosome original image through the Resize operation, and then the probability of each pixel category is obtained through the Softmax layer, and finally the chromosome cluster contour extraction operation FindContours is used to obtain the chromosome outer boundary point set from the pixel category image; the chromosome cluster recognition result is obtained.

[0067] Furthermore, the chromosome cluster recognition network is a U-Net network (U-shaped network), which can effectively capture the contextual information and detail features of the image through its unique symmetric encoder-decoder structure, and perform accurate pixel-level classification. In the U-Net network, the encoder extracts high-level features of the image by downsampling layer by layer, while gradually compressing the spatial dimension; the decoder restores the spatial resolution of the image by upsampling, and connects the feature maps of the encoder and decoder through skip connections to retain low-level detail information. This design enables the U-Net network to extract pixel information of all chromosomes while denoising, avoiding detail loss, thereby reducing the risk of chromosome missed detection. When processing chromosome segmentation tasks, the U-Net network can not only accurately segment chromosome clusters, but also effectively deal with noise in the image, improving the accuracy and robustness of segmentation. In addition, the structure of the U-Net network is particularly suitable for processing high-resolution images, and can be effectively trained with a small amount of labeled data, further enhancing its application advantages in chromosome segmentation tasks.

[0068] Step 1.4: For the contour information accurately marked in step 1.2, by finding and determining the number of chromosome instances in the chromosome cluster image, a chromosome cluster classification data set is automatically constructed, one type is a chromosome cluster image containing a single chromosome instance, and the other type is a chromosome cluster image containing multiple chromosome instances;

[0069] Step 1.5, classify the chromosome cluster classification data set obtained in step 1.4 through the chromosome cluster classification network, and classify the chromosome cluster images with multiple chromosome instances into positive classes, and classify the chromosome cluster images with a single chromosome instance into negative classes by judging whether there are multiple chromosome instances in the data set. Then, train the chromosome cluster classification network.

[0070] The chromosome cluster classification network consists of feature extraction and classification, wherein the feature extraction includes a 7×7 convolution layer, a batch normalization layer BN, a ReLU activation function layer, a maximum pooling layer Maxpool and four stacked SE (Squeeze-and-Excitation) blocks; each SE block includes a residual connection Residual, a global pooling layer Global Pooling, the first fully connected layer FC, a Relu activation function, the second fully connected layer FC, a Sigmoid activation function and a Scale operation; the classification includes a global average pooling layer Avg and a fully connected layer FC;

[0071] The input chromosome instance image first passes through a 7×7 convolutional layer, a batch normalization layer BN, a ReLU activation function layer, and a maximum pooling layer Maxpool in sequence to reduce the image dimension; then, the image passes through four sets of stacked SE blocks in sequence to obtain the chromosome instance feature image. The four sets of stacked SE blocks are composed of 3, 4, 6, and 3 SE blocks, respectively. In each SEblock, the residual structure extracts the global information of the feature through global pooling, and then passes through the first fully connected layer FC, the ReLU activation function, the second fully connected layer, the Sigmoid activation function, and the Scale operation in sequence to dynamically adjust the channel weights of the feature and strengthen the key information; finally, the chromosome instance feature image passes through the global average pooling layer Avg and the fully connected layer to generate a probability distribution for classification; the chromosome cluster classification result is obtained.

[0072] Furthermore, the chromosome cluster classification network is a ResNet-50 network with SE attention mechanism. The ResNet-50 network significantly improves the feature expression ability, especially in the chromosome cluster classification task, and can more effectively capture and utilize key information. The SE module can automatically emphasize the key features in the image and suppress redundant or irrelevant features by adaptively adjusting the importance of each channel feature. Specifically, the SE module can increase the activation of important channels while suppressing those unimportant or noisy channels, effectively enhancing the network's perception of fine features in the chromosome cluster classification task.

[0073] Step 1.6: for the chromosome original image of step 1.1 and the contour information accurately annotated in step 1.2, the chromosome cluster image containing multiple chromosome instances and the chromosome contour annotation information are extracted to automatically construct a chromosome cluster segmentation data set;

[0074] Step 1.7: Use the chromosome cluster segmentation network to perform instance segmentation on the chromosome cluster images with multiple chromosome instances in the chromosome cluster segmentation data set constructed in step 1.6, obtain chromosome instances, and train the chromosome cluster segmentation network.

[0075] The chromosome cluster segmentation network is based on the Mask R-CNN network, including feature extraction, region proposal network RPN, RoI alignment and branch network; wherein, the feature extraction includes the backbone network BackBone, four pooling pyramid module PPM branches, a concatenation layer Concat and a convolution layer Conv, and the backbone network BackBone selects the ResNet-50 network; the region proposal network RPN includes an RPN module; the RoI alignment includes an RoI alignment module (RoIAlign); the branch network includes the category Class, the rotation detection box Rotated Bounding Box and the segmentation mask Mask;

[0076] First, the original image of the chromosome cluster is input, and the feature extraction uses the backbone network BackBone to extract the original image features. Then, the pooling pyramid module PPM is used to pool the original image features at four different scales, and the pooling result is concatenated with the original image features. Then, the number of channels is restored through the convolution layer Conv. The original image of the chromosome cluster is transformed into a feature image after feature extraction.

[0077] The feature image is input into the region proposal network RPN, which generates a large number of candidate regions on the feature image, predicts whether the generated candidate regions contain the target and the corresponding precise position, obtains multiple candidate regions Proposals through non-maximum suppression, and extracts fixed-size regional features from the feature image through RoI alignment operation; finally, the extracted fixed-size regional features are input into the branch network, and for each candidate region, the branch network performs the following tasks in parallel: predict the category of the target, generate a rotated bounding box, and generate a pixel-level binary segmentation mask to obtain the chromosome cluster segmentation result;

[0078] The input and output formula of the pooling pyramid module PPM is:

[0079] in, Represents the input feature map, whose height is H, width is W, and number of channels is C; Indicates output; Indicates the use of k×k pooling window for the input feature map Perform pooling; Indicates that the pooled feature map is upsampled to H×W; Indicates concatenating feature maps in the channel dimension; Indicates that the fused feature map is processed using a 1×1 convolution layer, reducing the number of channels to C;

[0080] The Rotated Bounding Box introduces an additional rotation angle parameter to change the representation of the target box from the traditional (x, y, w, h) to (x, y, w, h, θ), where (x, y) is the center coordinate of the box, w and h are the width and height of the box respectively, and θ is the rotation angle.

[0081] Furthermore, the chromosome cluster segmentation network is based on the Mask R-CNN network, combined with the pooling pyramid module PPM and the rotation detection box Rotated Bounding Box, aiming to efficiently process chromosome clusters of different scales and rotation angles. First, the ResNet-50 network as the backbone network BackBone extracts high-level features of the image and provides rich visual information; then, the pooling pyramid module PPM performs pooling operations at four different scales, concatenates the pooling results with the original image features, and then restores the number of channels through the convolution layer Conv. This process enhances the network's multi-scale context perception ability, enabling it to better handle chromosome clusters of different sizes. After that, the region proposal network RPN generates candidate regions containing chromosome clusters, providing a basis for subsequent target positioning and segmentation. Finally, the chromosome cluster segmentation network accurately regresses the rotation angle of each target through the rotation detection box Rotated Bounding Box, so that it can accurately locate and segment rotated or tilted chromosome clusters. This structure effectively improves the accuracy and robustness of the network in the chromosome segmentation task, especially when dealing with complex chromosome morphology.

[0082] Furthermore, the addition of the pooling pyramid module PPM allows the chromosome cluster segmentation network to extract features at different pooling scales, thereby fusing information from multiple scales into the chromosome cluster segmentation network and focusing more on the segmentation of touching or overlapping chromosomes.

[0083] Furthermore, since the directions of chromosomes in the image are usually not uniform, the Rotated Bounding Box can more accurately fit the rotated or tilted chromosome clusters. Compared with the traditional rectangular frame, the Rotated Bounding Box can wrap the target more tightly, which not only improves the accuracy of target positioning, but also improves the accuracy of subsequent segmentation. Especially when dealing with complex chromosome morphology, it can effectively reduce the interference of background noise and ensure that the segmentation results are more accurate.

[0084] Step 2, reasoning process: Input the original chromosome image, and use the chromosome cluster recognition network, chromosome cluster classification network and chromosome cluster segmentation network trained in step 1 to perform reasoning in turn to obtain the chromosome segmentation result.

[0085] The step 2 reasoning process includes the following steps:

[0086] Step 2.1, identifying all chromosome cluster images and their position information in the chromosome original image through the trained chromosome cluster recognition network obtained in step 1.3;

[0087] Step 2.2, classify the chromosome cluster image identified in step 2.1 by the chromosome cluster classification network trained by step 1.5, and determine whether there are multiple chromosome instances therein, and classify the chromosome cluster image with multiple chromosome instances into a positive class, and classify the chromosome cluster image with a single chromosome instance into a negative class;

[0088] Step 2.3, for the chromosome cluster image and its position information of the single chromosome instance determined in step 2.2, obtain the chromosome instance and its position information therein; perform instance segmentation on the chromosome cluster image determined in step 2.2 with multiple chromosome instances determined by the trained chromosome cluster segmentation network obtained in step 1.7, obtain the remaining chromosome instances, and obtain the position information of the remaining chromosome instances based on the position information of the chromosome cluster image determined in step 2.2 with multiple chromosome instances, and finally obtain all the chromosome instances and their position information;

[0089] Step 2.4: Based on all the chromosome instances and their position information obtained in step 2.3, the position information of the chromosome instances in the chromosome original image in step 2.1 is regressed to obtain a segmentation mask relative to the chromosome original image in step 2.1, and finally the segmentation result of the chromosome original image in step 2.1 is obtained.

[0090] The chromosome segmentation architecture of the present invention comprises two stages. First, the goal of the first stage is to identify all chromosome clusters and classify the chromosome clusters. The chromosome cluster recognition network is based on the powerful semantic segmentation ability of the U-Net network, accurately obtains all chromosome pixels and identifies chromosome clusters, and effectively completes the denoising process. The U-Net network can capture the contextual information and detail features in the image through its symmetrical encoder-decoder structure, ensuring that each chromosome pixel is correctly classified. Subsequently, for chromosome clusters containing multiple chromosomes, the chromosome cluster segmentation network proposed by the present invention performs refined segmentation on chromosome clusters containing multiple chromosomes. This segmentation method can significantly reduce the probability of chromosome missed detection and ensure the integrity and accuracy of the segmentation results. At the same time, the design of the architecture has high downward compatibility with image resolution, can process images of different resolutions, and still maintain high segmentation accuracy at different resolutions. Through this two-stage segmentation strategy, not only the accuracy of detection is improved, but also the robustness of the network in diversified input data is enhanced.

[0091] like Figure 2 As shown, the chromosome segmentation method based on chromosome clusters provided by an embodiment of the present invention, the chromosome cluster recognition network is U-Net, which can effectively capture the context information and detail features of the image through its unique symmetric encoder-decoder structure, can accurately perform pixel-level classification, and extract all chromosome pixels while denoising, reducing the risk of chromosome missed detection.

[0092] The U-Net network can accurately mark the pixels of the chromosomes, generate a binary chromosome mask, and set the background pixels in the image to white. Next, the image is converted into a binary image using Gaussian blur and thresholding operations, and morphological operations (closing operations) are applied to connect the disconnected parts. Then, the FindContours function of OpenCV is used to detect contours in the image and filter out small noise contours.

[0093] like Figure 3 As shown, the chromosome segmentation method based on chromosome clusters provided by an embodiment of the present invention, the chromosome cluster classification network is a ResNet-50 network with SE attention mechanism, which can significantly improve the feature expression ability of the network. By adaptively adjusting the importance of channel features, the SE module can emphasize key information and suppress redundant features, thereby improving the accuracy of the network in image classification. In addition, the computational overhead of the SE module is relatively small.

[0094] like Figure 4 As shown, the chromosome segmentation method based on chromosome clusters provided by an embodiment of the present invention, the chromosome cluster segmentation network is a Mask R-CNN network including a pooling pyramid module PPM and a rotation detection frame; wherein the pooling pyramid module PPM uses a pyramid pooling mechanism. The addition of the pooling pyramid module PPM allows the segmentation network to extract features at different pooling scales, thereby fusing information at multiple scales into the network, and focusing more on the segmentation of touching or overlapping chromosomes. The input and output formulas of the pooling pyramid module PPM are:

[0095] in, Represents the input feature map, whose height is H, width is W, and number of channels is C; Indicates output; Indicates the use of k×k pooling window for the input feature map Perform pooling; Indicates that the pooled feature map is upsampled to H×W; Indicates concatenating feature maps in the channel dimension; It means that the fused feature map is processed using a 1×1 convolution layer to reduce the number of channels to C.

[0096] Moreover, compared with the ordinary Mask R-CNN network, the use of a rotated detection frame will add a rotation angle, and the representation of the target frame becomes (x, y, w, h, θ). In the chromosome detection task, the directions of different chromosomes in the image are not uniform. Using a rotated frame can make the target frame wrap the target more tightly, thereby improving the accuracy of detection and segmentation.

[0097] The two-stage chromosome segmentation architecture obtains all chromosome pixels based on the powerful semantic segmentation capability of the U-Net network, and simultaneously performs denoising; then identifies chromosome clusters; and then uses the chromosome cluster segmentation network proposed in the present invention to segment chromosome clusters containing multiple chromosomes. This segmentation method can significantly reduce the probability of chromosome missed detection, and the network has high downward compatibility with image resolution.

[0098] The present invention is further described below in conjunction with experiments.

[0099] The present invention conducts multiple comparative experiments on a proposed chromosome segmentation method based on chromosome clusters. The present invention constructs a chromosome segmentation dataset and conducts experiments on the dataset of the present invention and the AutoKary 2022 public dataset (as shown in Table 1). The AutoKary2022 dataset is proposed by document 1 [Document 1: You D, Xia P, Chen Q, et al. AutoKary2022: A Large-Scale Densely Annotated Dataset for Chromosome InstanceSegmentation[C] / / 2023 IEEE International Conference on Multimedia and Expo(ICME). IEEE, 2023: 1577-1582.]. The dataset of the present invention includes 678 training images and 59 test images. The training images include 30758 chromosome instances with a resolution of 1008×888. The AutoKary dataset includes 547 training images and 67 test images. The training images include 27109 chromosome instances with a resolution of 3200×2200.

[0100]

[0101] The present invention uses different specifications of Mask R-CNN networks for comparative experiments. The Mask R-CNN network is proposed by document 2 [Document 2: He K, Gkioxari G, Dollár P, et al. Mask r-cnn[C] / / Proceedings of theIEEE international conference on computer vision. 2017: 2961-2969.]; Rotate Mask R-CNN (RMask R-CNN) is proposed by document 3 [Document 3: Wang P, Hu W, Zhang J, et al. Enhanced rotated mask R-CNN for chromosome segmentation[C] / / 2021 43rd AnnualInternational Conference of the IEEE Engineering in Medicine&Biology Society(EMBC). IEEE, 2021: 2769-2772.] and is used for chromosome segmentation scenarios. All experiments were trained for 50 rounds, and all network backbone networks were ResNet-50, all trained on 2 * NVIDIA RTX3090 GPUs using the Adam optimizer, with an initial learning rate of 0.0002 and a decay exponent of 0.96 per round. The key indicators for evaluating network performance in the present invention include mAP (average precision), AP50, AP75, and mAR (average recall rate) on the predicted box and predicted segmentation mask. Among them, mAP is the average precision under different IoU (Intersection over Union) thresholds, which can comprehensively reflect the detection performance of the network in various situations. Specifically, AP50 represents the average precision at an IoU threshold of 0.5, which is usually used to evaluate the basic detection ability of the model network, while AP75 is the average precision at an IoU threshold of 0.75, reflecting the performance of the network under more stringent conditions. mAR represents the average recall rate, which reflects the recall ability of the network on various targets, that is, the proportion of positive samples successfully identified by the network.

[0102] The segmentation results of the dataset of the present invention (as shown in Table 2) and the segmentation results of the AutoKary 2022 dataset (as shown in Table 3) show that the chromosome distribution in the dataset of the present invention is denser than that in the AutoKary2022 dataset, so the Mask R-CNN with a rotation detection box has a slightly better effect on the dataset of the present invention, while the MaskR-CNN without a rotation box has a slightly better effect on the AutoKary2022 dataset. But overall, the mAP index of the present invention has been greatly improved. The segmentation index mAP on the dataset of the present invention has increased by 5.2%, and the segmentation index mAP on the AutoKary 2022 dataset has increased by 2.1%. It also shows that the present invention has improved more in the scenario of chromosome contact and overlap. The present invention can make the chromosome cluster segmentation network have a stronger segmentation ability for contacting or overlapping chromosomes.

[0103] In order to verify the effect of the pooling pyramid module PPM, a comparative test was conducted on the dataset of the present invention, and the pooling pyramid module PPM was added to four specifications of segmentation networks. For the network without FPN, the pooling pyramid module PPM was added directly after the feature extraction; for the network with FPN, the pooling pyramid module PPM was added to the last layer of FPN. The experimental results of the pooling pyramid module PPM on the dataset of the present invention (as shown in Table 4) show that in the four specifications of the network, the segmentation index mAP brought 2.5%, 2.7%, 2.6% and 3.8% improvement respectively after adding the pooling pyramid module PPM. Through comparative experiments, it is proved that the pooling pyramid module PPM can improve the accuracy of the chromosome instance segmentation network.

[0104]

[0105]

[0106]

[0107] The present invention also provides a chromosome segmentation system based on chromosome clusters, comprising:

[0108] The network training module is used to implement the labeling of the original chromosome image in step 1, and to construct a chromosome cluster recognition data set for different tasks, and train the chromosome cluster recognition network; construct a chromosome cluster classification data set, and train the chromosome cluster classification network; construct a chromosome cluster segmentation data set, and train the chromosome cluster segmentation network;

[0109] The network inference module is used to input the original chromosome image in step 2, and use the chromosome cluster recognition network, chromosome cluster classification network and chromosome cluster segmentation network trained in step 1 to perform inference in turn to obtain the chromosome segmentation result.

[0110] The present invention also provides a chromosome segmentation device based on chromosome clusters, comprising:

[0111] Memory: a computer program storing the above-mentioned chromosome segmentation method based on chromosome clusters, which is a computer-readable device;

[0112] Processor: used to implement the chromosome segmentation method based on chromosome clusters when executing the computer program.

[0113] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the chromosome segmentation method based on chromosome clusters.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A chromosome segmentation method based on chromosome clusters, characterized in that: The following steps are involved: Step 1: Annotate the original chromosome image, and build a chromosome cluster recognition data set for different tasks, and train the chromosome cluster recognition network; build a chromosome cluster classification data set, and train the chromosome cluster classification network; build a chromosome cluster segmentation data set, and train the chromosome cluster segmentation network; The chromosome cluster segmentation network is based on the Mask R-CNN network, including feature extraction, region proposal network RPN, RoI alignment and branch network; wherein, the feature extraction includes the backbone network BackBone, four pooling pyramid module PPM branches, a concatenation layer Concat and a convolution layer Conv, and the backbone network BackBone selects the ResNet-50 network; the region proposal network RPN includes an RPN module; the RoI alignment includes an RoI alignment module; the branch network includes a category Class, a rotation detection box Rotated Bounding Box and a segmentation mask Mask; First, the original image of the chromosome cluster is input, and the feature extraction uses the backbone network BackBone to extract the original image features. Then, the pooling pyramid module PPM is used to pool the original image features at four different scales, and the pooling result is concatenated with the original image features. Then, the number of channels is restored through the convolution layer Conv. The original image of the chromosome cluster is transformed into a feature image after feature extraction. The feature image is input into the region proposal network RPN, which generates a large number of candidate regions on the feature image, predicts whether the generated candidate regions contain the target and the corresponding precise position, obtains multiple candidate regions Proposals through non-maximum suppression, and extracts fixed-size regional features from the feature image through RoI alignment operation; finally, the extracted fixed-size regional features are input into the branch network, and for each candidate region, the branch network performs the following tasks in parallel: predict the category of the target, generate a rotated detection box RotatedBounding Box, and generate a pixel-level binary segmentation mask to obtain the chromosome cluster segmentation result; The input and output formula of the pooling pyramid module PPM is: I out =Conv 1×1 (Concat(I in ,Upsp(Pool1(I in ),Upsp(Pool2(I in ),Upsp(Pool3(I in ),Upsp(Pool4(I in ))))))))Among them, I in I represents the input feature map, whose height is H, width is W, and number of channels is C; out Indicates output; Pool k (I in ) indicates that the input feature map I is pooled using a k×k pooling window in Pooling is performed; Upsp(·) means upsampling the pooled feature map to H×W; Concat(·) means concatenating the feature map in the channel dimension; Conv 1×1 (·) indicates that the fused feature map is processed using a 1×1 convolutional layer, reducing the number of channels to C; The Rotated Bounding Box introduces an additional rotation angle parameter to change the representation of the target box from the traditional (x, y, w, h) to (x, y, w, h, θ), where (x, y) is the center coordinate of the box, w and h are the width and height of the box respectively, and θ is the rotation angle; The specific steps include: Step 1.1: Use the public chromosome segmentation network to pre-label the original chromosome image and obtain the pre-labeled contour information; Step 1.2: Use the labelme annotation tool to manually review the pre-annotated contour information obtained in step 1.1, mark the contours of all chromosome instances, obtain the annotated contour information, and construct a chromosome cluster recognition dataset; Step 1.3, identifying the chromosome cluster image of the chromosome original image in step 1.1 through the chromosome cluster recognition network, and outputting the contour information of the chromosome cluster image; using the contour information of the chromosome cluster image and the chromosome cluster recognition data set constructed in step 1.2 to train the chromosome cluster recognition network; Step 1.4: For the contour information accurately marked in step 1.2, by finding and determining the number of chromosome instances in the chromosome cluster image, a chromosome cluster classification data set is constructed, one type is a chromosome cluster image containing a single chromosome instance, and the other type is a chromosome cluster image containing multiple chromosome instances; Step 1.5, classify the chromosome cluster classification data set obtained in step 1.4 through the chromosome cluster classification network, and classify the chromosome cluster images with multiple chromosome instances into positive classes, and classify the chromosome cluster images with a single chromosome instance into negative classes by judging whether there are multiple chromosome instances in the data set. Then, train the chromosome cluster classification network. Step 1.6: for the original chromosome image of step 1.1 and the contour information accurately annotated in step 1.2, the chromosome cluster image containing multiple chromosome instances and the chromosome contour annotation information are extracted to construct a chromosome cluster segmentation data set; Step 1.7, using the chromosome cluster segmentation network to perform instance segmentation on the chromosome cluster image with multiple chromosome instances in the chromosome cluster segmentation data set constructed in step 1.6, obtain chromosome instances, and train the chromosome cluster segmentation network; Step 2: Input the original chromosome image, and use the chromosome cluster recognition network, chromosome cluster classification network, and chromosome cluster segmentation network trained in step 1 to perform inference in turn to obtain the chromosome segmentation result.

2. A chromosome segmentation method based on chromosome clusters according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1, identifying all chromosome cluster images and their position information in the chromosome original image through the trained chromosome cluster recognition network obtained in step 1.3; Step 2.2, classify the chromosome cluster image identified in step 2.1 by the chromosome cluster classification network trained by step 1.5, and determine whether there are multiple chromosome instances therein, and classify the chromosome cluster image with multiple chromosome instances into a positive class, and classify the chromosome cluster image with a single chromosome instance into a negative class; Step 2.3, for the chromosome cluster image and its position information of the single chromosome instance determined in step 2.2, obtain the chromosome instance and its position information therein; perform instance segmentation on the chromosome cluster image determined in step 2.2 with multiple chromosome instances determined by the trained chromosome cluster segmentation network obtained in step 1.7, obtain the remaining chromosome instances, and obtain the position information of the remaining chromosome instances based on the position information of the chromosome cluster image determined in step 2.2 with multiple chromosome instances, and finally obtain all the chromosome instances and their position information; Step 2.4: Based on all the chromosome instances and their position information obtained in step 2.3, the position information of the chromosome instances in the chromosome original image in step 2.1 is regressed to obtain a segmentation mask relative to the chromosome original image in step 2.1, and finally the segmentation result of the chromosome original image in step 2.1 is obtained.

3. The chromosome segmentation method based on chromosome clusters according to claim 1, characterized in that: In the step 1.3, the chromosome cluster recognition network includes chromosome pixel classification and chromosome cluster recognition, wherein the chromosome pixel classification is a U-Net network, and the chromosome cluster recognition is composed of a Resize operation, a Softmax layer, and a chromosome cluster contour extraction operation FindContours; The original chromosome image is input, and the U-Net network extracts multi-level features through the encoder. Subsequently, the multi-level features are further refined through the bridge layer, and the spatial resolution is gradually restored in the decoder. In this process, the low-level features in the encoder are fused with the high-level features of the corresponding layer of the decoder through jump connections; finally, the U-Net network outputs a pixel category image, and each pixel corresponds to a classification label; then, the pixel category image is converted into a pixel category image of the same size as the input chromosome original image through the Resize operation, and then the probability of each pixel category is obtained through the Softmax layer, and finally the chromosome cluster contour extraction operation FindContours is used to obtain the chromosome outer boundary point set from the pixel category image; the chromosome cluster recognition result is obtained.

4. The chromosome segmentation method based on chromosome clusters according to claim 1, characterized in that: In the step 1.5, the chromosome cluster classification network is based on the ResNet-50 network, which consists of feature extraction and classification, wherein the feature extraction includes a 7×7 convolution layer, a batch normalization layer BN, a ReLU activation function layer, a maximum pooling layer Maxpool and four groups of stacked SE blocks; each SE block includes a residual connection Residual, a global pooling layer GlobalPooling, the first fully connected layer FC, a Relu activation function, the second fully connected layer FC, a Sigmoid activation function and a Scale operation; the classification includes a global average pooling layer Avg and a fully connected layer FC; The input chromosome instance image first passes through a 7×7 convolutional layer, a batch normalization layer BN, a ReLU activation function layer, and a maximum pooling layer Maxpool in sequence to reduce the image dimension; then, the image passes through four groups of stacked SE blocks in sequence to obtain the chromosome instance feature image. The four groups of stacked SE blocks are composed of 3, 4, 6, and 3 SE blocks, respectively. In each SE block, the residual structure extracts the global information of the feature through global pooling, and then passes through the first fully connected layer FC, the ReLU activation function, the second fully connected layer, the Sigmoid activation function, and the Scale operation in sequence to dynamically adjust the channel weights of the feature and strengthen the key information; finally, the chromosome instance feature image passes through the global average pooling layer Avg and the fully connected layer to generate a probability distribution for classification; the chromosome cluster classification result is obtained.

5. A chromosome segmentation system based on chromosome clusters based on the method according to any one of claims 1 to 4, characterized in that: include: The network training module is used to annotate the original chromosome image, build chromosome cluster recognition data sets in sequence for different tasks, and train the chromosome cluster recognition network; Construct a chromosome cluster classification data set and train the chromosome cluster classification network; construct a chromosome cluster segmentation data set and train the chromosome cluster segmentation network; The chromosome cluster segmentation network is based on the Mask R-CNN network, including feature extraction, region proposal network RPN, RoI alignment and branch network; wherein, the feature extraction includes the backbone network BackBone, four pooling pyramid module PPM branches, a concatenation layer Concat and a convolution layer Conv, and the backbone network BackBone selects the ResNet-50 network; the region proposal network RPN includes an RPN module; the RoI alignment includes an RoI alignment module; the branch network includes a category Class, a rotation detection box Rotated Bounding Box and a segmentation mask Mask; First, the original image of the chromosome cluster is input, and the feature extraction uses the backbone network BackBone to extract the original image features. Then, the pooling pyramid module PPM is used to pool the original image features at four different scales, and the pooling result is concatenated with the original image features. Then, the number of channels is restored through the convolution layer Conv. The original image of the chromosome cluster is transformed into a feature image after feature extraction. The feature image is input into the region proposal network RPN, which generates a large number of candidate regions on the feature image, predicts whether the generated candidate regions contain the target and the corresponding precise position, obtains multiple candidate regions Proposals through non-maximum suppression, and extracts fixed-size regional features from the feature image through RoI alignment operation; finally, the extracted fixed-size regional features are input into the branch network, and for each candidate region, the branch network performs the following tasks in parallel: predict the category of the target, generate a rotated detection box RotatedBounding Box, and generate a pixel-level binary segmentation mask to obtain the chromosome cluster segmentation result; The input and output formula of the pooling pyramid module PPM is: I out =Conv 1×1 (Concat(I in ,Upsp(Pool1(I in ),Upsp(Pool2(I in ),Upsp(Pool3(I in ),Upsp(Pool4(I in ))))))))Among them, I in I represents the input feature map, whose height is H, width is W, and number of channels is C; out Indicates output; Pool k (I in ) indicates that the input feature map I is pooled using a k×k pooling window in Pooling is performed; Upsp(·) means upsampling the pooled feature map to H×W; Concat(·) means concatenating the feature map in the channel dimension; Conv 1×1 (·) indicates that the fused feature map is processed using a 1×1 convolutional layer, reducing the number of channels to C; The Rotated Bounding Box introduces an additional rotation angle parameter to change the representation of the target box from the traditional (x, y, w, h) to (x, y, w, h, θ), where (x, y) is the center coordinate of the box, w and h are the width and height of the box respectively, and θ is the rotation angle; The specific steps include: Step 1.1: Use the public chromosome segmentation network to pre-label the original chromosome image and obtain the pre-labeled contour information; Step 1.2: Use the labelme annotation tool to manually review the pre-annotated contour information obtained in step 1.1, mark the contours of all chromosome instances, obtain the annotated contour information, and construct a chromosome cluster recognition dataset; Step 1.3, identifying the chromosome cluster image of the chromosome original image in step 1.1 through the chromosome cluster recognition network, and outputting the contour information of the chromosome cluster image; using the contour information of the chromosome cluster image and the chromosome cluster recognition data set constructed in step 1.2 to train the chromosome cluster recognition network; Step 1.4: For the contour information accurately marked in step 1.2, by finding and determining the number of chromosome instances in the chromosome cluster image, a chromosome cluster classification data set is constructed, one type is a chromosome cluster image containing a single chromosome instance, and the other type is a chromosome cluster image containing multiple chromosome instances; Step 1.5, classify the chromosome cluster classification data set obtained in step 1.4 through the chromosome cluster classification network, and classify the chromosome cluster images with multiple chromosome instances into positive classes, and classify the chromosome cluster images with a single chromosome instance into negative classes by judging whether there are multiple chromosome instances in the data set. Then, train the chromosome cluster classification network. Step 1.6: for the original chromosome image of step 1.1 and the contour information accurately annotated in step 1.2, the chromosome cluster image containing multiple chromosome instances and the chromosome contour annotation information are extracted to construct a chromosome cluster segmentation data set; Step 1.7, using the chromosome cluster segmentation network to perform instance segmentation on the chromosome cluster image with multiple chromosome instances in the chromosome cluster segmentation data set constructed in step 1.6, obtain chromosome instances, and train the chromosome cluster segmentation network; The network inference module is used to input the original chromosome image, and use the trained chromosome cluster recognition network, chromosome cluster classification network and chromosome cluster segmentation network to perform inference in turn to obtain the chromosome segmentation result.

6. A chromosome segmentation device based on chromosome clusters, characterized in that: include: Memory: a computer program storing a chromosome segmentation method based on chromosome clusters as described in any one of claims 1 to 4, which is a computer-readable device; Processor: used to implement the chromosome segmentation method based on chromosome clusters as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the chromosome segmentation method based on chromosome clusters as described in any one of claims 1 to 4.

Images