Method, device and equipment for automatically segmenting overlapped cells and medium

Abstract

The invention provides an automatic segmentation method for overlapped cells, which can be applied to the technical field of image segmentation. The method comprises the following steps: acquiring a to-be-segmented cell image; extracting a multi-scale feature map of the to-be-segmented cell image by using the backbone network, and performing target category prediction, boundary regression prediction and coarse segmentation mask prediction to obtain a coarse segmentation mask; performing overlapping cell decomposition and recombination on the coarse segmentation mask by using a deformable overlapping removal module to obtain a recombinant segmentation mask; performing bilinear interpolation up-sampling on the recombined segmentation mask by using a point rendering boundary refining module, extracting low-confidence points, and extracting fine-grained features and coarse-grained features of the low-confidence points from different multi-scale feature maps; based on the fine-grained features and the coarse-grained features, the low-confidence points are subjected to reclassification, and categories of the low-confidence points are obtained; and iteratively interpolating the up-sampling and reclassification process until the resolution of the recombined segmentation mask is recovered to the resolution of the original image, and outputting an overlapped cell segmentation result.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more specifically to an automatic segmentation method, apparatus, device, and medium for overlapping cells. Background Technology

[0002] Automated cell segmentation in biological microscopic images can assist biologists and medical professionals in quantitative analysis of biological or medical images, improving detection efficiency, ensuring the accuracy and objectivity of data analysis, and avoiding the subjectivity of manual segmentation. However, due to the tendency of cells to aggregate and overlap in biological microscopic images, and the presence of background interference, existing automated segmentation methods suffer from low cell boundary segmentation accuracy, undersegmentation, or missegmentation when segmenting multiple overlapping cells. This affects the reliability of subsequent biological analyses or clinical diagnoses such as cell counting, cell morphology analysis, or disease diagnosis. Therefore, achieving high-precision automated segmentation of multiple cells in complex backgrounds and improving the robustness and generalization ability of segmentation models is of significant research importance.

[0003] Cell segmentation methods in biological microscopic images mainly fall into two categories: those based on manual feature extraction and those based on deep learning. Segmentation methods based on manual feature extraction rely on hand-designed features and low-level features such as edges, textures, and thresholds. While they do not require a large amount of training data, they are sensitive to noise and have poor generalization ability. In biological microscopic image scenes with complex backgrounds, diverse cell morphologies, and occlusion or overlap of targets, undersegmentation or missegmentation is prone to occur. Compared to segmentation methods based on manual feature extraction, deep learning-based segmentation methods can automatically learn multi-level target feature representations, combining global contextual information and local detail information to capture complex cell morphology and boundary information. They exhibit better robustness and generalization ability for segmenting overlapping or dense cells with interfering background information. Currently, network models based on decomposition-reconstruction strategies can achieve structural decomposition of overlapping cells, effectively improving the model's ability to perceive the spatial structure, boundary contours, and semantic information of overlapping regions. However, this method still suffers from insufficient segmentation accuracy at cell boundaries and is easily affected by background noise, leading to undersegmentation and missegmentation. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To achieve high-precision segmentation of overlapping cells in biological microscopic images and address the issues of low cell boundary segmentation accuracy and undersegmentation or missegmentation caused by background interference, this invention provides an automatic segmentation method, apparatus, device, and medium for overlapping cells. It uses a deep neural network as the backbone to achieve target detection and coarse segmentation; employs deformable convolution to adaptively sample features from geometrically complex overlapping cells, enabling cell decomposition and recombination and effectively suppressing background noise interference; and extracts low-confidence points in the cell boundary region from different feature maps, providing feature enhancement for low-confidence regions at the cell boundary. This allows for reclassification of low-confidence points to accurately determine boundary points, thereby improving the model's segmentation accuracy for cell boundaries.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, embodiments of the present invention provide a method, apparatus, device, and medium for automatic segmentation of overlapping cells.

[0008] According to a first aspect of the present invention, an automatic overlapping cell segmentation method is provided, comprising: acquiring a cell image to be segmented; extracting multi-scale feature maps of the cell image to be segmented using a backbone network, and performing target category prediction, boundary regression prediction, and coarse segmentation mask prediction to obtain a coarse segmentation mask; constructing a deformable de-overlapping module based on deformable convolution, and using the deformable de-overlapping module to decompose and reconstruct overlapping cells on the coarse segmentation mask to obtain a reconstructed segmentation mask; performing bilinear interpolation upsampling on the reconstructed segmentation mask using a point rendering boundary refinement module to extract low-confidence points, and extracting fine-grained features and coarse-grained features of the low-confidence points from different multi-scale feature maps; reclassifying the low-confidence points based on the fine-grained features and coarse-grained features to obtain the category of the low-confidence points; iterating the interpolation upsampling and reclassification process until the resolution of the reconstructed segmentation mask is restored to the original image resolution, and outputting the overlapping cell segmentation result.

[0009] In some exemplary embodiments, a backbone network is used to extract multi-scale feature maps of the cell image to be segmented, and target category prediction, boundary regression prediction, and coarse segmentation mask prediction are performed to obtain a coarse segmentation mask. This includes: inputting the cell image to be segmented into a feature pyramid network to perform bottom-up multi-level feature extraction and top-down feature fusion to generate a multi-scale feature map; inputting the multi-scale feature map into a region proposal network to generate candidate bounding boxes; performing a region of interest alignment operation on the candidate bounding boxes to generate regions of interest and region of interest features; inputting the region of interest features into the bounding box regression branch and classification branch of the backbone network to generate the target category and bounding box position; and generating a coarse segmentation mask and coarse segmentation mask features based on the region of interest features using a coarse segmentation mask head composed of standard convolutional layers and deconvolutional layers.

[0010] In some exemplary embodiments, the backbone network includes: a feature pyramid network for generating multi-scale feature maps via bottom-up and top-down paths; a region proposal network for generating candidate bounding boxes based on the multi-scale feature maps; a region of interest alignment layer for aligning the candidate bounding boxes to generate a fixed-size region of interest feature map; bounding box regression and classification branches for predicting the target category and bounding box coordinates; and a coarse segmentation mask head, consisting of four standard convolutional layers and one deconvolutional layer, for generating coarse segmentation mask features.

[0011] In some exemplary embodiments, a deformable de-overlapping module is used to decompose and reconstruct overlapping cells in a coarse segmentation mask to obtain a reconstructed segmentation mask. This includes: fusing features of the region of interest (ROI) and features of the coarse segmentation mask to generate a context-enhanced feature representation; using the deformable convolutional layer of the deformable de-overlapping module to add a learnable offset to each sampling position, allowing the convolutional kernel to adaptively change to extract features of overlapping cells, generating an intersection feature map and a complement feature map of overlapping cells, and predicting and generating overlapping region segmentation masks and non-overlapping region segmentation masks; and using a deformable reconstruction mask head to generate a reconstructed segmentation mask based on the overlapping region features in the overlapping region segmentation mask, the non-overlapping region features in the non-overlapping region segmentation mask, and the ROI features.

[0012] In some exemplary embodiments, the deformable de-overlapping module includes a context information enhancement unit for fusing the region of interest features output by the backbone network with the coarse segmentation mask features to generate an enhanced feature representation; a deformable intersection mask head, consisting of 4 deformable convolutional layers and 1 deconvolutional layer, for predicting overlapping region segmentation masks; a deformable complement mask head, consisting of 4 deformable convolutional layers and 1 deconvolutional layer, for predicting non-overlapping region segmentation masks; a deformable reconstruction mask head for fusing intersection, complement features, and shallow region of interest features to generate a reconstructed segmentation mask; and a loss function constraint unit for jointly training the deformable de-overlapping network module using decomposition loss, reconstruction loss, and semantic consistency regularization loss functions.

[0013] In some exemplary embodiments, a point rendering boundary refinement module is used to perform bilinear interpolation upsampling on the reconstructed segmentation mask to extract low-confidence points. Fine-grained and coarse-grained features of these low-confidence points are then extracted from different multi-scale feature maps. This includes: performing resolution upsampling on the reconstructed segmentation mask using bilinear interpolation to obtain a resolution-upgraded segmentation mask; selecting points with confidence levels below a preset confidence threshold from the resolution-upgraded segmentation mask as low-confidence points; and extracting fine-grained and coarse-grained features from the corresponding locations of the low-confidence points as their fine-grained and coarse-grained features, respectively. The fine-grained features originate from higher-resolution feature maps and can capture more refined local information; the coarse-grained features originate from lower-resolution feature maps and contain broader contextual information.

[0014] In some exemplary embodiments, low-confidence points are reclassified based on fine-grained and coarse-grained features to obtain the category of the low-confidence points. This includes: concatenating and fusing the fine-grained and coarse-grained features of the low-confidence points to obtain a point-level feature vector; using a pre-trained classifier based on the point-level feature vector to obtain the probability value of the low-confidence points belonging to each category; and selecting the category corresponding to the highest probability value as the category of the low-confidence points.

[0015] According to a second aspect of the present invention, an automatic overlapping cell segmentation device is provided, comprising: an image input module for acquiring a cell image to be segmented; a backbone network processing module for extracting multi-scale feature maps of the cell image to be segmented using a backbone network, and performing target category prediction, boundary regression prediction, and coarse segmentation mask prediction to obtain a coarse segmentation mask; a de-overlap processing module for constructing a deformable de-overlap module based on deformable convolution, and using the deformable de-overlap module to decompose and recombine overlapping cells on the coarse segmentation mask to obtain a reconstructed segmentation mask; a boundary refinement module for performing bilinear interpolation upsampling on the reconstructed segmentation mask using a point rendering boundary refinement module, extracting low-confidence points, and extracting fine-grained and coarse-grained features of the low-confidence points from different multi-scale feature maps; a reclassification module for reclassifying the low-confidence points based on the fine-grained and coarse-grained features to obtain the category of the low-confidence points; and a result output module for iteratively performing the interpolation upsampling and reclassification process until the resolution of the reconstructed segmentation mask is restored to the original image resolution, and outputting the overlapping cell segmentation result.

[0016] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0017] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0018] (III) Beneficial Effects

[0019] As can be seen from the above technical solutions, the automatic segmentation method, apparatus, device, and medium for overlapping cells provided by the embodiments of the present invention have at least the following beneficial effects:

[0020] (1) It can achieve high-precision automatic segmentation of overlapping cells in complex backgrounds.

[0021] (2) By introducing deformable convolution, the model’s ability to perceive the spatial structure features of overlapping cells with complex geometric structures and the target region is enhanced, thereby reducing the probability of undersegmentation and missegmentation.

[0022] (3) By extracting low-level spatial fine-grained features and high-level semantic coarse-grained features of low-confidence points concentrated in the cell boundary region from different feature maps, the low-confidence features are enhanced and reclassified, the cell boundary is accurately determined, and the model’s segmentation and localization accuracy of the cell boundary is improved. Attached Figure Description

[0023] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0024] Figure 1 A flowchart illustrating an automatic segmentation method for overlapping cells according to an embodiment of the present invention is shown schematically;

[0025] Figure 2 The diagram schematically illustrates the architecture of an automatic segmentation network for overlapping cells according to an embodiment of the present invention.

[0026] Figure 3 The diagram illustrates the architecture of a backbone network according to an embodiment of the present invention.

[0027] Figure 4 This diagram illustrates a comparison of the segmentation results of overlapping cervical cells using different deep learning models.

[0028] Figure 5 An automatic overlapping cell segmentation device according to an embodiment of the present invention is illustrated schematically;

[0029] Figure 6 A block diagram of an electronic device for an automatic overlapping cell segmentation method according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0030] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0032] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0033] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0034] Figure 1 A flowchart illustrating an automatic segmentation method for overlapping cells according to an embodiment of the present invention is shown schematically; Figure 2 The diagram illustrates the architecture of an automatic segmentation network for overlapping cells according to an embodiment of the present invention.

[0035] like Figure 1 and Figure 2 As shown, an automatic segmentation method for overlapping cells according to an embodiment of the present invention includes steps S110-S160.

[0036] In step S110, the cell image to be segmented is obtained.

[0037] In step S120, the backbone network is used to extract multi-scale feature maps of the cell images to be segmented, and target category prediction, boundary regression prediction and coarse segmentation mask prediction are performed to obtain the coarse segmentation mask.

[0038] Figure 3 The diagram illustrates the architecture of a backbone network according to an embodiment of the present invention.

[0039] like Figure 3 As shown, the backbone network according to an embodiment of the present invention includes: a Feature Pyramid Network (FPN) for generating multi-scale feature maps (P2-P6) through bottom-up and top-down paths; a Region Proposal Network (RPN) for generating candidate bounding boxes based on the multi-scale feature maps; a Region of Interest Alignment Layer for aligning the candidate bounding boxes to generate a fixed-size Region of Interest feature map; a Bounding Box Regression Branch and a Classification Branch for predicting the target category and bounding box coordinates; and a coarse segmentation mask head, consisting of 4 standard convolutional layers and 1 deconvolutional layer, for generating coarse segmentation mask features.

[0040] In some exemplary embodiments, step S120 specifically includes: inputting the cell image to be segmented into a feature pyramid network, performing bottom-up multi-level feature extraction and top-down feature fusion to generate a multi-scale feature map; inputting the multi-scale feature map into a region proposal network to generate candidate bounding boxes; performing a region of interest alignment (RoIAlign) operation on the candidate bounding boxes to generate a region of interest (RoI) and region of interest features (RoIAlign). The region of interest features are input into the bounding box regression branch and classification branch of the backbone network to generate the target category and bounding box location. Based on the region of interest features, a coarse segmentation mask head composed of standard convolutional layers and deconvolutional layers is used to generate coarse segmentation mask and coarse segmentation mask features.

[0041] For example, first, the image is input into a Feature Pyramid Network (FPN) to perform bottom-up multi-level feature extraction and top-down feature fusion, generating multi-scale feature maps (P2-P6). The multi-scale feature maps are then input into a Region Proposal Network (RPN) to generate candidate bounding boxes.

[0042] Secondly, the RoIAlign operation is performed to generate the Region of Interest (RoI) and its features based on the candidate bounding boxes. .

[0043] Then, Input the bounding box regression and classification branches of the backbone network to generate target category predictions. Bounding box regression prediction Where k represents the k-th image, and i represents the i-th target in the k-th image. This represents the center coordinates of the i-th target in the predicted k-th image. Here, the width and height of the corresponding predicted target are given. A classification loss function is used. Calculate the classification of the classification branches; use the bounding box regression loss function. Calculate the bounding box of the bounding box regression branch.

[0044] (1)

[0045] (2)

[0046] Where K is the number of images in the current batch. For the number of samples in the category prediction process, This represents the number of positive samples in the bounding box regression prediction process. The i-th target in the k-th image is predicted to be the true category. The probability, and These represent the predicted offset and the actual offset of the bounding box regression, respectively. express Smoothing loss function.

[0047] Finally, a coarse segmentation mask head consisting of 4 standard convolutional layers and 1 deconvolutional layer is constructed and input. To the coarse segmentation mask head, generate a coarse segmentation feature map. With coarse segmentation mask A coarse segmentation mask loss function is adopted. Calculate the coarse segmentation mask of the coarse segmentation mask head.

[0048] (3)

[0049] in, This represents the annotation mask for the i-th target in the k-th image. Represents pixel coordinates, This represents the number of positive samples in the instance segmentation process.

[0050] The overall loss function of the backbone network is constructed based on classification loss, bounding box regression loss, and coarse segmentation mask head loss. :

[0051] (4)

[0052] In step S130, a deformable de-overlapping module is constructed based on deformable convolution. The deformable de-overlapping module is used to decompose and recombine overlapping cells in the coarse segmentation mask to obtain a recombined segmentation mask.

[0053] In some exemplary embodiments, the deformable de-overlap module includes: a context information enhancement unit, used to fuse the region of interest features output by the backbone network with the coarse segmentation mask features to generate an enhanced feature representation; and a deformable intersection mask head ( It consists of 4 deformable convolutional layers and 1 deconvolutional layer, used to predict overlapping region segmentation masks; the mask completion head ( The network consists of four deformable convolutional layers and one deconvolutional layer, used to predict non-overlapping region segmentation masks; a deformable reconstructed mask head, used to fuse intersection, complement features and shallow region of interest features to generate a reconstructed segmentation mask; and a loss function constraint unit, used to jointly train the deformable overlapping network using decomposition loss, reconstructed loss and semantic consistency regularization loss.

[0054] In some exemplary embodiments, step S130 specifically includes: fusing region of interest features and coarse segmentation mask features to generate context-enhanced feature representations; utilizing the deformable convolutional layer of the deformable de-overlap module to add learnable offsets to each sampling position, enabling the convolutional kernel to adaptively change to extract features of overlapping cells, generating overlapping cell intersection feature maps and complement feature maps, and predicting and generating overlapping region segmentation masks and non-overlapping region segmentation masks; and generating reconstructed segmentation masks using deformable reconstruction mask heads based on overlapping region features in the overlapping region segmentation mask, non-overlapping region features in the non-overlapping region segmentation mask, and region of interest features.

[0055] For example, firstly, the region of interest features output by the backbone network are fused with the coarse segmentation mask features to generate context-enhanced feature representations, which are then input into a deformable de-overlapping module.

[0056] Secondly, a deformable intersection mask head is constructed from four deformable convolutional layers and one deconvolutional layer, respectively. and deformable complement mask head The contextual information from the input fusion enhances the features. and The feature map is subjected to deformable convolution using equation (5).

[0057] (5)

[0058] Where x represents the input feature map, and R represents the sampling region. Indicates the current sampling position. Indicates the sampling point relative to coordinate offset, The weights of the sampling points represent the learning process. This represents the sampling offset during learning. After four layers of deformable convolution, adaptive feature sampling is performed on overlapping cells to generate intersection feature maps and complement feature maps of overlapping cells. These are then input into a deconvolutional layer to upsample the feature maps and predict and generate an overlapping region segmentation mask. Segmentation mask for non-overlapping regions Using a decomposed loss function The deformable intersection mask head and the deformable complement mask head are trained as shown in Equation (6).

[0059] (6)

[0060] in, These represent the labeled masks of the overlapping and non-overlapping regions of the i-th target in the k-th image, respectively. This represents the pixel-level binary classification cross-entropy loss function.

[0061] Then, a deformable reconstruction mask head, also consisting of 4 deformable convolutional layers and 1 deconvolutional layer, is constructed. Features from overlapping regions, non-overlapping regions, and shallow regions of interest are fused and input into the deformable reconstruction mask head. The generated recombination feature map is fed into a deconvolutional layer for upsampling, and an overlapping cell recombination segmentation mask is predicted and generated. The recombination loss function is adopted. The deformable and reconfigurable mask head is trained as shown in Equation (7).

[0062] (7)

[0063] Finally, merge the predicted intersection regions. With complement region Using semantic consistency regularization loss function The training deformable deoverlapping module is as shown in Equation (8).

[0064] (8)

[0065] Total loss function of deformable deoverlapping module ,in, , These are the weight coefficients for the decomposition loss term, the recombination loss term, and the semantic consistency regularization term, respectively.

[0066] (9)

[0067] In step S140, the point rendering boundary refinement module is used to perform bilinear interpolation upsampling on the recombined segmentation mask to extract low-confidence points, and to extract fine-grained and coarse-grained features of the low-confidence points from different multi-scale feature maps.

[0068] In some exemplary embodiments, step S140 specifically includes: performing resolution enhancement processing on the reconstructed segmentation mask using bilinear interpolation upsampling to obtain a resolution-enhanced segmentation mask; selecting points with confidence scores lower than a preset confidence threshold from the resolution-enhanced segmentation mask as low-confidence points; and extracting fine-grained and coarse-grained features at the corresponding locations based on the location information of the low-confidence points as the fine-grained and coarse-grained features of the low-confidence points, wherein the fine-grained features come from a higher resolution feature map and can capture more refined local information; and the coarse-grained features come from a lower resolution feature map and contain more extensive contextual information.

[0069] In step S150, based on fine-grained features and coarse-grained features, low-confidence points are reclassified to obtain the category of the low-confidence points.

[0070] In some exemplary embodiments, step S150 specifically includes: concatenating and fusing the fine-grained features and coarse-grained features of the low-confidence points to obtain a point-level feature vector; based on the point-level feature vector, using a pre-trained classifier, obtaining the probability values ​​of the low-confidence points belonging to each category; and selecting the category corresponding to the largest probability value as the category of the low-confidence point.

[0071] In step S160, the iterative interpolation, upsampling, and reclassification process continues until the resolution of the recombined segmentation mask is restored to the original image resolution, and the overlapping cell segmentation results are output.

[0072] For example, firstly, the reconstructed segmentation mask output by the deformable deoverlap module is upsampled by bilinear interpolation and used as input to the point rendering boundary refinement module.

[0073] Secondly, during the training phase, kN candidate points are randomly selected from the upsampled recombined segmentation mask, where N is the number of reclassification points and k (k>1) is the fold increase of the candidate points. From these candidate points, the points with the lowest prediction confidence, i.e., those concentrated at the boundaries, are selected. Points, and randomly selected One point, Uncertainty point sampling ratio. For each reclassification point, fine-grained local features are extracted from the high-resolution feature map P2 output by FPN, and coarse-grained global features are extracted from the upsampled segmentation results. The fine-grained and coarse-grained features are fused to generate an information-enhanced point-level feature vector.

[0074] Then, the point-level feature vectors are input into a lightweight multilayer perceptron consisting of three one-dimensional convolutional layers to reclassify the N points, using point rendering loss. The multilayer perceptron is trained as shown in Equation (10).

[0075] (10)

[0076] in, This represents the true label of the i-th reclassification point. This represents the prediction result of the multilayer perceptron for that point.

[0077] Finally, in the prediction phase, firstly, the top N low-confidence points with inference probabilities closest to 0.5 are selected from the segmentation results after bilinear interpolation upsampling of the recombined segmentation mask. Secondly, for each low-confidence point, the fine-grained features extracted in P2 and the coarse-grained features extracted in the upsampled segmentation results are fused to generate a point-level feature vector. Then, the point-level feature vector of the low-confidence points is input into the trained multilayer perceptron to reclassify the low-confidence points, determine their categories, and generate new segmentation results. This process of upsampling and reclassifying the segmentation results is iterated until the segmentation results reach the resolution of the original image, generating segmentation results with refined boundaries.

[0078] The overall loss function of the method of the present invention is shown in formula (11).

[0079] (11)

[0080] in, These are the weight coefficients for the backbone network, the deformable de-overlapping module, and the point rendering boundary refinement module, respectively.

[0081] Example 1:

[0082] Step S1: Construct training, validation, and test sets using 45, 90, and 810 overlapping cervical cell image segmentation challenge images from the publicly available dataset (https: / / cs.adelaide.edu.au / ~carneiro / isbi14_challenge / dataset.html). All images in this dataset contain high-quality cell nucleus and cytoplasm annotations.

[0083] Step S2: Construct the backbone network. The number of candidate boxes generated by RPN and fed into the detection head is 1000. The feature map size of each ROI generated by ROI Align is 7×7. The instance mask head convolutional layer uses a 3×3 kernel with a stride of 1 and padding of 1. The deconvolutional layer uses a 2×2 kernel with a stride of 2 and padding of 0. During the training phase, the backbone network loss is calculated according to equation (1), and the weight coefficients of the backbone network modules are taken. =1.

[0084] Step S3: Construct a deformable de-overlapping module. The deformable intersection mask, deformable convolutional mask, and deformable recombination mask all use deformable convolutional layers with a 3×3 kernel size, a stride of 1, and a padding of 1. A deconvolutional layer with a 2×2 kernel size is used, with a stride of 2 and a padding of 0. During the training phase, the loss function of the de-overlapping module is calculated according to equation (8), and the weight coefficients of the decomposed loss terms are taken respectively. Weighting coefficient for restructuring loss item Semantic consistency regularization term weight coefficient Take the weight coefficients of the deformable de-overlapping module. =1.

[0085] Step S4: Construct a point rendering boundary refinement module, with the following settings: number of sampling points N=1024, uncertainty point sampling ratio β=0.75, candidate point amplification factor k=3, and a 1×1 convolution kernel used for the multilayer perceptron, with a stride of 1 and padding of 0. During the training phase, calculate the loss function of the point rendering boundary refinement module according to equation (9). Weight coefficients of the point-based rendering boundary refinement module =1.

[0086] Step S5: Train the model using the SGD optimizer with a momentum of 0.9 and an initial learning rate of 0.001. Add linear warm-up in the first 1000 iterations. Train the network for a total of 30,000 iterations. Reduce the learning rate by 0.1 times after the 20,000th and 25,000th iterations, respectively.

[0087] To verify the performance of the deep learning-based automatic segmentation method for overlapping cells of this invention, the method of this invention was used to segment overlapping cervical cells.

[0088] Figure 4 The diagram illustrates a comparison of the segmentation results of overlapping cervical cells using different deep learning models.

[0089] like Figure 4 The segmentation results show that the Mask R-CNN convolutional neural network (MRCNN) segmented cell boundaries with jagged edges and misidentified multiple overlapping cells as single cells. While the DoNet model significantly improved segmentation accuracy, the segmented cell results were overly smoothed, and the problem of misidentifying multiple overlapping cells as single cells still existed. The method of this embodiment has higher segmentation accuracy, generates clearer segmentation contours, effectively preserves detailed information at cell boundaries, and avoids misidentifying overlapping cells as single cells.

[0090] Table 1 compares the mAP (memory accuracy) of the proposed method with other deep learning methods for segmenting the ISBI 2014 cervical cell image public dataset. Here, Nuc., Cyt., and Avg. represent the mAP values ​​for the cell nucleus, cytoplasm, and overall target, respectively. "Model-D" indicates a model without deformable convolution, and "Model-P" indicates a model without the point rendering boundary refinement module. The results in Table 1 show that the mAP of the proposed method is superior to other deep learning-based overlapping cell segmentation methods, demonstrating better performance in overlapping cell segmentation and verifying the accuracy of the proposed method for overlapping cell segmentation. Furthermore, without deformable convolution (Model-D), the mAP in the cytoplasm (Cyt.) region decreases by 0.37%, and without using the point rendering module to reconstruct the segmentation mask for iterative refinement (Model-P), the mAP in the cytoplasm (Cyt.) region decreases by 0.88%. This indicates that introducing the deformable convolution de-overlap module and the point rendering boundary refinement module into the model significantly improves the model's ability to accurately segment complex boundaries of overlapping cells.

[0091] Table 1. Target segmentation results of different deep learning-based overlapping cell target segmentation methods

[0092]

[0093] Figure 5 An automatic overlapping cell segmentation device according to an embodiment of the present invention is illustrated schematically.

[0094] like Figure 5 As shown, the overlapping cell automatic segmentation device 800 of this embodiment includes an image input module 810, a backbone network processing module 820, an overlap removal processing module 830, a boundary refinement module 840, a reclassification module 850, and a result output module 860.

[0095] Image input module 810 is used to acquire cell images to be segmented.

[0096] The backbone network processing module 820 is used to extract multi-scale feature maps of the cell image to be segmented using the backbone network, and to perform target category prediction, boundary regression prediction and coarse segmentation mask prediction to obtain a coarse segmentation mask.

[0097] The de-overlap processing module 830 is used to construct a deformable de-overlap module based on deformable convolution, and to use the deformable de-overlap module to decompose and recombine overlapping cells on the coarse segmentation mask to obtain a recombined segmentation mask.

[0098] The boundary refinement module 840 is used to perform bilinear interpolation upsampling on the recombined segmentation mask using the point rendering boundary refinement module, extract low-confidence points, and extract fine-grained and coarse-grained features of low-confidence points from different multi-scale feature maps.

[0099] The reclassification module 850 is used to reclassify the low-confidence points based on the fine-grained features and the coarse-grained features, and obtain the category of the low-confidence points.

[0100] The result output module 860 is used to iterate the interpolation upsampling and reclassification processes until the resolution of the recombined segmentation mask is restored to the original image resolution, and output the overlapping cell segmentation results.

[0101] According to embodiments of the present invention, any plurality of modules among the image input module 810, backbone network processing module 820, de-overlap processing module 830, boundary refinement module 840, reclassification module 850, and result output module 860 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the image input module 810, backbone network processing module 820, de-overlap processing module 830, boundary refinement module 840, reclassification module 850, and result output module 860 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the image input module 810, backbone network processing module 820, de-overlap processing module 830, boundary refinement module 840, reclassification module 850, and result output module 860 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0102] Figure 6 A block diagram of an electronic device for an automatic overlapping cell segmentation method according to an embodiment of the present invention is shown schematically.

[0103] like Figure 6As shown, an electronic device 900 according to an embodiment of the present invention includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0104] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0105] According to an embodiment of the present invention, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0106] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0107] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0108] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

Claims

1. A method for automatic segmentation of overlapping cells, characterized in that, include: Obtain the cell image to be segmented; The backbone network is used to extract multi-scale feature maps of the cell image to be segmented, and target category prediction, boundary regression prediction and coarse segmentation mask prediction are performed to obtain the coarse segmentation mask. A deformable de-overlapping module is constructed based on deformable convolution. The deformable de-overlapping module is used to decompose and recombine overlapping cells in the coarse segmentation mask to obtain a recombined segmentation mask. The recombined segmentation mask is upsampled by bilinear interpolation using the point rendering boundary refinement module to extract low-confidence points, and fine-grained and coarse-grained features of the low-confidence points are extracted from different multi-scale feature maps. Based on the fine-grained features and the coarse-grained features, the low-confidence points are reclassified to obtain the category of the low-confidence points; The interpolation upsampling and reclassification processes are iterated until the resolution of the recombined segmentation mask is restored to the original image resolution, and the overlapping cell segmentation results are output.

2. The method according to claim 1, characterized in that, The process involves extracting multi-scale feature maps from the cell image to be segmented using a backbone network, and performing target category prediction, boundary regression prediction, and coarse segmentation mask prediction to obtain the coarse segmentation mask, including: The cell image to be segmented is input into a feature pyramid network for bottom-up multi-level feature extraction and top-down feature fusion to generate a multi-scale feature map. The multi-scale feature map is input into a region proposal network to generate candidate bounding boxes; Perform a region of interest alignment operation on the candidate bounding boxes to generate regions of interest and region of interest features; The features of the region of interest are input into the bounding box regression branch and classification branch of the backbone network to generate the target category and bounding box location; Based on the region of interest features, a coarse segmentation mask and coarse segmentation mask features are generated using a coarse segmentation mask head composed of standard convolutional layers and deconvolutional layers.

3. The method according to claim 2, characterized in that, The backbone network includes: Feature pyramid network is used to generate multi-scale feature maps through bottom-up and top-down paths; Region proposal network for generating candidate bounding boxes based on multi-scale feature maps; The region of interest alignment layer is used to align the candidate bounding boxes with features and generate a fixed-size region of interest feature map. The bounding box regression branch and classification branch are used to predict the target category and bounding box coordinates; The coarse segmentation mask head consists of 4 standard convolutional layers and 1 deconvolutional layer, and is used to generate coarse segmentation mask features.

4. The method according to claim 2, characterized in that, The step of using the deformable de-overlap module to decompose and recombine overlapping cells in the coarse segmentation mask to obtain a recombined segmentation mask includes: By fusing the region of interest features with the coarse segmentation mask features, a context-enhanced feature representation is generated. The deformable convolutional layer of the deformable de-overlapping module adds a learnable offset to each sampling position, so that the convolutional kernel adapts to extract features of geometrically complex overlapping cells, generates overlapping cell intersection feature maps and complement feature maps, and predicts and generates overlapping region segmentation masks and non-overlapping region segmentation masks. Based on the overlapping region features in the overlapping region segmentation mask, the non-overlapping region features in the non-overlapping region segmentation mask, and the region of interest features, a reconstructed segmentation mask is generated using a deformable reconstructed mask head.

5. The method according to claim 4, characterized in that, The deformable de-overlap module includes: The context information enhancement unit is used to fuse the region of interest features output by the backbone network with the coarse segmentation mask features to generate enhanced feature representations; The deformable intersection mask head consists of 4 deformable convolutional layers and 1 deconvolutional layer, and is used to predict the segmentation mask of overlapping regions. The deformable complement mask head consists of 4 deformable convolutional layers and 1 deconvolutional layer, and is used to predict non-overlapping region segmentation masks. A deformable reconstructable mask head is used to fuse intersection, complement features and shallow region of interest features to generate a reconstructed segmentation mask. Loss function constraint unit, used to jointly train deformable deoverlap network modules using decomposition loss, recombination loss and semantic consistency regularization loss functions.

6. The method according to claim 1, characterized in that, The point rendering boundary refinement module performs bilinear interpolation upsampling on the reconstructed segmentation mask to extract low-confidence points, and extracts fine-grained and coarse-grained features of the low-confidence points from different multi-scale feature maps, including: The reconstructed segmentation mask is upsampled using bilinear interpolation to improve its resolution, resulting in a segmentation mask with improved resolution. Points with confidence levels lower than a preset confidence threshold are selected from the segmentation mask after resolution improvement and designated as low-confidence points. Based on the location information of the low-confidence points, the fine-grained and coarse-grained features of the corresponding locations are extracted as the fine-grained and coarse-grained features of the low-confidence points. Fine-grained features come from higher-resolution feature maps and can capture more detailed local information; coarse-grained features come from lower-resolution feature maps and contain more extensive contextual information.

7. The method according to claim 1, characterized in that, The step of reclassifying the low-confidence points based on the fine-grained features and the coarse-grained features to obtain the category of the low-confidence points includes: The fine-grained features and coarse-grained features of the low-confidence points are concatenated and fused to obtain a point-level feature vector; Based on the point-level feature vectors, a pre-trained classifier is used to obtain the probability values ​​of low-confidence points belonging to each category. The category corresponding to the highest probability value is selected as the category of the low confidence point.

8. An automatic cell segmentation device, characterized in that, The device includes: Image input module, used to acquire cell images to be segmented; The backbone network processing module is used to extract multi-scale feature maps of the cell image to be segmented using the backbone network, and to perform target category prediction, boundary regression prediction and coarse segmentation mask prediction to obtain the coarse segmentation mask. The de-overlap processing module is used to construct a deformable de-overlap module based on deformable convolution, and to use the deformable de-overlap module to decompose and recombine overlapping cells on the coarse segmentation mask to obtain a recombined segmentation mask. The boundary refinement module is used to perform bilinear interpolation upsampling on the recombined segmentation mask using the point rendering boundary refinement module, extract low-confidence points, and extract fine-grained and coarse-grained features of low-confidence points from different multi-scale feature maps. The reclassification module is used to reclassify the low-confidence points based on the fine-grained features and the coarse-grained features, and obtain the category of the low-confidence points. The result output module is used to iterate the interpolation upsampling and reclassification processes until the resolution of the recombined segmentation mask is restored to the original image resolution, and output the overlapping cell segmentation results.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs. The one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images