Cell nucleus instance segmentation method for digital pathology images based on hierarchical cascade network

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a method for instance segmentation of cell nuclei in digital pathology images based on a hierarchical cascade network, aiming to solve the problem that current methods are not capable of labeling cell nuclei with a small sample size. The present invention comprises: preprocessing the full-field digital pathology image, and then encoding it to obtain a hierarchical feature pyramid of M levels; semantically segmenting the hierarchical feature pyramid of the first level and sampling and mask instantiation in sequence to further obtain a first bounding box; proposing anchor box candidates based on the corresponding positions of the hierarchical feature pyramid of the M levels to further obtain a second bounding box; extracting local features and global features from the hierarchical feature pyramid of the M levels; and then performing triple convolution optimization and non-maximum suppression processing to obtain the cell nucleus instance segmentation result. The present invention can be applied to situations where the amount of annotation is small, and can achieve instance detection and segmentation of cell nuclei.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for segmenting cell nuclei in digital pathology images based on a hierarchical cascade network. Background Art

[0002] Cell nucleus instance segmentation and classification of hematoxylin-eosin (H&E) stained digital pathology images is an essential prerequisite for downstream cancer diagnosis and prognosis tasks. It can reduce the workload of pathologists in evaluating histopathological information such as tumor cellularity, tumor-infiltrating lymphocytes, and mitotic counts from digital pathology images. Each of these features has been shown to be useful as a biomarker for treatment prediction and survival analysis. Therefore, an automatic cell nucleus segmentation and classification framework is the first step for subsequent cancer diagnosis and prognosis analysis. Current work mainly focuses on bottom-up methods that use single-level feature maps to segment cell nucleus instances;

[0003] However, the current use of single-level feature maps to segment cell nucleus instances is not sufficient for cell nucleus annotation scenarios with small sample sizes. Summary of the Invention

[0004] In order to solve the above-mentioned problem in the prior art, namely, the problem that the current use of single-level feature maps to segment cell nucleus instances is not competent for the scenario of cell nucleus annotation with a small sample size, the present invention provides a method for cell nucleus instance segmentation in digital pathology images based on a hierarchical cascade network, the method comprising the following steps:

[0005] S100, obtaining a full-field digital pathology image stained with hematoxylin and eosin; preprocessing the full-field digital pathology image to obtain a preprocessed image; the full-field digital pathology image is an image containing cell nuclei:

[0006] S200, encoding the preprocessed image to obtain a hierarchical feature pyramid of M levels;

[0007] S300 , performing semantic segmentation on the hierarchical feature pyramid of the first level and sequentially performing sampling and mask instantiation to obtain an instantiated cell nucleus mask, and converting the instantiated cell nucleus mask into an auxiliary bounding box as a first bounding box;

[0008] S400, based on the positions corresponding to the hierarchical feature pyramid of M levels, multiple candidate boxes are proposed as anchor box candidate boxes; the specific position features of the multiple anchor box candidate boxes are convolved and fully connected, and the processed anchor box candidate boxes are screened as the second bounding box,

[0009] S500: Extract local features and global features from a hierarchical feature pyramid of M levels based on the first bounding box and the second bounding box; and concatenate the extracted local features and global features as mixed features of a cell nucleus instance bounding box.

[0010] S600, performing triple convolution optimization on the hybrid features of the cell nucleus instance bounding box, and outputting a third instance mask of the cell nucleus;

[0011] S700 , performing non-maximum suppression processing on the third instance mask of the cell nucleus, and outputting the processed cell nucleus instance features as a cell nucleus instance segmentation result.

[0012] In a preferred embodiment, the step S100 of preprocessing the full-field digital pathology image specifically includes:

[0013] removing a background area of ​​the full-field digital pathology image by using the Otsu method at a first magnification;

[0014] At the second magnification, the foreground area of ​​the background area of ​​the full-field digital pathology image is cropped with a step size of 256 and a size of 256×256 to generate a series of non-overlapping 256×256 images;

[0015] Converting the series of non-overlapping 256×256 images into RGB images and converting them to 512×512 size by bilinear sampling to obtain a series of non-overlapping 512×512 RGB images;

[0016] The series of non-overlapping 512×512 RGB images are normalized to obtain processed images.

[0017] In a preferred embodiment, the series of non-overlapping 512×512 RGB images are normalized, specifically by processing the pixels of the three RGB channels using a formula, and the method is:

[0018] For R channel: I1=(I R -123.675) / 58.395;

[0019] For G channel: I2=(I G -116.28) / 57.12;

[0020] For channel B: I3=(I B -103.53) / 57.375;

[0021] Among them, I1, I2, and I3 represent the pixel values ​​of the R, G, and B channels of the image after normalization. R, I G , I B Respectively represent the pixel values ​​of the R, G, and B channels of the image before normalization.

[0022] In a preferred embodiment, the pre-processed image is coded to obtain a hierarchical feature pyramid of M levels, and the method is as follows:

[0023] The preprocessed image is encoded by a Swin-Transformer hierarchical encoder to obtain a hierarchical feature pyramid of M levels;

[0024] The Swin-Transformer hierarchical encoder includes a slicing module, a multi-head self-attention module, a normalization layer, a nonlinear activation layer, and a residual connection layer. After each digital pathology image slice passes through the Swin-Transformer hierarchical encoder, an M-level hierarchical feature pyramid can be generated.

[0025] In a preferred embodiment, step S300 specifically includes:

[0026] Performing semantic segmentation on the hierarchical feature pyramid of the first level by convolution, upsampling the semantic segmentation result to obtain an upsampled image; blurring the upsampled image by convolution with a Gaussian kernel function to obtain a blurred image;

[0027] Binarizing the blurred image, and then instantiating the mask using a watershed algorithm to obtain an instantiated cell nucleus mask;

[0028] The instantiated cell nucleus mask is filtered to obtain an instantiated cell nucleus mask, and then the filtered instantiated cell nucleus mask is converted into a first bounding box.

[0029] In a preferred embodiment, step 400 specifically includes:

[0030] According to the corresponding positions of the M-level hierarchical feature pyramid, multiple candidate boxes are proposed as anchor box candidates;

[0031] Perform convolution and full connection processing on the specific position features of multiple anchor frame candidate frames, and convert the processed anchor frame candidate frames into

[0032] The processed anchor box candidate box is subjected to non-maximum suppression processing to remove redundant bounding boxes, and then the bounding boxes with an area smaller than the set area are filtered out to obtain the second bounding box.

[0033] In a preferred embodiment, when M is 4, a hierarchical feature pyramid of four levels is obtained, namely the first level, the second level, the third level, and the fourth level; the step S500 specifically includes:

[0034] For the first level hierarchical feature pyramid and the second level hierarchical feature pyramid, local feature alignment and extraction are performed; specifically:

[0035] For the hierarchical feature pyramid, the coordinate of the upper left starting point of its bounding box is (x p ,y p ), the width and height are w p ,h p , then the coordinates of each level feature pyramid (x i ,y j )satisfy:

[0036] Among them, N is the side length of the feature map size corresponding to the hierarchical feature pyramid;

[0037] The feature value corresponding to each level of the feature pyramid is the coordinate (x i ,y j ) bilinear interpolation of the four integer coordinates of the nearest neighbors; assigning the bilinear interpolation of the four integer coordinates of the nearest neighbors to the corresponding hierarchical feature pyramid, and finally obtaining the instance feature map of the aligned and extracted hierarchical feature pyramid; the feature map size for the cell nucleus instance bounding box optimization is 7×7, that is, N=7, and the feature map size for the instance mask optimization is 14×14, that is, N=14;

[0038] In the third and fourth levels of the hierarchical feature pyramid, similarity global feature extraction is adopted; the specific method is:

[0039] The center point coordinates of the bounding box corresponding to each level of the feature pyramid (x c ,y c ), select each level feature pyramid and the center point coordinates (x c ,y c ) whose cosine similarity is greater than the first threshold is averaged, and the obtained mean is copied into the size of the corresponding bounding box optimized feature map and the instance mask feature map through a copy operation to obtain the extracted instance feature map;

[0040] The extracted instance feature maps are spliced ​​together as the mixed features of the cell nucleus instance bounding box.

[0041] In a preferred embodiment, step S600 specifically includes:

[0042] Inputting the mixed features of the cell nucleus instance bounding box into a first convolution optimization module to generate a first optimized bounding box, and performing the operation of step S500 on the first optimized bounding box to obtain the mixed features of the first cell nucleus instance bounding box;

[0043] Inputting the first instance bounding box hybrid feature into a second convolution optimization module to generate a second optimized bounding box, and performing the operation of step S500 on the second optimized bounding box to obtain a second instance bounding box hybrid feature of the cell nucleus;

[0044] Inputting the second instance bounding box hybrid feature into a third convolution optimization module to generate a third optimized bounding box, performing the operation of step S500 on the third optimized bounding box to obtain a third instance bounding box hybrid feature of the cell nucleus, and outputting a third instance mask of the cell nucleus;

[0045] The first convolution optimization module, the second convolution optimization module and the third convolution optimization module are each composed of two common convolution layers, one convolution module for predicting a bounding box and one convolution module for predicting an instance mask.

[0046] In a preferred embodiment, step S700 specifically includes:

[0047] S701, for the third instance mask of the cell nucleus, starting from the mask with the highest probability, calculate the intersection-and-union ratio (IoU) of the mask with all masks except itself, and remove all masks whose IoU ratio with the mask with the highest probability is greater than a first IoU threshold;

[0048] S702, repeating the process of S701 until there is no third instance mask whose IoU ratio with the mask with the highest probability is greater than the first IoU ratio threshold;

[0049] S703 , the remaining third instance mask of the cell nucleus is output as the cell nucleus instance segmentation result.

[0050] Beneficial effects of the present invention:

[0051] (1) The present invention can integrate multi-level feature map information of cell nuclei, and is applicable to situations where the amount of annotation is small, and can simultaneously realize the functions of cell nucleus detection, segmentation and classification;

[0052] (2) The present invention can provide accurate segmentation and classification effects for downstream work of digital pathology images;

[0053] (3) The present invention can be applied to cell nucleus segmentation in various scenarios, and can achieve good segmentation performance for various types of cell nuclei and can be applied to various complex scenarios, including blurry, overlapping, smaller cell nuclei, elongated cell nuclei and poorly stained scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0055] Figure 1 is a flowchart of a method for segmenting cell nuclei in digital pathology images based on a hierarchical cascade network according to an embodiment of the present invention;

[0056] Figure 2 3 is an effect diagram of the visualization result of the digital pathology image cell nucleus instance segmentation method based on the hierarchical cascade network according to an embodiment of the present invention;

[0057] Figure 3 The digital pathology image cell nucleus instance segmentation module based on the hierarchical cascade network of the embodiment of the present invention is composed of;

[0058] Figure 4 It is a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0060] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0061] The present invention provides a method for segmenting cell nuclei in digital pathology images based on a hierarchical cascade network, the method comprising the following steps:

[0062] S100, obtaining a full-field digital pathology image stained with hematoxylin and eosin; preprocessing the full-field digital pathology image to obtain a preprocessed image; the full-field digital pathology image is an image containing cell nuclei:

[0063] S200, encoding the preprocessed image to obtain a hierarchical feature pyramid of M levels;

[0064] S300 , performing semantic segmentation on the hierarchical feature pyramid of the first level and sequentially performing sampling and mask instantiation to obtain an instantiated cell nucleus mask, and converting the instantiated cell nucleus mask into an auxiliary bounding box as the first bounding box;

[0065] S400, based on the positions corresponding to the hierarchical feature pyramid of M levels, multiple candidate boxes are proposed as anchor box candidate boxes; the specific position features of the multiple anchor box candidate boxes are convolved and fully connected, and the processed anchor box candidate boxes are used as the second bounding box,

[0066] S500: Extract local features and global features from a hierarchical feature pyramid of M levels based on the first bounding box and the second bounding box; and concatenate the extracted local features and global features as mixed features of a cell nucleus instance bounding box.

[0067] S600, performing triple convolution optimization on the hybrid features of the cell nucleus instance bounding box, and outputting a third instance mask of the cell nucleus;

[0068] S700 , performing non-maximum suppression processing on the third instance mask of the cell nucleus, and outputting the processed cell nucleus instance features as a cell nucleus instance segmentation result.

[0069] In order to more clearly illustrate the digital pathology image cell nucleus instance segmentation method based on the hierarchical cascade network of the present invention, each step in the embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0070] The digital pathology image cell nucleus instance segmentation method based on the hierarchical cascade network of the first embodiment of the present invention is as follows: Figure 1 As shown, it includes steps S100 to S700, and each step is described in detail as follows:

[0071] S100, acquiring a full-field digital pathology image stained with hematoxylin and eosin; preprocessing the full-field digital pathology image to obtain a preprocessed image; the full-field digital pathology image is an image containing cell nuclei;

[0072] In a preferred embodiment, the step S100 of preprocessing the full-field digital pathology image specifically includes:

[0073] removing the background area of ​​the full-field digital pathology image by using the Otsu method at a first magnification; preferably, the first magnification may be 5 magnifications;

[0074] At a second magnification, the foreground area of ​​the background area of ​​the full-field digital pathology image is cropped with a step size of 256 and a size of 256×256 to generate a series of non-overlapping 256×256 images; preferably, the second magnification may be 40 magnification;

[0075] Converting the series of non-overlapping 256×256 images into RGB images and converting them to 512×512 size by bilinear sampling to obtain a series of non-overlapping 512×512 RGB images;

[0076] The series of non-overlapping 512×512 RGB images are normalized to obtain processed images.

[0077] In this embodiment, a series of non-overlapping 512×512 RGB images are normalized by normalizing all pixel values. The specific formula is: for the R channel, I=(I R -123.675) / 58.395, for G channel I=(I G -116.28) / 57.12, for B channel I=(I B -103.53) / 57.375. Among them, I1, I2, and I3 represent the pixel values ​​of the R, G, and B channels of the image after normalization. R , I G , I B Respectively represent the pixel values ​​of the R, G, and B channels of the image before normalization.

[0078] S200, encoding the preprocessed image to obtain a hierarchical feature pyramid of M levels;

[0079] In a preferred embodiment, the pre-processed image is coded to obtain a hierarchical feature pyramid of M levels, and the method is as follows:

[0080] The preprocessed image is encoded through a Swin-Transformer hierarchical encoder to obtain a hierarchical feature pyramid with M levels; wherein the Swin-Transformer hierarchical encoder includes a slicing module, a multi-head self-attention module, a normalization layer, a nonlinear activation layer and a residual connection layer; after each digital pathology image slice passes through the Swin-Transformer hierarchical encoder, an M-level hierarchical feature pyramid can be generated.

[0081] In this embodiment, when M is equal to 4, a 4-level feature pyramid network is output, which are feature maps downsampled by 4 times, 8 times, 16 times, and 32 times, respectively. The sizes are 128×128, 64×64, 32×32, 16×16, and the number of channels is 64.

[0082] S300, semantic segmentation is performed on the hierarchical feature pyramid of the first level and sampling and mask instantiation are performed in sequence to obtain an instantiated cell nucleus mask, .. According to the instantiated cell nucleus mask, it is converted into an auxiliary bounding box as the first bounding box; that is, semantic segmentation is performed on the feature map downsampled by 4 times and sampling and mask instantiation are performed in sequence to obtain an instantiated cell nucleus mask, .. According to the instantiated cell nucleus mask, it is converted into an auxiliary bounding box as the first bounding box; the first bounding box corresponds to the attached Figure 1 Auxiliary bounding box in ;

[0083] In a preferred embodiment, step S300 specifically includes:

[0084] Performing semantic segmentation on the hierarchical feature pyramid of the first level by convolution, upsampling the semantic segmentation result to obtain an upsampled image; blurring the upsampled image by convolution with a Gaussian kernel function to obtain a blurred image;

[0085] In this embodiment, based on the semantic segmentation result of 4-fold downsampled features, the semantic segmentation is upsampled to the input image size of 512×512, and then Gaussian blurring with a kernel size of 5 and a variance of 1.1 is performed.

[0086] The blurred image is binarized, and then the mask is instantiated using a watershed algorithm to obtain an instantiated cell nucleus mask. Specifically, in this embodiment, the blurred image can be binarized based on a threshold of 0, and then the mask is instantiated using a watershed algorithm.

[0087] The instantiated cell nucleus mask is filtered to obtain an instantiated cell nucleus mask, and then the filtered instantiated cell nucleus mask is converted into a first bounding box. Specifically, the specific method of filtering is to filter out the instantiated cell nucleus mask with an area greater than 128*128 or less than 10, and then convert it into the first bounding box based on the instantiated cell nucleus mask.

[0088] S400: Based on the positions corresponding to the M-level hierarchical feature pyramid, multiple candidate boxes are proposed as anchor box candidates; specific position features of the multiple anchor box candidates are subjected to convolution and full connection processing, and the processed anchor box candidates are used as the second bounding box; the second bounding box corresponds to the optimized bounding box in the accompanying figure;

[0089] In a preferred embodiment, step 400 specifically includes:

[0090] Based on the positions corresponding to the M-level hierarchical feature pyramid, multiple candidate boxes are proposed as anchor box candidates. When M is 4, in this embodiment, in the four-level feature maps, for the features at each position, 12 types of anchor box candidates are proposed, namely 4×4, 2×4, 4×2, 8×8, 8×4, 4×8, 16×16, 16×8, 8×16, 32×32, 32×16, and 16×32. Convolution and full connection processing are performed on the specific position features of multiple anchor box candidates.

[0091] The processed anchor box candidate box is subjected to non-maximum suppression processing to remove redundant bounding boxes, and then the bounding boxes with an area smaller than the set area are filtered out to obtain the second bounding box.

[0092] In this embodiment, the set area can be specifically set to a value of 10. Specifically, these candidate boxes are processed through the convolution layer and the fully connected layer according to their specific position features to generate corresponding bounding boxes, and the top 3000 candidate boxes are selected according to the foreground prediction probability value. The redundant bounding boxes are removed by the non-maximum suppression strategy with a threshold of 0.7, and finally the bounding boxes with an area less than 10 are filtered out to obtain the second bounding box.

[0093] The first bounding box and the second bounding box are mixed into a first-stage bounding box and output;

[0094] S500, based on the first bounding box and the second bounding box, that is, based on the first-stage bounding box; extract local features and global features from the M-level hierarchical feature pyramid; splice the extracted local features and global features as mixed features of the cell nucleus instance bounding box; in a preferred embodiment, when M is 4, a four-level hierarchical feature pyramid is obtained, namely the first level, the second level, the third level, and the fourth level, corresponding to feature maps of 4x, 8x, 16x, and 32x downsampling, respectively; step S500 specifically includes:

[0095] For the first level hierarchical feature pyramid and the second level hierarchical feature pyramid, local feature alignment and extraction are performed; specifically:

[0096] A local feature alignment strategy is adopted in the 4x (k=2) and 8x (k=3) feature maps. Specifically, the size of the feature map used for cell nucleus instance bounding box optimization after sampling is 7×7 (N=7), and the size of the feature map used for instance mask optimization is 14×14 (N=14);

[0097] For the hierarchical feature pyramid, the coordinate of the upper left starting point of its bounding box is (x p ,y p ), the width and height are w p , h p , then the coordinates of each level feature pyramid (x i ,y j )satisfy:

[0098] Among them, N is the side length of the feature map size corresponding to the hierarchical feature pyramid;

[0099] The feature value corresponding to each level of the feature pyramid is the coordinate (x i ,y j) bilinear interpolation of the four integer coordinates of the nearest neighbors; assigning the bilinear interpolation of the four integer coordinates of the nearest neighbors to the corresponding hierarchical feature pyramid, and finally obtaining the instance feature map of the aligned and extracted hierarchical feature pyramid; the feature map size for cell nucleus instance bounding box optimization is 7×7 (N=7), and the feature map size for instance mask optimization is 14×14 (N=14);

[0100] In the third and fourth levels of the hierarchical feature pyramid, similarity global feature extraction is adopted; the specific method is:

[0101] The center point coordinates of the bounding box corresponding to each level of the feature pyramid (x c ,y c ), select each level feature pyramid and the center point coordinates (x c ,y c ) whose cosine similarity is greater than the first threshold is averaged, and the obtained mean is copied into the size of the corresponding bounding box optimized feature map and the instance mask feature map through a copy operation to obtain the extracted instance feature map;

[0102] The extracted instance feature maps are spliced ​​together as the mixed features of the cell nucleus instance bounding box.

[0103] S600, performing triple convolution optimization on the hybrid features of the cell nucleus instance bounding box, and outputting a third instance mask of the cell nucleus;

[0104] In a preferred embodiment, step S600 specifically includes:

[0105] Inputting the mixed features of the cell nucleus instance bounding box into a first convolution optimization module to generate a first optimized bounding box, and performing the operation of step S500 on the first optimized bounding box to obtain the mixed features of the first cell nucleus instance bounding box;

[0106] Inputting the first instance bounding box hybrid feature into a second convolution optimization module to generate a second optimized bounding box, and performing the operation of step S500 on the second optimized bounding box to obtain a second instance bounding box hybrid feature of the cell nucleus;

[0107] Inputting the second instance bounding box hybrid feature into a third convolution optimization module to generate a third optimized bounding box, performing the operation of step S500 on the third optimized bounding box to obtain a third instance bounding box hybrid feature of the cell nucleus, and outputting a third instance mask of the cell nucleus;

[0108] The first convolution optimization module, the second convolution optimization module, and the third convolution optimization module are all composed of two common convolutional layers, one convolution module for predicting bounding boxes, and one convolution module for predicting instance masks. The three convolution structures are the same but have different parameters. The different parameters of the three convolution structures correspond to the different parameters of each layer caused by training. The settings for hyperparameters such as the convolution kernel size are the same. Specifically, the intersection-over-union ratio thresholds of the three convolution optimization modules during the label matching process are different. The intersection-over-union ratio thresholds of the first convolution optimization module, the second convolution optimization module, and the third convolution optimization module are 0.5, 0.6, and 0.7, respectively. At the same time, in the loss function calculation, the weights of the first convolution optimization module, the second convolution optimization module, and the third convolution optimization module are 1, 0.5, and 0.25, respectively.

[0109] S700 , performing non-maximum suppression processing on the third instance mask of the cell nucleus, and outputting the processed cell nucleus instance features as a cell nucleus instance segmentation result.

[0110] In a preferred embodiment, step S700 specifically includes:

[0111] S701, for the third instance mask of the cell nucleus, starting from the mask with the highest probability, calculate its intersection-over-union (IoU) with all masks except itself. All masks whose IoU with the mask with the highest probability is greater than a first IoU threshold are removed. In this embodiment, the first IoU threshold is 0.05. Specifically, for the output third instance mask of the cell nucleus, the initial maximum suppression based on the bounding box still cannot guarantee the removal of the high overlap between cell nuclei. In the post-processing stage, the cell nucleus mask is first run-length encoded to reduce memory usage. Starting from the mask with the highest probability, calculate its IoU with all mask instances except itself. All masks whose IoU with it is greater than 0.05 are removed.

[0112] S702, repeating the process of S701 until there is no third instance mask whose IoU ratio with the mask with the highest probability is greater than the first IoU ratio threshold;

[0113] S703 , the remaining third instance mask is output as the cell nucleus instance segmentation result.

[0114] Although the various steps in the above embodiment are described in the above-mentioned order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of the present invention.

[0115] The digital pathology image cell nucleus instance segmentation system based on a hierarchical cascade network according to the second embodiment of the present invention includes the following modules:

[0116] The digital pathology image segmentation and preprocessing module is used to obtain a full-field digital pathology image stained with hematoxylin and eosin; preprocess the full-field digital pathology image to obtain a preprocessed image; the full-field digital pathology image is an image containing cell nuclei:

[0117] A multi-scale feature generation module for digital pathology images, configured to encode the pre-processed image to obtain a hierarchical feature pyramid of M levels;

[0118] The cell nucleus bounding box generation module is used to perform semantic segmentation on the first level of the hierarchical feature pyramid and sequentially sample and instantiate the mask to obtain an instantiated cell nucleus mask. The instantiated cell nucleus mask is converted into an auxiliary bounding box as the first bounding box.

[0119] The cell nucleus hybrid feature extraction module based on the candidate frame is used to propose multiple candidate frames as anchor frame candidates according to the corresponding positions of the M-level hierarchical feature pyramid; the specific position features of multiple anchor frame candidate frames are convolved and fully connected, and the processed anchor frame candidate frames are used as the second bounding box.

[0120] a cascade convolution optimization module, configured to extract local features and global features from a hierarchical feature pyramid of M levels based on the first bounding box and the second bounding box; concatenate the extracted local features and global features as mixed features of a cell nucleus instance bounding box; perform triple convolution optimization on the mixed features of the cell nucleus instance bounding box, and output a third instance mask of the cell nucleus;

[0121] A post-processing module based on non-maximum suppression of the mask is used to perform non-maximum suppression processing on the third instance mask of the cell nucleus, and output the processed cell nucleus instance features as the cell nucleus instance segmentation result.

[0122] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working process and related instructions of the system described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0123] It should be noted that the digital pathology image cell nucleus instance segmentation system based on the hierarchical cascade network provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiment can be combined into one module or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are merely for distinguishing the modules or steps and are not to be considered as improper limitations of the present invention.

[0124] An electronic device according to a third embodiment of the present invention includes:

[0125] at least one processor; and

[0126] a memory communicatively connected to at least one of the processors; wherein,

[0127] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned digital pathology image cell nucleus instance segmentation method based on hierarchical cascade network.

[0128] A fourth embodiment of the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to be executed by the computer to implement the aforementioned method for digital pathology image cell nucleus instance segmentation based on a hierarchical cascade network.

[0129] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes and related instructions of the storage device and processing device described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0130] Those skilled in the art should be able to appreciate that, in conjunction with the modules and method steps of each example described in the embodiments disclosed herein, it is possible to implement them with electronic hardware, computer software, or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0131] Reference below Figure 4 , which shows a structural diagram of a computer system of a server for implementing the method, system, and device embodiments of the present application. Figure 4 The server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0132] like Figure 4 As shown, the computer system includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage part 608 to the random access memory (RAM) 603. Various programs and data required for system operation are also stored in the RAM 603. The CPU 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0133] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, are installed in the drive 610 as needed so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0134] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0135] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0136] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0137] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0138] The term "comprise" or any other similar term is intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0139] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for segmenting cell nuclei in digital pathology images based on a hierarchical cascade network, characterized in that: The method comprises the following steps: S100, obtaining a full-field digital pathology image stained with hematoxylin and eosin; preprocessing the full-field digital pathology image to obtain a preprocessed image; the full-field digital pathology image is an image containing cell nuclei: S200, encoding the pre-processed image to obtain a hierarchical feature pyramid of M levels: The preprocessed image is encoded by a Swin-Transformer hierarchical encoder to obtain a hierarchical feature pyramid of M levels; The Swin-Transformer hierarchical encoder includes a slicing module, a multi-head self-attention module, a normalization layer, a nonlinear activation layer, and a residual connection layer. After each digital pathology image slice passes through the Swin-Transformer hierarchical encoder, an M-level hierarchical feature pyramid can be generated. S300 , performing semantic segmentation on the hierarchical feature pyramid of the first level, and performing sampling and mask instantiation in sequence to obtain an instantiated cell nucleus mask, and converting the instantiated cell nucleus mask into an auxiliary bounding box as a first bounding box; S400, based on the positions corresponding to the hierarchical feature pyramid of M levels, multiple candidate boxes are proposed as anchor box candidate boxes; the specific position features of the multiple anchor box candidate boxes are convolved and fully connected, and the processed anchor box candidate boxes are screened as the second bounding box, S500: Extract local features and global features from a hierarchical feature pyramid of M levels based on the first bounding box and the second bounding box; and concatenate the extracted local features and global features as mixed features of the cell nucleus instance bounding box. S600, performing triple convolution optimization on the hybrid features of the cell nucleus instance bounding box, and outputting a third instance mask of the cell nucleus; S700 , performing non-maximum suppression processing on the third instance mask of the cell nucleus, and outputting the processed cell nucleus instance features as a cell nucleus instance segmentation result.

2. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 1, characterized in that: The step S100 of preprocessing the full-field digital pathology image specifically includes: At a first magnification, removing the background area of ​​the full-field digital pathology image by using the Otsu method; At the second magnification, the foreground area of ​​the background area of ​​the full-field digital pathology image is cropped with a step size of 256 and a size of 256×256 to generate a series of non-overlapping 256×256 images; Converting the series of non-overlapping 256×256 images into RGB images and converting them to 512×512 size by bilinear sampling to obtain a series of non-overlapping 512×512 RGB images; The series of non-overlapping 512×512 RGB images are normalized to obtain processed images.

3. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 2, characterized in that: The series of non-overlapping 512×512 RGB images are normalized, specifically by processing the pixels of the three RGB channels using the formula, and the method is: For R channel: I1=(I R -123.675) / 58.395; For G channel: I2=(I G -116.28) / 57.12; For channel B: I3=(I B -103.53) / 57.375; Among them, I1, I2, and I3 represent the pixel values ​​of the R, G, and B channels of the image after normalization. R , I G , I B Respectively represent the pixel values ​​of the R, G, and B channels of the image before normalization.

4. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 1, characterized in that: The step S300 specifically includes: Performing semantic segmentation on the hierarchical feature pyramid of the first level by convolution, upsampling the semantic segmentation result to obtain an upsampled image; blurring the upsampled image by convolution with a Gaussian kernel function to obtain a blurred image; Binarizing the blurred image, and then instantiating the mask using a watershed algorithm to obtain an instantiated cell nucleus mask; The instantiated cell nucleus mask is filtered to obtain an instantiated cell nucleus mask, and then the filtered instantiated cell nucleus mask is converted into an auxiliary bounding box of the cell nucleus, which is recorded as a first bounding box.

5. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 1, characterized in that: The step 400 specifically includes: According to the corresponding positions of the M-level hierarchical feature pyramid, multiple candidate boxes are proposed as anchor box candidates; Perform convolution and full connection processing on the specific position features of multiple anchor frame candidate frames, and convert the processed anchor frame candidate frames into The processed anchor box candidate box is subjected to non-maximum suppression processing to remove redundant bounding boxes, and then the bounding boxes with an area smaller than the set area are filtered out to obtain the second bounding box.

6. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 1, characterized in that: When M is 4, a hierarchical feature pyramid of four levels is obtained, namely the first level, the second level, the third level, and the fourth level; the step S500 specifically includes: In the first level hierarchical feature pyramid and the second level hierarchical feature pyramid, local feature alignment and extraction are adopted; specifically: For the hierarchical feature pyramid, the coordinate of the upper left starting point of its bounding box is (x p ,y p ), the width and height are w p ,h p , then the coordinates of each level feature pyramid (x i ,y j )satisfy: Among them, N is the side length of the feature map size corresponding to the hierarchical feature pyramid; The eigenvalues ​​corresponding to each level of the hierarchical feature pyramid are the coordinates (x i ,y j ) bilinear interpolation of the four integer coordinates of the nearest neighbors; assigning the bilinear interpolation of the four integer coordinates of the nearest neighbors to the corresponding hierarchical feature pyramid, and finally obtaining the instance feature map of the aligned and extracted hierarchical feature pyramid; the feature map size for the cell nucleus instance bounding box optimization is 7×7, that is, N=7, and the feature map size for the instance mask optimization is 14×14, that is, N=14; In the third and fourth levels of the hierarchical feature pyramid, similarity global feature extraction is adopted; the specific method is: The center point coordinates of the bounding box corresponding to each level of the hierarchical feature pyramid (x c ,y c ), select each level feature pyramid and the center point coordinates (x c ,y c ) whose cosine similarity is greater than the first threshold is averaged, and the obtained mean is copied to the size of the corresponding bounding box optimized feature map and the instance mask feature map through a copy operation to obtain the extracted instance feature map; The extracted instance feature maps are spliced ​​together as the mixed features of the cell nucleus instance bounding box.

7. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 6, characterized in that: The step S600 specifically includes: Inputting the mixed features of the cell nucleus instance bounding box into a first convolution optimization module to generate a first optimized bounding box, and performing the operation of step S500 on the first optimized bounding box to obtain the mixed features of the first cell nucleus instance bounding box; Inputting the first instance bounding box hybrid feature into a second convolution optimization module to generate a second optimized bounding box, and performing the operation of step S500 on the second optimized bounding box to obtain a second instance bounding box hybrid feature of the cell nucleus; Inputting the second instance bounding box hybrid feature into a third convolution optimization module to generate a third optimized bounding box, performing the operation of step S500 on the third optimized bounding box to obtain a third instance bounding box hybrid feature of the cell nucleus, and outputting a third instance mask of the cell nucleus; The first convolution optimization module, the second convolution optimization module and the third convolution optimization module are each composed of two common convolution layers, one convolution module for predicting a bounding box and one convolution module for predicting an instance mask.

8. The method for digital pathology image cell nucleus instance segmentation based on hierarchical cascade network according to claim 7, characterized in that: The step S700 specifically includes: S701, for the third instance mask of the cell nucleus, starting from the mask with the highest probability, calculate the intersection-and-union ratio (IoU) of the mask with all masks except itself, and remove all masks whose IoU ratio with the mask with the highest probability is greater than a first IoU threshold; S702, repeating the process of S701 until there is no third instance mask of the cell nucleus whose IoU ratio with the mask with the highest probability is greater than the first IoU ratio threshold; S703 , the remaining third instance mask of the cell nucleus is output as the cell nucleus instance segmentation result.

Images