Method for carrying out automatic cell labeling on HE pathological image by utilizing multiple immunofluorescence

Automatic cell annotation of H&E pathological images through multiple immunofluorescence technology, solving the problem of time-consuming and labor-intensive traditional manual annotation, achieving efficient cell annotation and accurate cell type recognition, and promoting the application of artificial intelligence in pathological image analysis.

CN118918150AActive Publication Date: 2024-11-08BEIJING SHIJITAN HOSPITAL CAPITAL MEDICAL UNIVERSITY +1

Patent Information

Application Number
CN202410706134.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-03
Publication Date
2024-11-08
Estimated Expiration
2044-06-03

AI Technical Summary

Technical Problem

In cell component analysis, traditional pathologists' manual cell outline method is time-consuming and laborious and difficult to meet the data volume requirements of deep learning, resulting in inefficient cell labeling.

Method used

Automatic cell annotation of H&E pathological images through multiple immunofluorescence technology, including data acquisition and preprocessing, image registration and rule-based cell type annotation. Multiple immunofluorescence staining images and H&E images are registered to automatically outline the cell type and Boundaries, generate a large number of cell annotations without artificial involvement.

Benefits of technology

Automatic cell annotation is realized, the efficiency and accuracy of cell annotation is improved, the requirements of deep learning for data volume are met, and the research on pathological image analysis tasks based on artificial intelligence is promoted.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918150A_ABST
    Figure CN118918150A_ABST
Patent Text Reader

Abstract

The invention discloses a method for detecting Hamp by utilizing multiple immunofluorescence. The invention discloses a method for carrying out automatic cell labeling on a pathological image. The method comprises the following steps: S1, data acquisition and preprocessing; s2, carrying out image registration; and S3, carrying out cell type labeling based on rules. According to the method disclosed by the invention, multiple immunofluorescence images of the same tissue can be combined with Hamp; e, registering the image, and automatically registering the Hamp by utilizing multiple immunofluorescence; and E, performing cell labeling on the image. According to the method, a large number of cell labels can be automatically generated, artificial participation is not needed, and the cell types capable of being labeled are very wide.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a medical image processing technology, and in particular to a method for automatically labeling cells in H&E pathological images using multiple immunofluorescence. Background Art

[0002] Performing cell component analysis on H&E stained pathological images is an important task, and it has many potential applications in cancer diagnosis and personalized treatment. However, due to the large number of cells, it is very difficult to perform cell component analysis. With the development of artificial intelligence technology, some methods have emerged that use deep learning for automatic cell recognition, and have achieved good results. However, deep learning technology requires a large amount of data annotation to work, which is a big problem in cell recognition tasks. The traditional cell annotation method is that pathologists manually outline cells on pathological images, which is very time-consuming and laborious. This approach can only mark a limited number of cells, and it is difficult to meet the data volume requirements of deep learning. In response to this problem, the present invention proposes a method that can automatically annotate cells in H&E images using multiple immunofluorescence. Summary of the invention

[0003] The purpose of the present invention is to provide a method for automatic cell annotation of H&E pathological images using multiple immunofluorescence, which can align multiple immunofluorescence images of the same tissue with H&E images and automatically outline the cell types and boundaries on the H&E images. The present invention can automatically generate a large number of H&E cell annotations without manual intervention, and the cell types that can be labeled are also very wide, thereby promoting the research of many downstream artificial intelligence-based pathological image analysis tasks.

[0004] To achieve the above object, the present invention provides a method for automatically annotating cells in H&E pathological images using multiple immunofluorescence, comprising the following steps:

[0005] S1. Data acquisition and preprocessing

[0006] Collecting H&E and multiple immunofluorescence staining images corresponding to tumor tissues and performing preprocessing, including downsampling to the same pixel size, removing artifacts, and normalizing grayscale values, to obtain at least one pair of low-resolution H&E and immunofluorescence images and at least one pair of high-resolution images;

[0007] S2. Image Registration

[0008] For the H&E and immunofluorescence image pairs obtained after preprocessing in S1, global registration is first performed through the low-resolution H&E and immunofluorescence image pairs, and then local registration is performed through the high-resolution H&E and immunofluorescence image pairs, so that the two images can achieve fine matching at the cellular level;

[0009] S3. Rule-based cell type annotation

[0010] Calculate the average gray value of the cell nucleus on H&E in each channel of the immunofluorescence image, and annotate the cell type with multiple labels based on the rules.

[0011] In a preferred embodiment of the present invention, the multiple immunofluorescence staining refers to multiple immunofluorescence staining of tumor tissue, and the markers include at least two or all of KI67, PANCK, CD3, CD20, CD21, CD23 and DAPI.

[0012] In a preferred embodiment of the present invention, the rule refers to:

[0013] (1) KI67 and DAPI are unconstrained and can coexist with any other cell type. A cell can be assigned to these two categories as long as its expression is greater than the corresponding threshold;

[0014] (2) As long as PANCK is greater than the threshold, the cell is assigned to PANCK rather than CD3, CD20, CD21, or CD23;

[0015] (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold at the same time, the one with the higher average gray value is selected as the type of the cell; if only CD3 is greater than the threshold, the cell belongs to the CD3 type; if only CD20 is greater than the threshold, the cell belongs to the CD20 type;

[0016] (4) For CD23 and CD21, CD23 can only appear on the basis of the appearance of CD21.

[0017] In a preferred embodiment of the present invention, during global registration, the H&E image is used as a fixed image and the DAPI channel of the immunofluorescence image is used as a moving image to calculate the transformation required for global registration.

[0018] In a preferred embodiment of the present invention, during local registration, the high-resolution H&E image and the immunofluorescence image are cut into image blocks of size 1024*1024, and then the H&E image is subjected to cell nucleus identification, converted into a cell nucleus segmentation map, and the image blocks with less than 5 cells are discarded; for the immunofluorescence image, each channel is added up to form a grayscale image, called a channel summation map; then, the H&E cell nucleus segmentation map is used as a fixed image and the immunofluorescence channel summation map is used as a moving image to calculate the transformation required for local registration; then, the obtained transformation is applied to each channel of the immunofluorescence image to complete the local registration process; finally, the mutual information gain is obtained by subtracting the mutual information before registration from the mutual information after registration, and the image blocks with a mutual information gain lower than 0.001 are filtered out for quality control.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The following is the overall flow chart;

[0021] Figure 2 This is a schematic diagram of global image registration;

[0022] Figure 3 This is a schematic diagram of local image registration;

[0023] Figure 4 Schematic diagram of H&E cell annotation results. DETAILED DESCRIPTION

[0024] The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that this embodiment is based on the technical solution and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to this embodiment.

[0025] Figure 1 The present invention is an overall flow chart of a method for automatically annotating cells in H&E pathological images using multiple immunofluorescence. The present invention includes the following steps:

[0026] S1. Data acquisition and preprocessing

[0027] In order to perform automatic cell annotation, it is necessary to collect H&E staining images and immunofluorescence staining images corresponding to the same tumor tissue section. First, multiple immunofluorescence staining was performed on the tumor tissue. We used 7 markers, including KI67, PANCK, CD3, CD20, CD21, CD23 and DAPI. (It should be noted that this method is theoretically applicable to all immunofluorescence markers and is not limited to the few examples given here) and the immunofluorescence sections were scanned at a magnification of 10X, and the image pixel size was 0.3461um / px. Subsequently, H&E (hematoxylin and eosin) staining was performed directly on the same section and scanned at a magnification of 40X, and the image pixel size was 0.1907um / px. There is no need for decolorization here because the interaction between the H&E reagent and the fluorescent group is very small. Therefore, we obtained H&E images of tumor tissue and corresponding multiple immunofluorescence images.

[0028] Then, some preprocessing operations are required for the H&E and immunofluorescence images. We need to obtain a pair of low-resolution H&E and immunofluorescence image pairs and a pair of high-resolution image pairs for the subsequent image registration steps. Among them, the low-resolution image pair is used for global coarse registration, and the high-resolution image pair is used for local fine registration. In a pair of images, the pixel size of the H&E and immunofluorescence images should be consistent, that is, the real length corresponding to each pixel is the same, to facilitate the subsequent registration operation. First, the original H&E image is downsampled by 64 times, and the immunofluorescence image is downsampled to make its pixel size the same as the H&E image to generate a low-resolution image pair. Then, the original H&E image is downsampled by 2 times, and the immunofluorescence image is downsampled to the same pixel size as the H&E image to generate a high-resolution image pair. The recommendation for the downsampling multiple here is that for low-resolution image pairs, the image size after downsampling is about 1000 to 2000 pixels; for high-resolution image pairs, the larger the pixel size, the better, but it is not recommended to exceed the pixel size of the original immunofluorescence image. In order to remove the artifacts of immunofluorescence, a grayscale value truncation operation is required. For each channel of the immunofluorescence image, the grayscale values ​​of each pixel are sorted from small to large. In order to speed up the calculation, this operation is performed after the original immunofluorescence image is downsampled 8 times. According to experience, the 90th percentile grayscale value is selected as the lower bound, and the 99.5th percentile grayscale value is selected as the upper bound, and the pixel truncation operation is performed on the immunofluorescence. It should be noted that for low-resolution images, only the upper bound is pixel truncation. Finally, since the grayscale values ​​of each immunofluorescence image vary greatly, ranging from hundreds to thousands, grayscale value normalization processing is required to make the grayscale values ​​of each image into the same range, so that the same threshold is used to process each image in the subsequent S3 step. For each channel, maximum-minimum normalization is performed, and then multiplied by 255, so that each channel of the immunofluorescence image is converted into an 8-bit grayscale image, and the grayscale value range of each pixel is 0 to 255.

[0029] S2. Image Registration

[0030] Based on step S1, we obtain the preprocessed H&E and immunofluorescence image pairs. Next, we need to perform registration so that the H&E and immunofluorescence images can match at the cell level. First, Figure 2As shown in the figure, global registration is performed on low-resolution image pairs so that a rough alignment at the tissue level is achieved between H&E and immunofluorescence. During global registration, the H&E image is used as a fixed image, and the DAPI channel of the immunofluorescence image is used as a moving image (because the DAPI channel labels almost all cells) to calculate the transformation required for global registration. Then, the obtained transformation is applied to all channels of the immunofluorescence image to complete the global registration process. The registration method uses affine transformation, and the registration indicator uses the Pearson correlation coefficient. Image registration is implemented using the Python SimpleITK library. In addition, the mutual information between the image pairs before and after registration is recorded for subsequent quality control.

[0031] After completing the global registration, local registration is required to achieve fine alignment at the cellular level between H&E and immunofluorescence. The registration process is as follows: Figure 3 As shown. The high-resolution H&E image and immunofluorescence image were cut into image blocks of size 1024*1024, and then the H&E image was subjected to cell nucleus identification, which was converted into a cell nucleus segmentation map, and the image blocks with less than 5 cells were discarded. The cell nucleus identification used the HoverNet model in the histocartography library of python. For immunofluorescence images, each channel was added up to form a grayscale image, called the channel sum map, to improve the problem of insufficient brightness of some cells in the DAPI channel. Subsequently, the cell nucleus segmentation map of H&E was used as a fixed image and the channel sum map of immunofluorescence was used as a moving image to calculate the transformation required for local registration. The specific method is the same as that of global registration. Then, the obtained transformation is applied to each channel of the immunofluorescence image to complete the local registration process. Finally, the mutual information gain is obtained by subtracting the mutual information before registration from the mutual information after registration, and the image blocks with a mutual information gain lower than 0.001 are filtered out for quality control.

[0032] S3. Rule-based cell type annotation

[0033] For the aligned H&E cell nucleus segmentation map and immunofluorescence image, calculate the average gray value of each cell nucleus of H&E in each channel of immunofluorescence. For the channel corresponding to a certain type of cell, the cell nucleus with a relatively high average gray value is very likely to belong to this type of cell. Next, it is necessary to determine the type of cell nucleus, but there is a problem that the cell on immunofluorescence does not necessarily have only one type, and a cell can express multiple markers at the same time, such as CD21 and CD23. Therefore, the labeling of cell types is actually a multi-label problem. Therefore, we have formulated a rule to determine the category of cells, which includes two steps. First, a threshold is determined for each channel, and for each channel, cells with an average gray value greater than the threshold are screened out as potential target cells. Here we empirically select 50 as the threshold for each channel. Then, some constraints are needed to guide the classification of cell types to avoid unreasonable cell labeling results. Taking the 7 cell types given in the present invention as an example, the following rules are formulated:

[0034] (1) KI67 and DAPI are unconstrained and can coexist with any other cell type. A cell can be assigned to these two categories as long as its expression is greater than the corresponding threshold;

[0035] (2) As long as PANCK is greater than the threshold, the cell is assigned to PANCK rather than CD3, CD20, CD21, or CD23;

[0036] (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold at the same time, the one with the higher average gray value is selected as the type of the cell; if only CD3 is greater than the threshold, the cell belongs to the CD3 type; if only CD20 is greater than the threshold, the cell belongs to the CD20 type;

[0037] (4) For CD23 and CD21, CD23 can only appear on the basis of the appearance of CD21.

[0038] The above are the rules that need to be followed when assigning cell nuclei to types. It should be noted that in a multi-label setting, a cell can belong to multiple categories, but it must follow the constraints of the above rules. So far, we have obtained the categories of each cell on H&E, and combined with the cell nucleus segmentation that has been performed before, we can get the boundaries of each cell nucleus. Figure 4 The labeling results of three types of cells, CD21, CD20 and CD3, obtained based on the present invention are shown. The quality of the labeling results can be judged by comparing the distribution of cell types labeled by this method with the distribution of cell types on the immunofluorescence image, and then the thresholds of various types of cells can be adjusted to improve the accuracy of the labeling results.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatic cell annotation of H&E pathological images using multiple immunofluorescence. Features: The following steps are involved: S1. Data acquisition and preprocessing Collecting H&E and multiple immunofluorescence staining images corresponding to tumor tissues and performing preprocessing, including downsampling to the same pixel size, removing artifacts, and normalizing grayscale values, to obtain at least one pair of low-resolution H&E and immunofluorescence images and at least one pair of high-resolution images; S2. Image Registration For the H&E and immunofluorescence image pairs obtained after preprocessing in S1, global registration is first performed through the low-resolution H&E and immunofluorescence image pairs, and then local registration is performed through the high-resolution H&E and immunofluorescence image pairs, so that the two images can achieve fine matching at the cellular level; S3. Rule-based cell type annotation Calculate the average gray value of the cell nucleus on H&E in each channel of the immunofluorescence image, and annotate the cell type with multiple labels based on the rules.

2. According to the method of claim 1, the multiple immunofluorescence staining refers to multiple immunofluorescence staining of tumor tissue, and the markers include at least two or all of KI67, PANCK, CD3, CD20, CD21, CD23 and DAPI.

3. The rules according to the method of claim 2 are: (1) KI67 and DAPI are unconstrained and can coexist with any other cell type. A cell can be assigned to these two categories as long as its expression is greater than the corresponding threshold; (2) As long as PANCK is greater than the threshold, the cell is assigned to PANCK rather than CD3, CD20, CD21, or CD23; (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold at the same time, the one with the higher average gray value is selected as the type of the cell; if only CD3 is greater than the threshold, the cell belongs to the CD3 type; if only CD20 is greater than the threshold, the cell belongs to the CD20 type; (4) For CD23 and CD21, CD23 can only appear on the basis of the appearance of CD21.

4. According to the method of claim 1, during global registration, the H&E image is used as a fixed image and the DAPI channel of the immunofluorescence image is used as a moving image to calculate the transformation required for global registration.

5. According to the method of claim 1, during local registration, the high-resolution H&E image and the immunofluorescence image are cut into image blocks of size 1024*1024, and then the H&E image is subjected to cell nucleus identification and converted into a cell nucleus segmentation map, and the image blocks with less than 5 cells are discarded; for the immunofluorescence image, each channel is added up to form a grayscale image, called a channel summation map; then, the cell nucleus segmentation map of H&E is used as a fixed image and the channel summation map of immunofluorescence is used as a moving image to calculate the transformation required for local registration; then, the obtained transformation is applied to each channel of the immunofluorescence image to complete the local registration process; finally, the mutual information gain is obtained by subtracting the mutual information before registration from the mutual information after registration, and the image blocks with a mutual information gain lower than 0.001 are filtered out for quality control.

Citation Information

Patent Citations

  • Computer scoring based on primary stain and immunohistochemistry images

    CN110088804A

  • Tumor component labeling method, device and equipment and storage medium

    CN111180048A

  • Training method and device based on image annotation and segmentation model

    CN115546605A

  • TLS pathology detection method and device based on deep learning, and medium

    CN117670794A

  • Immunohistochemical pathological image cell segmentation method based on slice restaining technology

    CN117670895A

Cited By

  • Renal tubule segment classification method of pathological section staining image

    CN120707947A

  • A method for classifying renal tubular segments in pathological section staining images

    CN120707947B

  • Quantitative analysis method, system and equipment for cancerization region of tissue slice and medium

    CN121258982A

  • Method, system, device and medium for quantitative analysis of cancerous regions of tissue sections

    CN121258982B