A method for automated cell annotation of H&E pathological images using multiplex immunofluorescence

Abstract

The application discloses a method for automatically marking cells of H&E pathological images by using multiplex immunofluorescence, which comprises the following steps: S1, data acquisition and pretreatment; S2, image registration; S3, cell type marking based on rules. The method can register multiplex immunofluorescence images and H&E images of the same tissue, and automatically mark cells of the H&E images by using multiplex immunofluorescence. The method can automatically generate a large number of cell markings without human intervention, and the cell types that can be marked are very extensive.

Description

Technical Field

[0001] This invention relates to a medical image processing technique, and more particularly to a method for automatic cell annotation of H&E pathological images using multiplex immunofluorescence. Background Technology

[0002] Cellular component analysis on H&E-stained pathological images is an important task with many potential applications in cancer diagnosis and personalized treatment. However, due to the large number of cells, cellular component analysis is very difficult. With the development of artificial intelligence technology, some methods using deep learning for automated cell identification have emerged and achieved good results. However, deep learning technology requires a large amount of data annotation to function, which is a significant problem in cell identification tasks. Traditional cell annotation methods involve pathologists manually delineating cells on pathological images, which is very time-consuming and labor-intensive. This approach can only label a limited number of cells, making it difficult to meet the data volume requirements of deep learning. To address this problem, this invention proposes a method that can automatically annotate cells in H&E images using multiplex immunofluorescence. Summary of the Invention

[0003] The purpose of this invention is to provide a method for automatic cell annotation of Hematologic and Epithelial Process (H&E) pathological images using multiplex immunofluorescence. This method can register multiplex immunofluorescence images of the same tissue with H&E images, automatically delineating cell types and boundaries on the H&E images. This invention can automatically generate a large number of H&E cell annotations without manual intervention, and can label a wide range of cell types, thereby promoting research in many downstream AI-based pathological image analysis tasks.

[0004] To achieve the above objectives, the present invention provides a method for automatic cell annotation of H&E pathological images using multiplex immunofluorescence, comprising the following steps:

[0005] S1. Data Acquisition and Preprocessing

[0006] Collect H&E and multiple immunofluorescence staining images corresponding to tumor tissues and perform preprocessing, including downsampling to the same pixel size, artifact removal, and grayscale normalization, to obtain at least one pair of low-resolution H&E and immunofluorescence image pairs and at least one pair of high-resolution image pairs.

[0007] S2, Image Registration

[0008] For the H&E and immunofluorescence image pairs obtained after preprocessing in S1, global registration is first performed using low-resolution H&E and immunofluorescence image pairs, and then local registration is performed using high-resolution H&E and immunofluorescence image pairs, so that the two images achieve fine matching at the cell level.

[0009] S3, Rule-based cell type labeling

[0010] The average gray value of cell nuclei in each channel of the immunofluorescence image was calculated in H&E, and cell types were labeled with multiple tags based on rules.

[0011] In a preferred embodiment of the present invention, the multiple immunofluorescence staining refers to performing multiple immunofluorescence staining on tumor tissue, wherein the markers include at least two or all of KI67, PANCK, CD3, CD20, CD21, CD23 and DAPI.

[0012] In a preferred embodiment of the present invention, the rule refers to:

[0013] (1) KI67 and DAPI are unrestricted and can coexist with any other cell type. A cell can be assigned to either of these two categories as long as it is larger than the corresponding threshold.

[0014] (2) As long as PANCK is greater than the threshold, the cell will be assigned as PANCK and will not belong to CD3, CD20, CD21 or CD23.

[0015] (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold, the cell with the higher average gray value is selected as the cell category. If only CD3 is greater than the threshold, the cell belongs to the CD3 category. If only CD20 is greater than the threshold, the cell belongs to the CD20 category.

[0016] (4) For CD23 and CD21, CD23 can only appear on the basis of CD21.

[0017] In a preferred embodiment of the present invention, during global registration, the H&E image is used as a fixed image and the DAPI channel of the immunofluorescence image is used as a moving image to calculate the transformation required for global registration.

[0018] In a preferred embodiment of the present invention, during local registration, the high-resolution H&E image and the immunofluorescence image are segmented into image blocks of size 1024*1024. Then, cell nucleus identification is performed on the H&E image to convert it into a cell nucleus segmentation map, and image blocks with fewer than 5 cells are discarded. For the immunofluorescence image, all channels are summed to form a grayscale image, called a channel summation map. Subsequently, the cell nucleus segmentation map of H&E is used as a fixed image, and the channel summation map of immunofluorescence is used as a moving image to calculate the transformation required for local registration. Then, the obtained transformation is applied to each channel of the immunofluorescence image to complete the local registration process. Finally, the mutual information gain is obtained by subtracting the mutual information before registration from the mutual information after registration, and image blocks with a mutual information gain lower than 0.001 are filtered out for quality control.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is the overall flowchart;

[0021] Figure 2 A schematic diagram of global image registration;

[0022] Figure 3 A schematic diagram of local image registration;

[0023] Figure 4 A schematic diagram of H&E cell annotation results. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0025] Figure 1 This is an overall flowchart of a method for automatic cell annotation of H&E pathological images using multiplex immunofluorescence, according to an embodiment of the present invention. The present invention includes the following steps:

[0026] S1. Data Acquisition and Preprocessing

[0027] To perform automated cell labeling, it is necessary to collect H&E staining images and immunofluorescence staining images corresponding to the same tumor tissue sections. First, the tumor tissue was subjected to multiplex immunofluorescence staining using seven markers: KI67, PANCK, CD3, CD20, CD21, CD23, and DAPI. (It should be noted that this method is theoretically applicable to all immunofluorescence markers, not just the examples given here.) The immunofluorescent sections were scanned at 10X magnification, resulting in an image pixel size of 0.3461 μm / px. Subsequently, the same section was directly stained with H&E (hematoxylin and eosin) and scanned at 40X magnification, resulting in an image pixel size of 0.1907 μm / px. No destaining was necessary here because the interaction between the H&E reagent and the fluorescent group is minimal. Therefore, we obtained the H&E image and the corresponding multiplex immunofluorescence image of the tumor tissue.

[0028] Next, some preprocessing operations are needed on the H&E and immunofluorescence images. We need to obtain a pair of low-resolution H&E and immunofluorescence images and a pair of high-resolution images for subsequent image registration steps. The low-resolution image pair is used for global coarse registration, and the high-resolution image pair is used for local fine registration. In a pair of images, the pixel size of the H&E and immunofluorescence images should be consistent, i.e., the actual length corresponding to each pixel is the same, to facilitate subsequent registration operations. First, the original H&E image is downsampled by 64 times, and the immunofluorescence image is downsampled to make its pixel size the same as the H&E image, to generate a low-resolution image pair. Then, the original H&E image is downsampled by 2 times, and the immunofluorescence image is downsampled to achieve the same pixel size as the H&E image, to generate a high-resolution image pair. The recommended downsampling factor is that for the low-resolution image pair, the downsampled image size should be approximately between 1000 and 2000 pixels; for the high-resolution image pair, the larger the pixel size, the better, but it is not recommended to exceed the pixel size of the original immunofluorescence image. To remove artifacts from immunofluorescence images, grayscale truncation is required. For each channel of the immunofluorescence image, the grayscale values ​​of each pixel are sorted from smallest to largest. To speed up computation, this operation is performed after downsampling the original immunofluorescence image by a factor of 8. Empirically, the 90th percentile grayscale value is selected as the lower bound, and the 99.5th percentile grayscale value is selected as the upper bound for pixel truncation. It should be noted that for low-resolution images, pixel truncation is only performed at the upper bound. Finally, since the grayscale values ​​vary significantly between immunofluorescence images, ranging from hundreds to thousands, grayscale normalization is performed to make the grayscale values ​​of each image fall into the same range, facilitating the use of the same threshold for processing each image in the subsequent S3 step. For each channel, maximum-minimum normalization is performed, followed by multiplication by 255, thereby converting each channel of the immunofluorescence image into an 8-bit grayscale image, with the grayscale value of each pixel ranging from 0 to 255.

[0029] S2, Image Registration

[0030] Based on step S1, we obtained preprocessed H&E and immunofluorescence image pairs. Next, registration is required to achieve cellular-level matching between the H&E and immunofluorescence images. First, as... Figure 2As shown, global registration is performed on low-resolution image pairs to achieve a coarse tissue-level alignment between H&E and immunofluorescence. During global registration, the H&E image is used as the stationary image, and the DAPI channels of the immunofluorescence image are used as the moving image (because the DAPI channels label almost all cells) to calculate the transformation required for global registration. Then, the resulting transformation is applied to all channels of the immunofluorescence image to complete the global registration process. The registration method uses affine transformation, and the registration metric is the Pearson correlation coefficient. Image registration is implemented using the SimpleITK library in Python. Furthermore, the mutual information between the image pairs before and after registration is recorded for subsequent quality control.

[0031] After global registration is completed, local registration is also required to achieve fine alignment at the cell level between H&E and immunofluorescence. The registration process is as follows: Figure 3 As shown, high-resolution H&E and immunofluorescence images were segmented into 1024*1024 image patches. Then, cell nucleus identification was performed on the H&E image, converting it into a cell nucleus segmentation map, and image patches with fewer than 5 cells were discarded. Cell nucleus identification used the HoverNet model from Python's histocartography library. For the immunofluorescence image, each channel was summed to form a grayscale image, called a channel summation map, to improve the insufficient brightness of some cells in the DAPI channel. Subsequently, the H&E cell nucleus segmentation map was used as a stationary image, and the immunofluorescence channel summation map was used as a moving image to calculate the transformation required for local registration, using the same method as global registration. Then, the obtained transformation was applied to each channel of the immunofluorescence image to complete the local registration process. Finally, the mutual information gain was obtained by subtracting the mutual information before registration from the mutual information after registration, and image patches with a mutual information gain lower than 0.001 were filtered out for quality control.

[0032] S3, Rule-based cell type labeling

[0033] For the registered H&E cell nucleus segmentation map and immunofluorescence image, the average gray value of each cell nucleus in each channel of the immunofluorescence is calculated. For a certain type of cell, the cell nucleus with a higher average gray value is very likely to belong to that type of cell. Next, it is necessary to determine the cell nucleus type. However, one problem is that cells in the immunofluorescence are not necessarily of only one type. A cell can express multiple markers simultaneously, such as CD21 and CD23. Therefore, cell type labeling is actually a multi-label problem. So, we have formulated a rule to determine the cell category, which includes two steps. First, a threshold is determined for each channel. For each channel, cells with an average gray value greater than the threshold are selected as potential target cells. Here, we empirically selected 50 as the threshold for all channels. Then, some constraints are needed to guide the classification of cell types and avoid cell labeling results that do not conform to common sense. Taking the seven cell types given in this invention as an example, the following rules are formulated:

[0034] (1) KI67 and DAPI are unrestricted and can coexist with any other cell type. A cell can be assigned to either of these two categories as long as it is larger than the corresponding threshold.

[0035] (2) As long as PANCK is greater than the threshold, the cell will be assigned as PANCK and will not belong to CD3, CD20, CD21 or CD23.

[0036] (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold, the cell with the higher average gray value is selected as the cell category. If only CD3 is greater than the threshold, the cell belongs to the CD3 category. If only CD20 is greater than the threshold, the cell belongs to the CD20 category.

[0037] (4) For CD23 and CD21, CD23 can only appear on the basis of CD21.

[0038] The above rules are the guidelines to follow when assigning cell nucleus types. It's important to note that in a multi-label setting, a cell can belong to multiple categories, but the above rules must still apply. At this point, we have obtained the categories of each cell in H&E, and combined with the previously performed nucleus segmentation, we can obtain the boundaries of each cell nucleus. Figure 4 The annotation results for CD21, CD20, and CD3 cells obtained based on this invention are presented. The quality of the annotation results can be judged by comparing the distribution of cell types labeled by this method with the distribution of cell types on immunofluorescence images. In this way, the thresholds for each type of cell can be adjusted to improve the accuracy of the annotation results.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method utilizing multiplex immunofluorescence to... A method for automatic cell annotation of pathological images, characterized by: Includes the following steps: S1. Data Acquisition and Preprocessing Collect the same tumor tissue slices The staining images and immunofluorescence staining images were obtained by first performing multiple immunofluorescence staining on the tumor tissue, followed by H&E staining directly on the same slide without destaining, thus obtaining the tumor tissue images. Images and corresponding multiplex immunofluorescence images were obtained; preprocessing was performed, including downsampling to the same pixel size, artifact removal, and grayscale normalization, to obtain at least one pair of low-resolution images. and immunofluorescence image pairs and at least one pair of high-resolution image pairs; S2, Image Registration The result after preprocessing in S1 The H&E and immunofluorescence image pairs were first globally registered using low-resolution H&E and immunofluorescence image pairs, and then registered using high-resolution H&E and immunofluorescence image pairs. Local registration was performed between the two images and the immunofluorescence image to achieve a fine match at the cell level. During global registration, The image is used as a stationary image, and the DAPI channel of the immunofluorescence image is used as a moving image to calculate the transformation required for global registration; During local registration, high-resolution images will be used. Images and immunofluorescence images are sliced ​​into sizes of Image patches, and then... The image undergoes nucleus identification and is converted into a nucleus segmentation map, discarding image patches with fewer than 5 cells. For immunofluorescence images, all channels are summed to form a grayscale image, called a channel summation map. Subsequently, the nucleus segmentation map from H&E is used as a stationary image, and the channel summation map from immunofluorescence is used as a moving image to calculate the transformation required for local registration. Then, the obtained transformation is applied to each channel of the immunofluorescence image to complete the local registration process. Finally, the mutual information gain is obtained by subtracting the mutual information before registration from the mutual information after registration, and image patches with a mutual information gain lower than 0.001 are filtered out for quality control. S3, Rule-based cell type labeling calculate The average gray value of cell nuclei in each channel of the immunofluorescence image is used to identify cell nuclei with higher average gray values ​​for a certain cell type. Cell types are then labeled with multiple tags based on rules. The multiple immunofluorescence staining refers to multiple immunofluorescence staining of tumor tissue, with markers including KI67, PANCK, CD3, CD20, CD21, CD23 and DAPI; The rules mentioned refer to: (1) KI67 and DAPI are unrestricted and can coexist with any other cell type. A cell can be assigned to these two categories as long as it is larger than the corresponding threshold. (2) As long as PANCK is greater than the threshold, the cell will be assigned as PANCK and will not belong to CD3, CD20, CD21 or CD23; (3) CD3 and CD20 cells are mutually exclusive types. If both are greater than the threshold, the cell with the higher average gray value is selected as the cell category. If only CD3 is greater than the threshold, the cell belongs to the CD3 category. If only CD20 is greater than the threshold, the cell belongs to the CD20 category. (4) For CD23 and CD21, CD23 can only appear on the basis of CD21.

Images