Digital image processing method and system

Abstract

The invention relates to an immunohistochemical image processing technology, and discloses a digital image processing method and system, and the method comprises the steps: decomposing an immunohistochemical digital image: carrying out the decomposition of the immunohistochemical digital image through the orthogonal normalization of optical density; performing cell segmentation of the digital image, performing automatic instance segmentation of a cell nucleus on the decomposed immunohistochemical digital image through a deep learning architecture Cell ViT of a visual Transform, and performing cell membrane segmentation of the digital image and cytoplasm segmentation of the digital image through a seed region growth algorithm Region Growing; according to the quantitative expression of the protein, the cell segmentation result of the digital image is combined with the decomposed immunohistochemical image, so that the quantification of the protein expression is realized. The digital image processing method adopted by the invention has excellent performance in the aspects of accuracy and recall rate, and has higher accuracy in the aspect of specific quantitative indexes.

Description

Technical Field

[0001] The present invention relates to immunohistochemical image processing technology, and particularly to a digital image processing method and system. Background Art

[0002] Accurately identifying and quantifying tumor markers is a crucial step in modern medical research, and has irreplaceable value for tumor diagnosis, selection of precise treatment plans, and prognosis assessment. As a detection technology based on highly specific antigen-antibody reactions, immunohistochemistry (IHC) has become an indispensable tool in the field of tumor marker identification because it can intuitively reveal the distribution and expression levels of proteins in cells and tissues. IHC plays a key role in many important aspects such as clinical pathological diagnosis, tumor classification, and prognosis judgment, providing valuable diagnostic information and treatment guidance for doctors.

[0003] However, traditional IHC analysis methods have significant limitations. This technology relies heavily on pathologists' visual observation and manual evaluation, which is not only time-consuming and laborious, but also extremely vulnerable to subjective factors. It is impossible to perform a complete quantitative analysis of different intensities of markers in whole slides. There may be differences in the evaluation results of the same tissue section by different pathologists or even the same pathologist at different times, which undoubtedly reduces the accuracy and consistency of the analysis results. This subjectivity and inconsistency not only affect the accuracy of diagnosis, but may also mislead the formulation of treatment plans, further affecting the treatment effect and prognosis of patients.

[0004] To solve this problem, the rapid development of deep learning (DL) technology in recent years has brought new breakthroughs to the field of medical image processing. DL algorithms, especially advanced image recognition and processing technologies such as convolutional neural networks (CNNs), have demonstrated excellent capabilities in image segmentation and recognition. By training on a large amount of pathological image data, DL models can automatically and accurately identify and segment tumor tissues and normal tissues, and further accurately identify the expression of specific tumor markers. The application of DL has greatly improved the accuracy and efficiency of IHC analysis, providing a more objective and reliable means for the identification of tumor markers. In terms of quantitative analysis, DL algorithms also show significant advantages. Traditional IHC methods rely on manual evaluation of staining intensity and distribution, which is not only time-consuming but also highly subjective, making it difficult to ensure the consistency of results.

[0005] As in the prior art 1: CN201910506460.4; a method, device and readable storage medium for cell segmentation, which separates the grayscale images of the DAB channel and the hematoxylin channel from the initial pathological staining image, performs image processing on the hematoxylin channel grayscale image according to the DAB channel grayscale image to obtain the processed hematoxylin channel grayscale image, and finally performs cell segmentation on the processed hematoxylin channel grayscale image to obtain the corresponding cell segmentation image.

[0006] As in the prior art 2: CN202010027735.9, which discloses a method for hierarchical segmentation of hematoxylin-eosin stained pathological images. According to the color intensity information of the pixels in the pathological image, the original image is preprocessed and feature selected, and hierarchical segmentation in three steps of K-means clustering, naive Bayes classification and watershed segmentation is gradually performed to obtain the precise boundary between cell nuclei.

[0007] The prior art only identifies cell types, but cannot quantify specific types of cells and can only calculate the total number of cells. Summary of the Invention

[0008] In view of the problems in the prior art that manual evaluation of staining intensity and distribution is not only time-consuming but also highly subjective, it is difficult to ensure the consistency of results, and it is impossible to specifically quantify specific types of cells, the present invention provides a digital image processing method and system.

[0009] To solve the above technical problems, the present invention is solved by the following technical solutions: A digital image processing method applied to quantitative analysis of staining intensity in immunohistochemistry, the method comprising: Decomposition of immunohistochemical digital images, which is performed by orthogonal normalization of optical density; Cell segmentation of digital images. For the decomposed immunohistochemical digital images, automatic instance segmentation of cell nuclei is performed through the deep learning architecture CellViT of vision transformers, and cell membrane segmentation and cytoplasm segmentation of digital images are performed through the seed region growth algorithm RegionGrowing. Quantitative expression of proteins, which is achieved by combining the cell segmentation results of digital images with the decomposed immunohistochemical images to realize quantitative analysis of protein expression.

[0010] Preferably, in the decomposition of the immunohistochemical digital images, through orthogonal normalization of optical density, separation and measurement of the concentration of each dye are realized. After the immunohistochemical digital images are stained and decomposed, the immunohistochemical digital images are decomposed into sub-images of hematoxylin and sub-images of DAB staining.

[0011] Preferably, the immunohistochemical digital image is decomposed into a hematoxylin-stained sub-image and a DAB-stained sub-image, including: cropping the immunohistochemical digital image into small image patches with the same physical field of view as the level 0 digital image, and adjusting the pixels of the cropped small image.

[0012] Preferably, the nucleus segmentation of the digital image takes the hematoxylin-stained sub-image as the input of CellViT and outputs the nucleus contour of the digital image, and each nucleus contour of the digital image represents an independent nucleus instance, containing the outer boundary coordinate information of the nucleus in the digital image.

[0013] Preferably, the cell membrane segmentation and cytoplasm segmentation of the digital image include: using the nucleus of the segmented digital image as the initial seed, and performing multiple morphological dilation iterations on the initial seed pixels, so as to realize the cell membrane segmentation and cytoplasm segmentation of the digital image.

[0014] Preferably, the expression of protein quantification is achieved by combining the cell segmentation result of the digital image with the decomposed DAB-stained sub-image; the combination method is to perform a bitwise AND operation on the cell segmentation result of the digital image and the decomposed DAB-stained sub-image, so as to obtain the gray-scale information of DAB staining of a single cell, and thus determine the protein quantification expression.

[0015] To solve the above technical problems, the present invention also provides a digital image processing system, which is applied to the quantitative analysis of staining intensity in immunohistochemistry, and includes: A decomposition module for immunohistochemical digital images, which decomposes immunohistochemical digital images through orthogonal normalization of optical density; A cell segmentation module for digital images, which performs automatic instance segmentation of nuclei on the decomposed immunohistochemical digital images through the deep learning architecture CellViT of Vision Transformer, and performs cell membrane segmentation and cytoplasm segmentation of digital images through the seed region growth algorithm Region Growing; An expression module for protein quantification, which realizes the quantification of protein expression by combining the cell segmentation result of the digital image with the decomposed immunohistochemical image.

[0016] To solve the above technical problems, the present invention also provides a computer program product, which, when running on a computer, executes the above method.

[0017] To solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and the program is configured to implement the above method when executed.

[0018] Due to the adoption of the above technical solutions, the present invention has remarkable technical effects: Through the intelligent analysis of the staining characteristics in the image, the present invention realizes the precise quantification of the expression of tumor markers. This intelligent analysis method not only improves the objectivity of the analysis, reduces the influence of human factors, but also provides more accurate and reliable data support for tumor grading, staging and prognosis evaluation. This is of great significance for formulating personalized treatment plans, improving treatment effects and improving the prognosis of patients.

[0019] The present invention effectively extracts high-level and low-level features by adopting the CellViT architecture, improving the segmentation accuracy. Brief Description of the Drawings

[0020] Figure 1 is the flow schematic diagram of the present invention.

[0021] Figure 2 is the CellViT processing flow chart of the present invention.

[0022] Figure 3 is the Region Growing segmentation flow chart of the present invention.

[0023] Figure 4 is the effect diagram of nuclear staining and membrane staining of the present invention. Detailed Description of the Invention

[0024] The present invention will be further described in detail below with reference to the drawings and embodiments. Embodiment

[0025] Combined with Figures 1-4 , in IHC staining, common dyes include hematoxylin and DAB (H-DAB). Hematoxylin is usually used to stain cell nuclei, showing blue, while DAB is used to label target antigens, showing brown-yellow. The combined use of these two dyes enables the clear observation of cell structures and the distribution of target antigens under a microscope. Through the orthogonal normalization of optical density, the separation and measurement of the concentration of each dye are achieved. After color decomposition processing, the IHC digital image is separated into a hematoxylin digital sub-image and a DAB-stained digital sub-image.

[0026] Since hematoxylin mainly stains the nucleus digital images, the hematoxylin-stained channel images obtained by color normalization are processed to avoid the influence of DAB staining on the nuclear segmentation results. StarDist is used for nuclear segmentation. Due to the huge size of WSI and the limited computer graphics memory, deep learning algorithms cannot segment the nuclei in the entire WSI at one time. The WSI needs to be cropped into multiple tiles. In addition, the resolution of the input image affects the results of StarDist, which refers to the physical size represented by each pixel in the WSI image in the real world. Taking the Hamamatsu pathological slide scanner NanoZoomer S360 as an example, the scanning resolution at 40X magnification is 0.23um / pixel. Therefore, in our process, the WSI is cropped into tiles with the same physical field of view (819.2×819.2um 2 ) as the level 0 image, and then uniformly resized to 512×512 pixels. This tile is separated into two sub-images stained with hematoxylin and DAB respectively after color decomposition. The sub-image stained with hematoxylin is used as the input of StarDist, and then the outlines of the nuclei in the output image are obtained. Each outline represents an independent nuclear instance, containing the outer boundary coordinate information of the nucleus in the image.

[0027] The CellViT algorithm uses a deep learning architecture based on vision transformers during training for automated instance segmentation of nuclei in digitized tissue samples. This method was trained and evaluated on the PanNuke dataset, a challenging dataset for nuclear instance segmentation that contains nearly 200,000 annotated nuclei in 19 tissue types, divided into 5 clinically important categories.

[0028] The main component in the nucleus is DNA, which has a double helix structure. The phosphate groups on the two strands face outwards, carrying negative charges and acidic properties, making it easy to bind to the positively charged hematoxylin staining solution, thus staining the nucleus. In contrast, the cytoplasm has a certain alkalinity and has a strong affinity for acidic dyes, so it is difficult to bind to hematoxylin, resulting in the cytoplasm not being stained. Cytoplasmic segmentation is performed using the seeded region growing algorithm, with the nucleus as the initial seed. These seeds are independent of each other. By performing multiple morphological dilation iterations on the seed pixels, the cell membrane and cytoplasm are generated. The refined pseudocode of the region growing algorithm is as follows: Figure 3Procedure for the region growing algorithm: Region Growing 1: Input: Image (pixel matrix), seed point set, similarity criterion function, growth stopping condition function 2: Output: Segmented image 3: For each seed point: 4: Add the seed point to the queue / stack and mark its position in the label matrix with the label of the current region (e.g., an integer incrementing from 0). 5: While the queue is not empty: 6: Dequeue a pixel (referred to as the current pixel) from the queue / stack. 7: For each neighboring pixel of the current pixel (based on 4 - connectivity or 8 - connectivity): 8: If the neighboring pixel is unlabeled in the label matrix (i.e., -1): 9: Calculate the similarity between the current pixel and the neighboring pixel using the similarity criterion function. 10: If the similarity satisfies the growth criterion (i.e., the similarity criterion function returns true): 11: Add the neighboring pixel to the queue / stack. 12: Mark the position of the neighboring pixel in the label matrix with the label of the current region. 13: End if 14: End if 15: End for 16: If the neighboring pixel is already labeled but the label is different from that of the current pixel (indicating possible region merging, which may not be considered in some applications): 17: Process region merging as needed, e.g., update the label matrix to merge region labels. 18: End if 19: End while 20: End for 21: Return the label matrix as the segmented image.

[0029] Quantification of protein expression can be achieved by combining the segmentation results with the DAB channel separated by deconvolution staining. A bit - wise AND operation is performed between the segmentation result and the DAB staining channel to obtain the gray - scale information of the DAB staining for each single cell. We recorded the minimum value, maximum value, and peak value of the DAB staining. Finally, based on these three parameters, the expression intensities of the nucleus, membrane, and cytoplasm are quantified as a fraction.

[0030] An image analysis software was developed based on C++ and OpenSlide (version 2.1). OpenSlide is an open - source WSI processing library. The algorithm is implemented using PyTorch and deployed using PyInstaller. The software adds a threshold option. For samples that require evaluating staining intensity, users can customize and set multiple thresholds to distinguish antibody expression levels from weak to strong, and cells with different intensities are marked with points of different colors on the software.

[0031] During IHC interpretation, pathologists examine stained tissue sections under a microscope. For staining including nuclear and membrane expression, 16 sections were selected for each staining method. These tissues were from the Pathology Diagnosis Center Co., Ltd. of Guangzhou Huitong Store No. 1, and all tissues were from surgical resection. Pathologists evaluate various aspects of the staining, such as the intensity, pattern, and location of the staining, to obtain meaningful information about tissue biology and pathology. Then, these sections were scanned into WSIs and interpreted twice by the same pathologist with the assistance of our image analysis software. Figure 3 shows the results after manual section interpretation. The lower right image in the upper half of the image is a 4x image and a local magnified image of the nuclear staining image, and the lower right image in the lower half of the image is a 4x image and a local magnified image of the membrane staining image. Red marks are strongly positive, yellow marks are moderately positive, pink marks are weakly positive, green marks are negative, and unmarked cells are non-tumor cells.

Claims

1. A digital image processing method for quantitative analysis of staining intensity in immunohistochemistry, comprising: Decomposition of immunohistochemical digital images, decomposition of immunohistochemical digital images by orthogonal normalization of optical density; Cell segmentation of digital images: For decomposed immunohistochemical digital images, CellViT, a deep learning architecture of visual transformer, is used to perform automatic instance segmentation of cell nuclei. The seed region growing algorithm Region Growing is used to perform cell membrane segmentation of digital images and cytoplasm segmentation of digital images. The quantitative expression of proteins is achieved by combining the cell segmentation results of digital images with the decomposed immunohistochemical images.

2. A digital image processing method according to claim 1, characterized in that: The decomposition of the immunohistochemical digital image is achieved by orthogonal normalization of optical density to separate and measure the concentration of each dye. After the immunohistochemical digital image is decomposed by staining, the immunohistochemical digital image is decomposed into a hematoxylin sub-image and a DAB staining sub-image.

3. A digital image processing method according to claim 2, characterized in that: The immunohistochemical digital image is decomposed into a hematoxylin-stained sub-image and a DAB-stained sub-image, including: cropping the immunohistochemical digital image into a small image patch with the same physical field of view as the level 0 digital image, and adjusting the pixels of the cropped small image.

4. A digital image processing method according to claim 3, characterized in that: The cell nucleus segmentation of the digital image is to use the hematoxylin-stained sub-image as the input of CellViT and output the cell nucleus contour of the digital image, and each cell nucleus contour of the digital image represents an independent cell nucleus instance, including the outer boundary coordinate information of the cell nucleus in the digital image.

5. A digital image processing method according to claim 1, characterized in that: The cell membrane segmentation of the digital image and the cytoplasm segmentation of the digital image include: using the cell nucleus of the segmented digital image as an initialization seed, and performing multiple morphological expansion iterations on the initialization seed pixels, thereby achieving cell membrane segmentation of the digital image and cytoplasm segmentation of the digital image.

6. A digital image processing method according to claim 5, characterized in that: The quantitative expression of the protein is achieved by combining the cell segmentation result of the digital image with the decomposed DAB-stained sub-image; the combination method is to perform a bitwise AND operation on the cell segmentation result of the digital image and the decomposed DAB-stained sub-image, thereby obtaining the grayscale information of DAB staining of a single cell, thereby determining the quantitative expression of the protein.

7. A digital image processing system for quantitative analysis of staining intensity in immunohistochemistry, characterized in that: include: The immunohistochemical digital image decomposition module decomposes the immunohistochemical digital image by orthogonal normalization of optical density; The cell segmentation module of digital images uses the deep learning architecture CellViT of the visual Transformer to perform automatic instance segmentation of cell nuclei for the decomposed immunohistochemical digital images, and uses the seed region growing algorithm RegionGrowing to perform cell membrane segmentation of digital images and cytoplasm segmentation of digital images; The protein quantitative expression module realizes the quantification of protein expression by combining the cell segmentation results of digital images with the decomposed immunohistochemical images.

8. A computer program product, characterized in that When running on a computer, the digital image processing method is executed.

9. A computer-readable storage medium having a computer program stored thereon, the program being configured to implement the digital image processing method described above when executed.

Images