Method for analyzing immunohistochemical staining images

Abstract

The application provides an immunohistochemical staining image analysis method, including: cutting a plurality of cell nuclei of an immunohistochemical staining image according to a machine learning model; removing pixels belonging to the cell nuclei in the immunohistochemical staining image and removing pixels meeting a color range to obtain a plurality of cytoplasm pixels; distributing the cytoplasm pixels to the cell nuclei according to the positions of the cytoplasm pixels to form a plurality of cells; and calculating a pixel staining score of each pixel in the cells, thereby calculating a cell staining score of the cells. Thus, the staining score of the cells can be objectively and automatically calculated.

Description

Technical Field

[0001] This invention relates to an analysis method for immunohistochemical staining images, which can use algorithms to objectively assess the degree of staining of each cell. Background Technology

[0002] Immunohistochemistry (IHC) is a widely used method for labeling specific organic substances. The principle lies in using a specific ratio of antigen and antibody to allow the specific antigen to be displayed in a whole-slide image (WSI) of a tissue section. IHC is frequently used in medical research to assess pathological parameters of stained cells and observe their relationship with diseases such as cancer. However, traditional analytical methods rely on the subjective judgment of physicians; therefore, developing objective analytical methods is a concern for professionals in this field. Summary of the Invention

[0003] This invention proposes an analysis method for immunohistochemical staining images, applicable to computer systems. This method includes: segmenting multiple cell nuclei from an immunohistochemical staining image using a machine learning model, wherein the immunohistochemical staining image includes multiple pixels, and each pixel includes multiple color channels; removing pixels belonging to cell nuclei and pixels conforming to a color range from the immunohistochemical staining image to obtain multiple cytoplasmic pixels; assigning the cytoplasmic pixels to one of the cell nuclei based on their positions to form multiple cells; and calculating the pixel staining score of the pixels within the cells, thereby calculating the cell staining score of the cells.

[0004] In some embodiments, the step of removing pixels that conform to a color range includes: converting the pixels to a color space, the color space including a hue, a saturation and a brightness; and retaining pixels whose hue is within a first preset range and whose saturation is within a second preset range, and removing pixels whose brightness is greater than a first threshold value.

[0005] In some embodiments, the step of assigning cytoplasmic pixels to cell nuclei to form cells based on the location of cytoplasmic pixels includes: treating cell nuclei as multiple low points and cytoplasmic pixels as water catchment areas, thereby performing a watershed algorithm.

[0006] In some embodiments, after forming cells, the analysis method further includes: calculating the number of pixels in the cells; and if the number of pixels in the first cell is greater than a second threshold and less than a third threshold, deleting cytoplasmic pixels in the first cell and adding the result of morphological expansion processing performed on the nucleus of the first cell to form the first cell.

[0007] In some embodiments, the step of calculating the pixel staining score of a pixel in a cell, thereby calculating the cell staining score of a cell, includes: calculating the pixel staining score of a pixel based on the ratio between the saturation and brightness of the pixel in the cell; and calculating a weighted average of the pixel staining scores of the pixels in the cell, using the number of pixels as a weight, as the cell staining score of the cell.

[0008] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an analysis method for immunohistochemical staining images, based on one embodiment.

[0010] Figure 2 This is a schematic diagram illustrating the acquisition of cytoplasmic pixels according to one embodiment.

[0011] Figure 3 This is a schematic diagram illustrating the result of the watershed algorithm based on one embodiment.

[0012] Figure 4 This is a schematic diagram illustrating the calculation of pixel chromaticity based on the HSV color space according to one embodiment.

[0013] In the attached figures, the following labels are used:

[0014] 101, 102, 201, 301: Immunohistochemical staining images

[0015] 111~114, 211~212: Steps

[0016] Figures 121-123, 202, 203, 302, 303:

[0017] 410: Range Detailed Implementation

[0018] The terms "first," "second," etc., used in this invention do not specifically refer to order or sequence; they are merely used to distinguish elements or operations described using the same technical terms.

[0019] The immunohistochemical staining images mentioned here are slices of liver cells, the purpose of which is to stain for hepatitis B surface antigen (HBsAg). HBsAg is the first viral antigen to appear in the serum after hepatitis B virus infection and is also the most important indicator of hepatitis B. In immunohistochemical staining images containing HBsAg, the cells will appear reddish. A method is proposed here to analyze such images.

[0020] Figure 1 This is a flowchart illustrating an analysis method for immunohistochemical staining images, based on one embodiment. Please refer to... Figure 1 ,exist Figure 1 The image shows two immunohistochemical staining images, 101 and 102. Immunohistochemical staining images 101 and 102 are color images, each containing multiple pixels, with each pixel including red, blue, and green color channels. Immunohistochemical staining image 101 contains more hepatitis B surface antigen, and therefore appears redder than immunohistochemical staining image 102.

[0021] In step 111, multiple cell nuclei from the immunohistochemically stained image are segmented according to a machine learning model. This machine learning model is, for example, a convolutional neural network (CNN), whose architecture can be LeNet, AlexNet, VGG, GoogLeNet, ResNet, or VOLO. In this embodiment, the CNN disclosed in the paper Graham, Simon, et al. "Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images." Medical Image Analysis 58(2019):101563 is used. The content of this paper is also included in this specification, and will not be repeated here. In this embodiment, unlike Hover-net, the loss function is modified as shown in the following mathematical formulas 1 to 5.

[0022] [Mathematical Expression 1]

[0023] L = L NP +LHover;

[0024] [Mathematical Expression 2]

[0025] L NP =L bce +L dice +L focal ;

[0026] [Mathematical Expression 3]

[0027]

[0028] [Mathematical Expression 4]

[0029]

[0030] [Mathematical Expression 5]

[0031]

[0032] The network has two branches: the nuclear pixel (NP) branch and the HoVer branch. L is the loss function for the entire network, and the loss function for the nuclear pixel branch is LNP. NP The loss function of the HoVer branch is L. Hover Loss function L NP It is the sum of binary cross entropy loss, dice loss, and focal loss. Where N is the total number of pixels in the image, x... i Represents the ground truth, y i ε is the value predicted by the network; for example, y=1 represents a cell nucleus, and y=0 represents background. ε is a very small value to avoid division by zero. p is the estimated probability of being classified as a cell nucleus. γ is a real number determined experimentally, which is 0.5 in this example.

[0033] On the other hand, the loss function L Hover Then, as shown in the following mathematical formulas 6 to 8.

[0034] [Mathematical Expression 6]

[0035] L Hover =L mse +L msge ;

[0036] [Mathematical Expression 7]

[0037]

[0038] [Mathematical Expression 8]

[0039]

[0040] Where n is the total number of pixels in the image. i It is the output of the HoVer branch, Γ i This is the actual output. To calculate the mean squared gradient error, the output horizontal and vertical components are represented as p... i，x and p i，y Γ, respectively, correspond to the actual output at the level. i，x and the true vertical output Γ i，y . The gradient represents the horizontal direction. The gradient represents the vertical direction. m represents the number and set of all pixels in the image classified as cell nuclei. After step 111, Figure 121 is obtained. The value of each pixel in Figure 121 indicates whether it is a cell nucleus, for example, "1" represents a cell nucleus and "0" represents the background.

[0041] After the cell nucleus is cut out, in step 112, the cytoplasm can be further identified based on the identified nucleus. However, since the cell edges are usually blurry, the cytoplasm is difficult to identify, so convolutional neural networks cannot produce good results. Here, four image processing operations are proposed: dilation, region-growing, texture thresholding, and contrast-based operations.

[0042] First, there is the morphological expansion treatment, which can be expressed as the following mathematical formulas 9 and 10.

[0043] [Mathematical Expression 9]

[0044] D = U b∈B A b ;

[0045] [Mathematical Expression 10]

[0046] Y = D∩A′.

[0047] Where A is a binary graph, where "1" represents the cell nucleus and "0" represents the background; in this embodiment, Figure 121 can be used as binary graph A. A′ is the opposite of binary graph A. B is a mask, for example, 31x31 in size, where each element is 1. b This is an AND operation between a binary graph and element b, where ∩ represents the AND operation. After processing with mathematical formula 10, the cell nucleus will be removed. The results of the morphological expansion processing can be found in [reference needed]. Figure 2 After cutting out the cell nuclei from the immunohistochemically stained image 201, morphological expansion processing is performed on the cell nuclei in step 211 to obtain Figure 202 representing the cytoplasm. The pixels representing the cytoplasm in Figure 202 are also called cytoplasmic pixels. As can be seen from Figure 202, the extracted cytoplasm extends a distance outward from the cell nucleus. If a cell is very large, some cytoplasm may be lost; if the cell is very small, incorrect cytoplasm may be extracted.

[0048] The second image processing operation is region growing. This involves treating the cell nucleus as a seed and determining if surrounding pixels are similar to the seed. If so, they are added to the nucleus as cytoplasmic pixels. This process is repeated until all pixels within a specific range have been processed. In some embodiments, this specific range can be represented by the result of morphological dilation. Region growing can be expressed as the following mathematical formula 11.

[0049] [Mathematical Expression 11]

[0050] Color similar =|G(Seed i )-G(neighbor i )|<ε.

[0051] Where G(Seed) i The grayscale value represents the seed color, such as the brightness grayscale obtained after color space conversion, for example, the HSV color space. i The neighbor of the i-th seed can be defined as the 8 pixels surrounding a pixel in some embodiments. ε is a threshold value, which can be set to 7 in some embodiments. Region growth relies on the similarity of brightness in the cytoplasm; if the brightness of the cytoplasm differs greatly, the cytoplasm may be inaccurately captured.

[0052] The third image processing operation is texture thresholding. In this embodiment, Gabor kernels are used to obtain multiple feature values, and then principal component analysis is performed to reduce dimensionality. Each pixel is represented by one dimension, and then a threshold value is used to determine whether the pixel is a cytoplasmic pixel or the background.

[0053] The fourth image processing operation is contrast processing. In immunohistochemical staining images, some colors are clearly not cytoplasm. For example, white areas may be oil droplets, blue areas may be cell nuclei, and black areas may be portal cells. Therefore, pixels in the immunohistochemical staining image that conform to a specific color range can be removed, leaving cytoplasmic pixels. Specifically, the pixels can first be converted to the HSV color space, which includes hue, saturation, and brightness. Here, only pixels with a hue in the range of 100-125 (also known as the first preset range), a saturation in the range of 80-255 (also known as the second preset range), and a brightness in the range of 0-110 will be retained (that is, pixels with a brightness greater than 110 will be removed), and the remaining pixels will be removed. In addition, the cell nuclei identified in step 111 will also be removed, as shown in the figure. Figure 2 As shown, in step 212, pixels belonging to cell nuclei and pixels in a specific color range were removed from the immunohistochemically stained image 201, as shown in Figure 203. In some embodiments, step 212 may also remove small fragments, for example, by performing morphological erosion.

[0054] Regardless of the image processing method described above, multiple cells are formed by assigning cytoplasmic pixels to their corresponding nuclei based on their positions. In some embodiments, the concept of region competition can be used to allocate cytoplasmic pixels, meaning that different nuclei must compete for which cytoplasm belongs to them. For example, a watershed algorithm can be used, where the nuclei are considered low points and the cytoplasmic pixels are considered catchment areas. Essentially, the watershed algorithm involves injecting different types of water into different low points, causing the water level to rise in the catchment area until the different types of water meet, forming a dividing line. Those skilled in the art will understand the watershed algorithm, and it will not be elaborated upon here. In some techniques, the watershed algorithm sets a gradient for the catchment area; in this embodiment, the gradient is set to be uniform, but it expands outward by one pixel each time the water level rises. The result of the watershed algorithm is as follows... Figure 3 As shown, after the cell nucleus is cut out, the cytoplasm is identified, and the immunohistochemical staining image 301 is processed by a watershed algorithm, Figure 302 is obtained. The "black holes" in Figure 302 represent the locations of the cell nuclei. In Figure 302, the cytoplasmic pixels are assigned different gray levels (representing different cells). In some embodiments, the number of pixels in each cell can be further calculated. If the number of pixels is greater than a second threshold (e.g., 30,000) and less than a third threshold (1,500), it indicates that the cell may be a false detection, possibly belonging to portal veins or other tissues. Therefore, if the above conditions are met, the cytoplasmic pixels in the corresponding cell can be deleted. In some embodiments, the result of morphological expansion processing based on the cell nucleus (e.g., Figure 202, but Figure 202 is a different slice) can also be added to Figure 302. The result after addition is shown in Figure 303. The cytoplasm plus the cell nucleus in Figure 303 forms a cell. Our goal is to analyze the color of each cell, but we shouldn't be fixated on obtaining all the cytoplasm in each cell. The result of morphological expansion processing based on the cell nucleus already contains a portion of the cytoplasm, and in some cases, these cell values ​​are sufficient for analysis. Therefore, we add the result of the watershed algorithm to the result of morphological expansion processing. Please refer to the previous section. Figure 1 After the processing in step 112, Figure 122 can be obtained to represent the cytoplasm.

[0055] In some embodiments, the concept of regional competition described above can also be determined by the location of the cytoplasm, for example by calculating the spatial distance between each cytoplasmic pixel and each cell nucleus, and assigning the cytoplasmic pixel to the nearest cell nucleus.

[0056] Next, in step 113, the pixel color score for each pixel in each cell is calculated. The HSV color space is also used here. First, "red" pixels are selected. Pixels with a hue (H) in the range of 0-10 and 156-180, a saturation (S) in the range of 43-255, and a lightness (V) in the range of 46-255 are chosen as red pixels. All non-red pixels are assigned a pixel color score of "0". Please refer to [the next step]... Figure 4 , Figure 4 This diagram illustrates the calculation of pixel hue scores based on the HSV color space, according to one embodiment. The horizontal axis represents saturation (S), and the vertical axis represents lightness (V). Since saturation (S) is limited to the range of 43–255 and lightness (V) to the range of 46–255, only pixels within the range of 410 are considered. The pixel hue score is calculated based on the ratio (I) between saturation (S) and lightness (V). When the ratio I is greater than 0.742, the pixel hue score is set to "3+"; when the ratio I is between 0.482 and 0.742, the pixel hue score is set to "2+"; and when the ratio I is less than or equal to 0.482, the pixel hue score is set to "1+". The aforementioned critical values ​​of 0.742 and 0.482 are merely examples; different critical values ​​can be selected based on experimental results in other embodiments. Furthermore, LAB or other suitable color spaces can be used in other embodiments. In some color spaces, hue is also referred to as "hue". In the above embodiments, the pixel color fraction is specified as four levels: "0", "1+", "2+", and "3+". However, in other embodiments, more or fewer levels can be set, and this invention is not limited thereto. Please refer to... Figure 1 After the processing in step 113, Figure 123 can be obtained, where darker gray levels represent higher pixel coloring scores.

[0057] After each pixel has been assigned a pixel staining score, the cell staining score for each cell can be calculated in step 114. The cell nucleus can be obtained from Figure 121, and the cytoplasm from Figure 122. Allocating the cytoplasm to the corresponding cell nucleus yields a pixel for each cell. In this embodiment, the cell staining score is calculated by weighting the number of pixels as the number of pixels. Specifically, the cell staining score is calculated as shown in the following mathematical formula 12.

[0058] [Mathematical Expression 12]

[0059]

[0060] Where α represents the proportion of pixels with a staining score of "1+" in the corresponding cell, β represents the proportion of pixels with a staining score of "2+" in the corresponding cell, and γ represents the proportion of pixels with a staining score of "3+" in the corresponding cell. For example, if 1.7% of the pixels in a cell are "3+", 7.2% are "2+", and 26.1% are "1+", then the calculated cell staining score is 15.2. The calculated cell staining score can be provided to the doctor for judgment. The "100", "200", and "300" in the above mathematical formula 12 are only examples, and any other values ​​can be used in other embodiments.

[0061] In the aforementioned immunohistochemical staining image analysis method, cells can be segmented in a relatively objective manner, and the degree of redness of each cell can be represented by a score, saving manpower and assisting doctors in interpretation. Unlike using convolutional neural networks to identify cytoplasm, this invention proposes several image processing operations (such as the watershed algorithm) to identify cytoplasm, overcoming the problem of difficulty in identifying cytoplasm. Furthermore, when processing cytoplasm, this invention uses a deletion method to obtain cytoplasm, which solves the problem of blurred cell boundaries. Compared to actively detecting cytoplasm, this invention's approach is more effective. Finally, this invention uses the ratio between saturation and brightness to calculate pixel staining scores, obtaining objective results.

[0062] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone with common knowledge in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the appended claims.

Claims

1. A method for analyzing immunohistochemical staining images, applicable to a computer system, characterized in that, The analytical method includes: Multiple cell nuclei are cut out from the immunohistochemical staining image according to a machine learning model, wherein the immunohistochemical staining image includes multiple pixels, and each of the multiple pixels includes multiple color channels. Remove the plurality of pixels belonging to the plurality of cell nuclei from the immunohistochemical staining image and remove the plurality of pixels that conform to at least one color range to obtain a plurality of cytoplasmic pixels; Each of the plurality of cytoplasmic pixels is assigned to one of the plurality of cell nuclei according to the position of the plurality of cytoplasmic pixels to form a plurality of cells; and Calculate the pixel staining score of each of the plurality of pixels in each of the plurality of cells, thereby calculating the cell staining score of each of the plurality of cells; The step of removing the plurality of pixels that conform to the at least one color range includes: Each of the plurality of pixels is converted to a color space, the color space comprising a hue, a saturation, and a brightness; and Retain the plurality of pixels whose hue is within a first preset range, whose saturation is within a second preset range, and whose brightness is within a preset brightness range, and remove the plurality of pixels whose hue is not within the first preset range, whose saturation is not within the second preset range, and whose brightness is not within the preset brightness range; The first preset range is a hue of 100 to 125, the second preset range is a saturation of 80 to 255, and the preset brightness range is a brightness of 0 to 110.

2. The analytical method according to claim 1, characterized in that, The step of assigning each of the plurality of cytoplasmic pixels to one of the plurality of cell nuclei to form the plurality of cells according to the position of the plurality of cytoplasmic pixels includes: The multiple cell nuclei are treated as multiple low points in the region, and the multiple cytoplasmic pixels are treated as water collection areas, thereby executing a watershed algorithm.

3. The analytical method according to claim 2, characterized in that, The analytical method further includes, after the formation of the plurality of cells: Calculate the number of pixels in each of the plurality of cells; If the number of pixels in a first cell of the plurality of cells is greater than a second threshold and less than a third threshold, the cytoplasmic pixels in the first cell are deleted, and the first cell is formed by adding the result of morphological expansion processing performed on the nucleus of the first cell.

4. The analytical method according to claim 3, characterized in that, The step of calculating the pixel staining score of each of the plurality of pixels in each of the plurality of cells, thereby calculating the cell staining score of each of the plurality of cells, includes: The pixel staining score of a pixel is calculated based on the ratio between the saturation and the brightness of each of the plurality of pixels in each of the plurality of cells; and Using the number as a weight, the weighted average of the pixel staining scores of the plurality of pixels in each of the plurality of cells is calculated as the cell staining score of the cell.

Images