Model training apparatus and model application apparatus for histopathology stained section images

Abstract

The application provides a model training device and a model application device for histopathology staining section images. The model training device (100) comprises a histopathology staining section image blocking unit (101), an immunohistochemical staining section image blocking unit (102), a tumor tissue classification unit (103), a registration unit (104) and a model training unit (105). The application obtains multiple IHC re-staining section images by re-staining the same section of the histopathology staining section multiple times, classifies the tumor tissues corresponding to the multiple IHC re-staining section images according to biological rules, can obtain the quantitative spatial mask of the biomarkers of the tumor tissues, then takes the histopathology staining section image as an input value, takes the quantitative spatial mask as the ground truth (GT), and registers the two, so that a more accurate deep learning model can be established.

Description

TECHNICAL FIELD

[0001] The present application relates to a model training device and a model application device for histopathology staining section images, which respectively train and apply a model for determining a biomarker state of tumor tissue according to histopathology staining section images. BACKGROUND

[0002] Cancer immunotherapy has changed the treatment of cancer. In 2019, two of the top ten drugs sold globally involved cancer immunotherapy, and the sales of Keytruda, one of them, reached more than 10 billion US dollars. It has been approved for the treatment of cancers such as unresectable or metastatic microsatellite instability-high (MSI-H) or mismatch repair-deficient (dMMR) colorectal cancer.

[0003] Microsatellite instability-high (MSI-H) or mismatch repair-deficient (dMMR) is a biomarker approved by the FDA for cancer immunotherapy. Microsatellite instability (MSI) refers to the increase or loss of the number of repeated sequences in the genome due to gene replication errors, resulting in shortening or lengthening of microsatellite fragments. Generally, such errors are repaired by the DNA mismatch repair system (MMR). However, when the related genes in MMR malfunction due to promoter hypermethylation or gene mutations, DNA replication errors cannot be repaired, and the number of repeated units of some microsatellite sites fluctuates, resulting in microsatellite instability.

[0004] Recent studies have shown that MSI is of great significance to the diagnosis, prognosis, and chemotherapy sensitivity of Lynch syndrome and colorectal cancer. In addition to colorectal cancer, researchers have also found MSI in endometrial cancer, ovarian cancer, gastric cancer, and breast cancer. As a sensitive indicator of tumor genetic instability, the detection of MSI is of great significance to early diagnosis of tumors, prognosis, chemotherapy sensitivity, and identification of high-risk groups.

[0005] The conventional MSI / MMR detection methods are polymerase chain reaction (PCR) and immunohistochemistry (IHC), but both require professional personnel and are expensive. Although the conventional hematoxylin-eosin (H&E) staining method is easy to operate and inexpensive, it cannot detect the MSI / MMR state.

[0006] The advent of digital pathology has made it possible to predict MSI / MMR status from images of histopathology sections stained with conventional H&E, for example, a method for detecting biomarkers from images of histopathology sections is described in patent document 1 (hereinafter also referred to as "prior art") which includes two stages of model establishment and model application. In the model establishment, the tumor tissue sections are stained with H&E to obtain images of H&E stained sections, while the MSI / MMR status of the tumor tissue is qualitatively detected using PCR or IHC method, then the images of H&E stained sections are taken as input values, and the MSI / MMR status qualitative detection data are taken as ground truth (GT), and a deep learning model is established. In the model application, the qualitative data of MSI / MMR status can be obtained by inputting the H&E stained section images of the target tumor into the model.

[0007] The prediction of MSI / MMR based on deep learning can be widely used in cancer patients and can reduce the related medical burden.

[0008] Prior art documents

[0009] Patent documents

[0010] Patent document 1: US2020258223A1 Summary of the invention

[0011] Problems to be solved by the invention

[0012] One of the reasons for the inaccuracy of the ground truth (GT) of MSI / MMR status in the deep learning model established in the prior art is the heterogeneity of tumors, specifically, the tumor tissue stained with H&E is not the same tissue as the tumor tissue for MSI / MMR status detection. In addition, the current deep learning model can only qualitatively detect the MSI / MMR status, but cannot quantitatively determine the MSI / MMR level. Figure 1 is a schematic diagram illustrating the problems in the deep learning model of the prior art, wherein, Figure 1A is the case when training the model, Figure 1B is the case when applying the model. As Figure 1A shown, in the prior art, since only the stained sections with different MSI / MMR levels can be labeled as positive without distinction, the obtained label data is not accurate, thereby affecting the accuracy of the obtained deep learning model. Further, as Figure 1B shown, when applying the model, although the three cases 1-3 are different in the spatial quantitative distribution of biomarkers, only qualitative results without spatial and quantitative information can be obtained by the established model (for example, for Case 1 and Case 2), and even incorrect conclusions that do not match the actual situation can occur (for example, for Case 3).

[0013] The present application is completed in order to solve the problems existing in the prior art, and aims to provide a device capable of accurately training a model for determining the biomarker state of tumor tissue according to a histopathology stained slice image, thereby establishing a more accurate deep learning model and further obtaining a more accurate biomarker state prediction result.

[0014] To this end, the present inventors have conducted extensive and in-depth research, and as a result, it has been found that by performing multiple IHC re-staining on the same slice of the histopathology stained slice to obtain multiple IHC re-stained slice images, classifying the tumor tissue corresponding to the multiple IHC re-stained slice images according to biological rules, obtaining a quantitative spatial mask of the biomarker of the tumor tissue, and then registering the histopathology stained slice image as an input value and the quantitative spatial mask as a ground truth (GT), an accurate deep learning model can be established, thereby completing the present application.

[0015] Means for solving the problem

[0016] That is, the present application provides the following technical solutions:

[0017] (1) A model training device that trains a model for determining the biomarker state of tumor tissue according to a histopathology stained slice image, comprising:

[0018] a histopathology stained slice image blocking unit that performs blocking processing on a histopathology stained slice image of a tumor tissue,

[0019] an immunohistochemical staining slice image blocking unit that performs blocking processing on immunohistochemical staining slice images obtained by performing multiple immunohistochemistry (IHC) re-staining on the same slice that has been histopathologically stained,

[0020] a tumor tissue classification unit that classifies the same tumor tissue corresponding to the blocked images obtained by the immunohistochemical staining slice image blocking unit according to biological rules, and obtains a quantitative spatial mask (mask) of the biomarker of the tumor tissue,

[0021] a registration unit that registers each of the blocked images obtained by the histopathology stained slice image blocking unit as an input value, registers the quantitative spatial mask as a ground truth (GT), and registers each input value with each ground truth, and

[0022] a model training unit that performs model training according to the registration results obtained in the registration unit to obtain a deep learning model.

[0023] (2) The model training apparatus according to the above (1), wherein the histopathology stained section image is a hematoxylin-eosin (H&E) stained section image.

[0024] (3) The model training apparatus according to the above (1) or (2), wherein the biomarker status is a mismatch repair (MMR) status.

[0025] (4) The model training apparatus according to any one of the above (1) to (3), wherein the quantitative spatial mask is defined based on a combination of loss status of four MMR proteins of MLH1, MSH2, MSH6, and PMS2.

[0026] (5) The model training apparatus according to any one of the above (1) to (4), further comprising:

[0027] a histopathology staining mechanism that stains a tumor tissue section, and / or

[0028] an immunohistochemistry staining mechanism that performs multiple immunohistochemistry staining on a tumor tissue section.

[0029] (6) A model application apparatus that applies a model that determines a biomarker status of a tumor tissue from a histopathology stained section image, comprising:

[0030] a histopathology stained section image patching unit that patches a histopathology stained section image of a target tumor,

[0031] a model application unit that uses a deep learning model established by the model training apparatus according to any one of the above (1) to (5) to predict a biomarker status of a target tumor using a patch image obtained by the histopathology stained section image patching unit as an input value, and

[0032] an output unit that generates a report in which the patch image, the biomarker status, and a number that visualizes the presence of the predicted biomarker are superimposed as a prediction result.

[0033] Effects of the Invention

[0034] The present application can obtain a quantitative index of a biomarker in a tumor tissue by performing multiple IHC re-staining on the same section of a histopathology stained section, can obtain a more accurate deep learning model by using the quantitative biomarker index at the cell / pixel level as a GT, and thus can more accurately classify a biomarker at the tumor level, and can obtain a higher AUC, sensitivity, and specificity, thereby providing a reliable basis for subsequent tumor treatment. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a schematic diagram illustrating the problems existing in current deep learning models. Figure 1A This is the case when training the model. Figure 1B This refers to the case when applying the model.

[0036] Figure 2 This is a block diagram illustrating the configuration of the model training device of the present invention.

[0037] Figure 3 A flowchart illustrating the process of establishing a deep learning model using the model training apparatus of the present invention.

[0038] Figure 4 The diagram illustrates the steps involved in building the deep learning model according to the present invention.

[0039] Figure 5 is a schematic diagram comparing the model of the prior art with the model of the present invention, wherein, Figure 5A This is the case when training the model. Figure 5B This refers to the case when applying the model.

[0040] Symbol Explanation

[0041] 100 Model Training Device

[0042] Image block units for histopathological stained sections 101 and 201

[0043] 102 Immunohistochemical staining slide image block unit

[0044] 103 Tumor Tissue Classification Units

[0045] 104 registration units

[0046] 105 model training units

[0047] 200 Model Application Device

[0048] 202 Model Application Unit

[0049] 203 Output Unit

[0050] 300 Histopathological Staining Facilities

[0051] 400 Immunohistochemical Staining Institution Detailed Implementation

[0052] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the following description of the embodiments is merely for illustrating the inventive concept of the present invention and is not intended to limit the scope of the invention.

[0053] The present application relates to a model training device and a model application device for histopathology staining section images, which respectively train and apply a model for determining a biomarker state of tumor tissue from a histopathology staining section image.

[0054] [Model training device]

[0055] One embodiment of the present application relates to a model training device that trains a model for determining a biomarker state of tumor tissue from a histopathology staining section image.

[0056] Figure 2 A block diagram showing the configuration of the model training device of the present application. As shown in Figure 2 The model training device 100 of the present application is provided with a histopathology staining section image blocking unit 101, an immunohistochemistry staining section image blocking unit 102, a tumor tissue classification unit 103, a registration unit 104, and a model training unit 105. In addition, the model training device 100 of the present application can be connected to a histopathology staining mechanism 300 and an immunohistochemistry staining mechanism 400, both of which are optional. For the present application, it is also possible to directly scan and input the stained section image into the model training device 100 of the present application.

[0057] After the tumor tissue section is stained by the staining mechanisms 300, 400, a digital whole slide image (WSI) is obtained by a scanning mechanism (not shown). The digital whole slide image described in the present application is specifically obtained as follows: first, a digital microscope or magnification system is used to scan and collect images of the glass section under a low-power objective lens, the microscopic scanning platform automatically scans and moves in the XY axis direction of the section, and automatically focuses in the Z axis direction; then, the scanning control software collects high-resolution digital images using a programmed scanning method based on the effective magnification of the optical magnification device, and the image compression and storage software automatically performs seamless splicing processing on the images to generate a whole digital slide (WSI).

[0058] In the model training device 100, the histopathology staining section image blocking unit 101 performs blocking processing on the above-mentioned digital whole slide image of the histopathology staining of the tumor tissue.

[0059] In the present application, the histopathological staining method is not particularly limited, and can be, for example, hematoxylin-eosin (H&E) staining (for showing the morphology of the entire cellular tissue), Masson staining (for discriminating collagen fibers and muscle fibers), oil red O fat staining (for showing fat in the tissue), Sirius red staining (for showing collagen fiber abnormalities or fibrosis), or ponceau-fuchsin staining (for showing the structure of articular cartilage, subchondral bone, and bone tissue), and the like. Among these, hematoxylin-eosin (H&E) staining is the most basic and widely used staining method in histology and pathology, and is preferred because it is easy to handle and inexpensive.

[0060] The division into blocks can be performed, for example, for a stained section image of 100,000 x 100,000 pixels, by dividing it into a plurality of images of 256 x 256 pixels. The specific method of division into blocks can use a method generally used in digital pathology in the related art, and is not particularly limited, and can be, for example, a method described in Patent Literature 1, paragraphs 0077 to 0090.

[0061] In the present application, the immunohistochemical (IHC) staining is multiple IHC re-staining of the same section subjected to the above-described histopathological staining, and specifically, for example, the section after the histopathological staining can be washed and then subjected to IHC staining. Since the histopathological staining and the immunohistochemical staining are sequentially performed on the same section, the problem of heterogeneity of the tumor tissue can be avoided. The IHC re-staining in the present application is performed with respect to the biomarker status of the tumor tissue. As the biomarker in the present application, there is no particular limitation, and can be, for example, microsatellite instability (MSI) or mismatch repair (MMR), tumor infiltrating lymphocytes (TIL), or programmed cell death-ligand 1 (PD-L1), and the like. A person skilled in the art can select an appropriate biomarker as needed. In the present application, MSI / MMR is preferably used as the biomarker.

[0062] The MMR system can involve multiple mismatch repair proteins, including two families of MutS (MSH2, MSH3, and MSH6, etc.) and MutL (MLH1, MLH3, PMS1, and PMS2) when participating in DNA repair. Among these, MLH1, MSH2, MSH6, and PMS2 are the dominant proteins of MMR. When the expression of one or more of these four main MMR proteins is absent (negative), it can be determined as mismatch repair gene defect (dMMR), and when all are positive, it can be determined as mismatch repair gene complete (pMMR).

[0063] In the IHC re-staining of the present application, the expression of one protein can be determined by one IHC staining, i.e., one IHC stained section image reflecting the deletion status of one MMR protein can be obtained by one IHC staining. Therefore, in order to determine the deletion status of the above-mentioned four MMR proteins, at least four IHC re-staining is required to obtain four IHC stained section images.

[0064] For the obtained IHC stained section images, scanning processing is performed in the same manner as the above-mentioned histopathological stained section images to obtain digital whole field section images. For these digital whole field section images, the block processing is performed by the immunohistochemical stained section image blocking unit 102, and the method of the block processing is the same as the processing performed by the above-mentioned block unit 101, which can be referred to.

[0065] The tumor tissue classification unit 103 classifies the same tumor tissue corresponding to the block image obtained by the immunohistochemical stained section image blocking unit 102 according to the biological rule to obtain the quantitative spatial mask of the biomarker of the tumor tissue. Specifically, in the present application, the classification of the tumor tissue is performed according to the biological rule (F) of the biomarker state (B) based on a plurality of proteins to define the biomarker state (B). For example, when proteins 1, 2,..., n are denoted as p1, p2,..., pn, the biomarker state (B) can be expressed as follows:

[0066] B = F(p1, p2,..., pn)

[0067] In the present application, when the four MMR proteins of MLH1, MSH2, MSH6, and PMS2 are used, the above-mentioned n is 4, and the biological rule (F) for defining the biomarker state (also referred to as "dMMR mask") by the combination of the deletion status of the four MMR proteins is shown in Table 1. As can be seen from Table 1, the dMMR mask obtained by the present application is a quantitative spatial mask of the biomarker, which is a quantitative spatial metric at the cell / pixel level.

[0068] Table 1

[0069]

[0070] In the registration unit 104, each block image obtained by the histopathological stained section image blocking unit 101 is used as an input value, the above-mentioned quantitative spatial mask obtained by the tumor tissue classification unit 103 is used as a ground truth (GT), and each input value and each ground truth are one-to-one corresponding, and registration is performed.

[0071] In the model training unit 105, model training is performed based on the registration result obtained in the registration unit 104, and a deep learning model is obtained. The method of training the deep learning model, such as an AI model, can be performed using a deep learning process such as a convolutional neural network (CNN) or a fully convolutional network (FCN), without particular limitation, and can be performed, for example, in accordance with the process described in paragraphs 0084 to 0087 of the specification of Patent Document 1.

[0072] Figure 3 To explain the flowchart of establishing a deep learning model by the model training apparatus of the present application, Figure 4 To explain the flowchart of establishing a deep learning model by the model training apparatus of the present application,

[0073] In step

[10] , the tumor tissue section is subjected to histopathological staining, such as H&E staining, the obtained H&E-stained section of the tumor tissue is scanned as an H&E-stained section image (digital whole field section image), and the model training apparatus 100 is input, and then the section image is subjected to patching processing, and each patch image is used as an input value.

[0074] In step

[20] , the H&E-stained section of the tumor tissue described above is subjected to multiple IHC re-staining, and each re-staining is performed for one MMR protein selected from MLH1, MSH2, MSH6, and PMS2. Similarly, they are scanned and subjected to patching processing, and each patch image is classified (also referred to as "labeled") according to the different tissues corresponding to the respective biological rules, and a quantitative spatial mask of the biomarker in the tumor tissue corresponding to each patch image is obtained, and the quantitative spatial mask is used as a ground truth (GT).

[0075] In step

[30] , the AI model is obtained by performing training by registering each input value and each ground truth.

[0076] [Model application apparatus]

[0077] Another embodiment of the present application relates to a model application apparatus that applies a deep learning model obtained by a model training apparatus. As shown in Figure 2 The model application apparatus 200 of the present application has a histopathological stained section image patching unit 201, a model application unit 202, and an output unit 203. In addition, the model application apparatus 200 can be connected to the histopathological staining mechanism 300, and the stained section image can be directly scanned and input to the model application apparatus 200 of the present application.

[0078] In the model application device 200, the histopathology staining section image blocking unit 201 is configured to block the histopathology staining section image (digital whole field section image) of the target tumor whose biomarker state is to be determined. The blocking unit 201 can segment the digital whole field section image of the target tumor and input the segmented image to the model application unit 202 as an input value.

[0079] The model application unit 202 is a deep learning model established by the model training device 100. With the deep learning model, the biomarker state of the target tumor can be predicted without complex IHC staining of the tumor tissue of the target tumor, thereby providing a reliable basis for the treatment of the tumor.

[0080] The output unit 203 outputs a report in which the prediction result is superimposed on the segmented image, the biomarker state, and the number visualizing the existence of the predicted biomarker. The output form is not particularly limited and can be displayed on a display or printed as needed.

[0081] Figure 3 、 Figure 4 Step

[40] in the above method is an application step of the deep learning model of the present application. In this step, the H&E staining image of the target tumor is input to the deep learning model (AI model) as an input value, and the quantitative spatial measure of MMR of the target tumor is output by the model.

[0082] FIG. 5 is a schematic diagram comparing the model of the prior art with the model of the present application, in which Figure 5A is the case when the model is trained, Figure 5B is the case when the model is applied. As Figure 5A shown, when the model is trained, unlike the method of the prior art which can only perform qualitative classification, the present application can classify (label) the tumor tissue corresponding to each segmented image using biological rules, assign different labels, and obtain a quantitative spatial mask of the biomarker in the tumor tissue. Therefore, the deep learning model obtained by the present application is more accurate. As a result, when applied, as Figure 5B shown, when the histopathology staining section image of a different target tumor is input, a more accurate output result reflecting the true state of the biomarker in the tumor tissue can be obtained, thereby improving the sensitivity and specificity of biomarker state detection.

[0083] As can be seen, by the present application, the quantitative spatial measure of the biomarker of the target tumor can be directly predicted from the histopathology staining (e.g., H&E staining) section image, thereby providing a reliable basis for subsequent tumor treatment.

Claims

1. A model training device for training a model that determines the biomarker status of tumor tissue based on histopathological stained slide images, comprising: The histopathological stained section image segmentation unit divides the histopathological stained section image of tumor tissue into blocks. The immunohistochemical staining section image segmentation unit divides the immunohistochemical staining section images obtained by repeatedly counterstaining the same section stained for histopathology into separate segments. The tumor tissue classification unit classifies tumor tissues corresponding to the same segmented image obtained from immunohistochemical staining section image segmentation units according to biological rules, thereby obtaining a quantitative spatial mask of the biomarkers of the tumor tissue. The registration unit takes the segmented images obtained by the histopathological stained section image segmentation unit as input values, uses the quantitative spatial mask as the ground truth (GT), and registers each input value with each GT. The model training unit trains the model based on the registration results obtained from the registration unit, thus obtaining a deep learning model.

2. The model training apparatus according to claim 1, wherein, The histopathological stained section images are hematoxylin-eosin (H&E) stained section images.

3. The model training apparatus according to claim 1 or 2, wherein, The biomarker status is the mismatch repair (MMR) status.

4. The model training apparatus according to any one of claims 1 to 3, wherein, The quantitative spatial mask is defined based on the combination of the deletion states of the four MMR proteins: MLH1, MSH2, MSH6, and PMS2.

5. The model training apparatus according to any one of claims 1 to 4, further comprising: Immunohistochemical staining facilities perform multiple immunohistochemical stainings on tumor tissue sections, and / or Histopathological staining facilities stain tumor tissue sections.

6. A model application device for applying a model that determines the biomarker status of tumor tissue based on histopathological stained section images, comprising: The histopathological stained section image segmentation unit divides the histopathological stained section image of the target tumor into blocks. The model application unit, using the deep learning model established by the model training device according to any one of claims 1 to 5, takes the segmented images obtained by the histopathological stained section image segmentation unit as input values ​​to predict the biomarker status of the target tumor, and The output unit generates a report of the prediction results as an overlay of patch images, biomarker states, and numbers that visualize the presence of the predicted biomarkers.

Images