Histopathological reading support device and method

Abstract

This specification provides a tissue pathology reading support device and method that, when a lesion is present, divides a learning slide tissue image showing the location of the lesion into multiple patches, infers a classification for each of the multiple patches, proceeds with learning of the patch classification results, integrates the multiple patches and the patch classification results to infer a classification for the reconstructed slide tissue image, proceeds with learning of the slide tissue image classification results, infers a classification for each of the multiple patches using the trained artificial intelligence model to generate a patch classification result, and integrates the multiple patches and the patch classification results to infer a classification for the reconstructed slide tissue image, thereby generating a slide tissue image classification result.

Description

[Technical Field]

[0001] The present invention relates to a tissue pathology reading support device and method. [Background technology]

[0002] Digital pathology is a general term for a technology that uses a scanner to convert glass pathology slides into digital slides or slide tissue images (WSI; Whole Slide Image), which are then stored and viewed on a monitor via a personal computer or computer network, instead of reading them using a conventional optical microscope.

[0003] A key technology in digital pathology is the conversion of the overall geometry of tissues embedded on glass slides into high-speed, high-resolution digital images.

[0004] The ultimate goal of digitizing histopathology images is to enable "computer-assisted diagnosis" through the use of various automated image analysis computer algorithms.

[0005] Meanwhile, while the workload of pathologists is steadily increasing, the number of pathologists is relatively insufficient. This relative shortage of pathologists and their increasing workload can lead to human error. A pathological diagnosis from a histopathological test is essentially a "confirmative diagnosis." Therefore, if a false negative results in early diagnosis and missed treatment, there is no opportunity to recognize the error until the next test. Unfortunately, false negatives are the most common error in pathological diagnosis. Therefore, tools are needed to reduce false negatives in a repetitive, labor-intensive, and habitual reading environment or to provide an opportunity for early detection and immediate correction. Blinded review is an important and effective method for improving quality control. Therefore, it is practically impossible to double-check or review all slides read each day.

[0006] The commonly recommended quality control for pathology laboratories is to review a certain percentage of slides each month on a voluntary basis, but this percentage can vary depending on the institution's circumstances. Sometimes, significant discrepancies arise during this process and are corrected late. However, this usually occurs one to two months after the initial diagnosis report date, so these follow-up measures are often too late to provide clinically meaningful results for patients. If an artificial intelligence model could assist in this function, it would be a great help to both patients and pathologists.

[0007] In order to apply the artificial intelligence classification model as a screening tool before specialists read it, all of the target glass slides must be digitized in advance.

[0008] In a predictable workflow, glass slides are first prepared and digitized (scanned), and AI predictions are made. Based on the predictions, positive cases are prioritized and listed. A specialist then opens the whole slide image (WSI) in a viewer and makes a priority diagnosis based on the scanned slide. Based on the AI ​​predictions and heatmaps, the specialist can directly check the glass slide through a microscope if necessary, and otherwise make a diagnosis based solely on the slide image.

[0009] Digital pathology has recently developed rapidly and deeply, and has already been rapidly adopted into clinical practice, offering several advantages over traditional pathology. However, in the field of pathology, it remains difficult to completely replace the reading capabilities of traditional methods based on glass slides. In particular, there are concerns that the detection of microorganisms such as Helicobacter pylori in gastric biopsy tissues is difficult using slide histology images. For pathology laboratories that routinely report classification and histologic grading of gastritis according to the Updated Sydney System, there is a real burden in directly applying digital pathology to primary diagnosis of gastric biopsies without glass slides.

[0010] In particular, for direct application of slide histology images to primary diagnosis, they must be scanned at the highest possible resolution (40x or higher) to minimize the discrepancy with traditional reading via a microscope. In this case, the construction and operation of an information computing infrastructure for storing and processing high-resolution slide histology images requires considerable costs. However, for quality control applications other than primary diagnosis, slide histology images scanned at a lower resolution (e.g., 20x) are sufficient.

[0011] Furthermore, in most commercial pathology laboratories, where reading is performed immediately after slide production, changing the workflow to postpone the reading step until after slide scanning can be a significant burden. Due to these limitations, some researchers have proposed Augmented Reality Microscope (ARM) with real-time artificial intelligence integration, rather than using slide histology images. However, even the most powerful AI-based ARMs are limited in their ability to compensate for the human error of pathologists in daily life, as they cannot detect lesions outside the microscope's field of vision. Furthermore, there are currently no publicly available or commercially available models of ARM methods applicable to endoscopic biopsy reading.

[0012] Finally, AI models prioritize readings, visualize suspicious lesions, and provide predicted classification results before the pathologist reads them, potentially inducing AI-dependent bias on the part of pathologists. While many AI models have been introduced that perform well enough to rival the diagnostic capabilities of pathologists, legal responsibility and authority for each case's diagnostic results ultimately rests with the pathologist. Bias itself is a problem, and the better the AI ​​model's performance, the greater the likelihood of dependency. This may also be a concern for patients. If a pathologist's biased diagnosis, relying on a high-performance AI model, leads to a misdiagnosis (e.g., when both the AI ​​model and the pathologist miss a lesion, or when the pathologist ignores the slide tissue image and reads it, believing the AI ​​model's prediction that the result will be negative), legal disputes over liability may arise.

[0013] In recent years, many researchers have developed and proposed artificial intelligence models based on histological images of histopathological slides, and some studies have actually claimed that they can perform as well as or better than pathologists. However, in the end, full responsibility and authority for each diagnosis remains with the pathologist, and artificial intelligence cannot completely replace this role.

[0014] In particular, the differentiation of carcinoma (CA) / high-grade dysplasia (HGD) in gastrointestinal pathology exhibits interobserver variability or inconsistency across groups. It is easy to predict that the performance of each AI classification developed for gastrointestinal pathology will vary depending on how the classification is defined. In fact, other similar studies have shown differences in classification definitions across groups, and some studies have excluded diagnoses corresponding to the gray zone from their studies.

[0015] Furthermore, in their daily work, pathologists often use vague terms when reporting on endoscopic biopsy (or other small biopsy) specimens, unlike when reading resection specimens. For example, they may use terms such as "atypical glandular proliferation of undetermined significance," "suspicious for dysplasia," "cannot be ruled out malignancy," "favor neoplastic," etc. In this reality, the expectation that AI models can solve these problems for pathologists is almost an illusion, due to the limitations of reading small biopsy tissues.

[0016] However, most research focuses on improving the performance of AI models, i.e., how accurately they diagnose a region of interest (ROI), which often leads to low reproducibility in clinical practice. This is because the conditions for high accuracy of AI models are significantly different from the real-world conditions in daily practice. Well-refined data preparation is crucial for high accuracy of AI models. In fact, poor scan quality (out of focus, missing tissue, air bubbles, etc.) and poor slide quality (poor staining, poor fixation, tissue artifacts, air bubbles, tissue folding, poor dehydration, etc.) are factors that reduce model performance. Model performance is maximized when all of these artifacts are artificially eliminated and training is performed using only highly refined data, i.e., clean scans without blur from well-prepared slides. However, this seems rather contradictory and unrealistic when considering the transformation to a "fully digitalized pathology laboratory."

[0017] As mentioned above, the number of histopathology examinations is steadily increasing, placing an excessive burden not only on pathologists but also on the technicians in the pathology labs who prepare the slides. This increased workload also impacts histopathology technicians' work capacity, leading to a decline in slide quality. Pathologists cannot control the perfect quality of every slide every day, and in reality, a higher percentage of slides than expected are encountered that do not meet the standard. The same is true for scan quality. Companies report scan error rates of 1-3%. These slides can be so poorly quality that they are completely unreadable. While some slides are readable, low-quality slide histology images exhibiting various artifacts, such as those mentioned above, are also common. While readability is possible, rescanning all of the low-quality slide histology images is practically impossible and inefficient. From the perspective of "developing usable AI models and applying them in practice," it would be more appropriate to develop and apply reliable AI models that reflect reality and are tailored to routine practice, rather than being forced to take on such inefficient workloads simply to maximize the performance of AI models. Summary of the Invention [Problem to be solved by the invention]

[0018] Embodiments provide a histopathology reading support device and method that reduces potential human errors, particularly false negatives, routinely experienced by pathologists. [Means for solving the problem]

[0019] This specification provides a tissue pathology reading support device and method that, when a lesion is present, divides a learning slide tissue image showing the site of the lesion into multiple patches, infers the classification of each of the multiple patches, and proceeds with learning of the patch classification results for an artificial intelligence model, by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image, and generates a patch classification result by inferring the classification of each of the multiple patches using the trained artificial intelligence model, and generates a slide tissue classification result by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image.

[0020] According to one embodiment, a tissue pathology reading support device is provided, which includes an input unit that inputs slide information and slide tissue images; a preprocessing unit that divides the input slide tissue images into multiple patches; a tissue classification unit that generates patch classification results by inferring the classification of each of the multiple patches using a trained artificial intelligence model, and generates slide tissue classification results by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image; a memory unit that stores slide information and slide tissue images, multiple patches, patch information related to the patches, patch classification results, and slide tissue classification results; and an output unit that outputs the patch classification results and slide tissue classification results for each of the multiple patches and slide tissue images.

[0021] This trained artificial intelligence model can proceed with learning about the patch classification results by dividing a training slide tissue image showing the location of the lesion into multiple patches using a preprocessing unit when a lesion is present, and inferring the classification of each of the multiple patches, and can proceed with learning about the slide tissue classification results by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image.

[0022] According to another embodiment, a method for supporting histopathological reading of tissues is provided, including: an input step of inputting slide information and a tissue slide image; a preprocessing step of dividing the input tissue slide image into a plurality of patches; a learning step of dividing a training tissue slide image showing the location of the lesion into a plurality of patches if a lesion is present, and inferring a classification for each of the plurality of patches to proceed with learning of the patch classification result, and integrating the plurality of patches and the classification results for the patch to infer a classification for the reconstructed tissue slide image, thereby proceeding with learning of the tissue slide classification result; a tissue classification step of generating a patch classification result by using the trained artificial intelligence model to infer a classification for each of the plurality of patches, and integrating the classification results for the plurality of patches and the patch to infer a classification for the reconstructed tissue slide image, thereby generating a tissue slide classification result; a storage step of storing the slide information and the tissue slide image, the plurality of patches, patch information for the patches, the patch classification result, and the tissue slide classification result; and an output step of outputting the patch classification result for each of the plurality of patches and the tissue slide classification result. [Effects of the Invention]

[0023] The tissue pathology reading support device and method according to the embodiment can reduce potential human errors that are common among pathologists, particularly false negatives. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram of a tissue pathology reading support system according to one embodiment. [Figure 2] FIG. 1 is a block diagram of a tissue pathology reading support device according to one embodiment. [Figure 3] Figure 2 shows the process of learning an artificial intelligence model. [Figure 4] 4 is a flowchart of a data preprocessing process for generating patch images in the data preprocessing process of FIG. 3. [Figure 5] 4 shows the operation flow and results of the patch classification unit in FIG. 3. [Figure 6] 4 shows the operation flow and results of the slide tissue image classification unit in FIG. 3. [Figure 7] 3 shows the operation flow and results of the tissue classification unit in FIG. 2. [Figure 8] 3 is a flowchart for comparing the slide analysis results with the slide tissue image classification results via the histopathology reading support device of FIG. 2. [Figure 9] FIG. 2 shows a slide tissue image displayed on the display unit. [Figure 10] The display in Figure 2 displays the specialist reading results, slide preview image, artificial intelligence reading results, and previous result information. [Figure 11] The statistical information and slide analysis results displayed on the display unit in FIG. 2 indicate whether or not there is a mismatch between the slide tissue image classification results. [Figure 12] 10 is a flowchart of a method for assisting in histopathology reading according to another embodiment. [Figure 13] 1 is a block diagram of a computing system according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not relevant to the description are omitted, and similar parts are designated by similar reference numerals throughout the specification.

[0026] Throughout this specification, when a part is said to be "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. Furthermore, when a part is said to "comprise" a certain component, this does not mean excluding other components, but may further include other components, unless otherwise specified, and it should be understood that this does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0027] As used throughout this specification, the terms "about," "substantially," and the like, when presenting the tolerances of manufacturing and materials inherent in the referred meaning, are used to mean a numerical value or a approximation thereof, and are used to prevent unscrupulous infringers from unfairly exploiting disclosures in which precise or absolute numerical values ​​are recited to aid in the understanding of the present invention. As used throughout the specification of the present invention, the terms "step of" or "step of" do not mean "step for."

[0028] In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized by both hardware and software. Also, one unit may be realized by two or more pieces of hardware, and two or more units may be realized by one piece of hardware.

[0029] Some of the operations or functions described herein as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described as being performed by a server may be performed by a terminal, apparatus, or device connected to the server.

[0030] In this specification, the learning process may be expressed as training and the result may be expressed as learning, although either training or learning may be used to refer to either the learning process or the result.

[0031] FIG. 1 is a conceptual diagram of an automated verification system for routine diagnosis of histopathology readings according to one embodiment.

[0032] Referring to FIG. 1, an automated verification system 100 for routine diagnosis of histopathology reading according to one embodiment includes a slide scanner 110, a server computer 120, and a display device .

[0033] When an operator inserts a tissue slide 112, the slide scanner 110 generates a digitized slide tissue image 114, which is uploaded to a server computer 120 where a developed artificial intelligence model 122 generates an output on a display device 130.

[0034] The output may be, for example, lesion location information 132 including the location of the tissue lesion and patient readings 134 including the patient-specific classification results. That is, display device 130 displays lesion location information 132 including the location of the tissue lesion and patient readings 134 including the patient-specific classification results.

[0035] Furthermore, as will be described later, the output further displays whether or not there is a discrepancy between the slide tissue classification result diagnosed by the histopathology support device using an AI model and the pathological diagnosis that is the slide reading result by the pathologist, so it is possible to construct an automatic verification system 100 for diagnoses in the daily work of histopathological reading using an AI model, thereby making it possible to manage the diagnostic quality of daily work during histopathological diagnosis.

[0036] The server computer 120 described above can include a tissue pathology reading support device 200, which will be described with reference to Figures 2 to 10 and 11. The tissue pathology reading support device 200 will be described in detail below.

[0037] FIG. 2 is a block diagram of a tissue pathology reading support device according to one embodiment.

[0038] Referring to FIG. 2, a tissue pathology reading support device 200 according to one embodiment may include an input unit 210, a preprocessing unit 220, a tissue classification unit 230, a storage unit 240, and an output unit 250.

[0039] The input unit 210 receives input of slide information, which is information relating to a tissue slide, and a slide tissue image obtained by digitizing the tissue slide.

[0040] 1, when a tissue slide 112 is input by an operator, the slide scanner 110 generates a digitized slide tissue image 114. The slide tissue image 114 may be a training slide tissue image and a reading slide tissue image. The training slide tissue image is a slide tissue image obtained by digitizing the training slide 112a, and the reading slide tissue image is a slide obtained by digitizing the reading slide 112b.

[0041] The input unit 210 may input a slide tissue image of tissue from outside the histopathology reading support device 200. A slide tissue image is an image of tissue.

[0042] The tissue may include human or animal cells. As an example, the tissue may be a biopsy tissue extracted from a human organ (e.g., stomach, colon, small intestine, liver) to confirm the presence of cancer cells (e.g., stomach cancer, colon cancer).

[0043] The tissue can be photographed using various types of photographing devices (e.g., a camera, a video camera, a scanner), and the format of the image of the tissue can be determined in various ways, such as JPG, GIF, PNG, BMP, etc.

[0044] In this case, the tissue may be hematoxylin and eosin stained tissue, which stains cell nuclei blue and extracellular matrix and cytoplasm pink.

[0045] Meanwhile, the training slide tissue image may further include annotation information. The annotation information is information indicating which region of the tissue image corresponds to which group. In one example, the annotation information may indicate that a specific region of the tissue image is an abnormal region where a lesion (e.g., cancer) is present, and the remaining region is a normal region where no lesion is present. For example, as will be described later with reference to FIG. 4, if a lesion is present, the training slide tissue image may display the location of the lesion as a single closed curve with a different color for each lesion as annotation information.

[0046] The annotation information may be used as information indicating the correct answer when training the artificial intelligence model 231 that classifies slide tissue images.

[0047] The preprocessing unit 220 may divide the slide tissue image 114 input from the input unit 210 into a plurality of patches 116 .

[0048] The size of each of the plurality of patches 116 may be smaller than the tissue image, for example, the size of each of the plurality of patches 116 may be equal to n×n pixels (e.g., 256×256, 128×128).

[0049] In another example, the sizes of the multiple patches 116 may be different from each other. Some of the multiple patches 116 may be patches with a size of n×n pixels (e.g., 256×256) and some may be patches with a size of m×m pixels (e.g., 128×128), where m is less than n.

[0050] The tissue classification unit 230 generates a patch classification result by inferring the classification of each of the multiple patches 116 using the trained artificial intelligence model 231, and generates a slide tissue classification result by integrating the multiple patches 116 and the patch classification results to infer the classification of the reconstructed slide tissue image.

[0051] When a lesion is present, the trained artificial intelligence model 231 divides the training slide tissue image 114 displaying the lesion site into multiple patches 116 using the preprocessing unit 220, and infers the classification of each of the multiple patches 116, thereby proceeding with learning the patch classification results, and can proceed with learning the slide tissue classification results by integrating the multiple patches 116 and the patch classification results to infer the classification of the reconstructed slide tissue image 114.

[0052] The artificial intelligence model 231 may be a deep learning model. In this specification, a deep learning model may be a model in which artificial neural networks are stacked in multiple layers. The deep learning model can be realized as a model that automatically learns the features of each image by training a deep neural network consisting of multiple networks with a large amount of data, thereby training the network in a way that minimizes the objective function, i.e., the error in prediction accuracy.

[0053] For example, the deep learning model may be a convolutional neural network (CNN), which can be implemented in various forms, such as Visual Geometry Group (VGG) network, Inception (GoogleNet), ResNet, and DenseNet.

[0054] However, the deep learning model described in the embodiments of the present invention is not limited to CNN, and other types of deep learning models that may be used now or in the future (e.g., Deep Hierarchical Network (DHN), Convolutional Deep Belief Network (CDBN), Deconvolutional Deep Network (DDN), Recurrent Neural Network (RNN)) can be used.

[0055] The deep learning model may be implemented via a deep learning framework. The deep learning framework provides commonly used functions in the form of libraries when developing a deep learning model, and serves to support the appropriate use of system software or hardware platforms. In this embodiment, the deep learning model may be implemented using a currently publicly available or future publicly available deep learning framework.

[0056] Executing training of a deep learning model means adjusting the parameters of the deep learning model so that the predicted values ​​of the inputs of the deep learning model are as close as possible to the actual values ​​(ground truth). The tissue classification unit 230 inputs the patch 116 and the reconstructed slide tissue image (reconstructed WSI) into the deep learning model 231, compares the predicted values ​​of the deep learning model with the actual values, and adjusts the parameters of the deep learning model 231 repeatedly, thereby improving the prediction accuracy of the deep learning model.

[0057] At this time, the tissue classification unit 230 can adjust the parameters of the deep learning model 231 in a direction that minimizes the loss function value of the input patch 116 and the reconstructed slide tissue image.

[0058] The loss function is a function that calculates how similar the predicted type of the target patch is to the actual type of the target patch. The loss function can be, for example, the mean absolute error (MAE), the mean squared error (MSE), the root mean squared error (RMSE), the binary cross-entropy, or the categorical cross-entropy.

[0059] There may be two or more trained artificial intelligence models 231, and the slide information may include an organ in the slide tissue image. In this case, the tissue classification unit 230 can select one of the two or more artificial intelligence models according to the organ in the slide tissue image to classify the tissue image. Below, the trained artificial intelligence model 231 will be described as an example of an artificial intelligence model trained on the organs of the stomach and large intestine. However, in addition to the example stomach and large intestine models, models of other organs (such as the breast, prostate, and skin) may also be further developed in a similar manner and applied as a routine quality control system in the same manner.

[0060] In embodiments of the present invention, patch classification results and slide tissue classification results can be classified in a variety of ways.

[0061] For example, patch classification results and slide tissue classification results may be classified into two groups or classes: malignant and non-malignant, or neoplastic and non-neoplastic.

[0062] As another example, patch classification results and slide tissue classification results may be classified into three groups: malignant, dysplasia, and non-neoplastic.

[0063] As another example, patch classification results and slide tissue classification results may be classified into four groups: malignant, dysplasia, uncategorized, and non-neoplastic.

[0064] As an example, the malignant group may be defined as a group representing malignant neoplasms, including adenocarcinoma, suspicious for adenocarcinoma, suggestive of adenocarcinoma, high-grade lymphoma, and other carcinomas.

[0065] The dysplasia group can be defined as a group showing dysplasia including tubular adenoma with any grade of dysplasia.

[0066] The non-neoplastic type can be defined as a group that exhibits non-neoplastic benign lesions (eg, gastritis, polyps, etc.).

[0067] The uncategorized group can be defined as a type that represents the remaining lesions that do not fall into the three previous groups, such as atypical glandular proliferations, neuroendocrine tumors, submucosal tumors, low-grade lymphomas, and stromal tumors.

[0068] In this case, a portion of the patch classification results and the slide tissue classification results may be defined as a normal group, and the remaining portions may be defined as an abnormal group. For example, if the patch classification results and the slide tissue classification results are classified into four groups, malignant, dysplasia, uncategorized, and non-neoplastic, malignant, dysplasia, and uncategorized may be defined as the abnormal group, and non-neoplastic may be defined as the normal group.

[0069] The storage unit 240 can store slide information, slide tissue images, multiple patches, patch information related to the patches, patch classification results, and slide tissue classification results. The storage unit 240 may temporarily store some of the information and then automatically delete it, or may store the information and then continue to store it until a deletion request is received.

[0070] The output unit 250 can output the patch classification results for each of the plurality of patches 116 and the slide tissue classification results. The output unit 250 can output not only the patch classification results for each of the plurality of patches 116 and the slide tissue classification results, but also all of the information input via the input unit 210 and all of the information stored in the storage unit 240.

[0071] In line with the development intention of developing a tissue classification model using an artificial intelligence model231 for use in quality control in daily work in real life, the principle of data preparation is to use the data as is, preserving real-life inter-observer variability and daily artifacts, in order to thoroughly reflect the situation in the field of practice, rather than artificially preparing a well-refined slide tissue image dataset to ensure the high accuracy of the model231.

[0072] Slide quality cannot be manipulated to ensure the highest model accuracy. Rather, common artifacts—slide cuts, poor staining, poor fixation, air bubbles, etc.—can be used for training without artificial modification (such as reproducing high-quality slides specifically for training). However, slides with poor quality that make them difficult for a pathologist to read can be excluded.

[0073] In other words, the training slide tissue images can be slide tissue images scanned by a slide scanner without any artificial improvement.

[0074] Scan quality may not be artificially adjusted. Scanner manufacturers report an average defect rate of 1–3% requiring rescanning. Most other defects are minor and do not interfere with reading. Examples include narrow focus outs, overlapping tile errors, and edge cutting of the borders of some scanned tissues. To reflect these minor everyday issues, artificial improvements, such as rescanning to obtain high-quality slide tissue images for training purposes, may not be performed. However, scan defects that are difficult for pathologists to read due to noise, as shown in Figure 2, may be excluded.

[0075] The patch classification results can be defined as a four-classification model into four groups to classify all possible diagnoses (Table 1). However, by definition, "Group U," which has the highest heterogeneity within the group, can significantly impede model performance. Therefore, a three-classification model excluding "Group U" may be adopted. Cases that fall into Group U may ultimately be excluded from the training data used to develop the three-classification model. However, as a next step, the model can be upgraded to a four-classification model by redefining Group U in Table 1 to include only NET (grade 1 or 2) cases. This allows for practical application.

[0076] Table 1 TIFF2025529864000002.tif99155 The slide histology images corresponding to each classification can be annotated independently by each pathologist. A single closed curve can be drawn using a red line for group M and a blue line for group D. If the boundary between the lesion and normal area is unclear, an outline can be drawn to include only the clearly defined lesion (including both epithelium and stroma). Only patches within the single closed curve can be used as data for each classification, and all other parts can be deleted.

[0077] For Group N, slide tissue images that meet the above definition can be reviewed and confirmed by a pathologist, and can be used as data after immediately generating patches without additional annotation. In this case, non-lesion portions of Group M and Group D slides do not need to be used as Group N data. Non-neoplastic glands or crypts should be avoided as much as possible within a single closed curve, and non-tissue components such as necrosis, ulcer detritus, extracellular mucin pools, and blood should be avoided as much as possible. If necessary, annotations can be reviewed by another pathologist and, in some cases, revised or removed.

[0078] The slide tissue images 114 are input into the artificial intelligence model 231, which must classify groups of slide tissue images and predict the results. The slide tissue images 114 are converted and processed into a form that can be recognized by the artificial intelligence model 231, and the artificial intelligence model is configured to process without information loss in 1) the learning step and 2) the prediction step.

[0079] Since the slide tissue image 114 contains gigapixel-level information, it is not suitable to use it all at once in a deep learning model. Therefore, the slide tissue image 114 containing gigapixel-level information can be converted and processed by the artificial intelligence model 231.

[0080] We build a model based on how the slide tissue image 114 is converted into a patch format and processed. As noted in many convolutional neural network (CNN)-based slide tissue image processing studies, processing the entire slide tissue image at once can result in a loss of information and resource consumption due to the complexity of CNNs. Therefore, we train the model according to the operating procedure shown in Figure 3.

[0081] The output unit 250 can display the patch classification results in a unique color according to the location of the lesion in each patch, and can display the slide tissue classification results in at least one of letters, numbers, and colors.

[0082] The reconstructed slide tissue image is reconstructed by combining multiple patches displayed in unique colors according to the location of the lesion, and the output unit 250 can display the slide tissue analysis results written in letters or numbers along with the reconstructed slide tissue image.

[0083] The input unit 210 can input a pathological diagnosis, which is the result of a slide reading by a pathologist, when the pathologist examines a slide corresponding to a slide tissue image using a microscope or by viewing the slide tissue image on a screen. The display unit 250 can further display, along with the pathological diagnosis, which is the result of the slide reading by the pathologist, whether or not there is a discrepancy between the slide tissue image classification result by the tissue classification unit and the pathological diagnosis, which is the result of the slide reading by the pathologist. This allows the artificial intelligence model 231 to be used to build an automatic verification system for diagnoses in the daily work of pathological tissue reading, and to manage the quality of diagnoses in the daily work.

[0084] Specifically, the information input to the input unit 210 may be the content of the slide read by the pathologist, i.e., "pathological diagnosis" information (text information). This pathological diagnosis information may be "classified" into one of three classes, M, D, or N, according to a classification rule. The artificial intelligence model may independently predict one of the three classes, M, D, or N. The display unit 250 displays whether the "classification result based on the pathological diagnosis information (text information)" matches or does not match the corresponding "classification result based on the artificial intelligence model (image information)." If the display unit 250 further displays whether there is a match between the slide tissue image classification result and the pathological diagnosis, which is the slide reading result by the pathologist, for the two or more cases, the display unit 250 may preferentially display a positive result for either the slide tissue image classification result or the pathological diagnosis, which is the slide analysis result by the pathologist.

[0085] Figure 3 shows a process for training the artificial intelligence model of Figure 2. Figure 4 is a flowchart of a data preprocessing process for generating patch images in the data preprocessing process of Figure 3.

[0086] 3 and 4, the process includes a data preprocessing step S310, a patch classification step S320, and a slide tissue image classification step S330 to train the artificial intelligence model used in the tissue classification unit 230.

[0087] In the data preprocessing step S310, the preprocessing unit 220 organizes the data and converts it into a number of patches in a form that can be learned by the artificial intelligence model.

[0088] The data preprocessing step S310 proceeds with training of the model 231 based on the actual value information generated as described above and the labeled information as shown in Table 1. The training slide tissue images WSI of group N may be set as is, while groups M and D may be set as data including annotation information, for example, displaying lesion areas in color.

[0089] To train the training slide tissue image WSI, two deep neural network (DNN) models are mainly required. One model may be used for the patch classification unit 232, and the other model may be used for the slide tissue image classification unit 234. Therefore, two types of data, "patch image data" and "slide tissue image data," are prepared for training each model.

[0090] First, the slide tissue image data is checked for the scan configuration values ​​of the collected slide tissue images WSI, and slide tissue images with different values ​​are excluded from the dataset. This is to prevent slide tissue images (WSI) taken under different conditions from causing errors in the model. The final selected slide tissue image dataset is shown in Table 1.

[0091] For Group N, rectangular patches with a size of, for example, 256 × 256 pixels are generated from the training slide tissue images (WSI). Regarding patch size, the trade-off between user convenience and performance can be defined through interviews. For example, patches with a large pixel size may perform well, but may not be suitable for providing an explanation of the lesion location information provided by the user interface.

[0092] Then, for groups M and D, based on the annotation information, only patches that are inside the annotation can be selected and saved. Patches outside the annotation may not be included in the dataset.

[0093] Furthermore, for accurate model performance evaluation and feedback, the training, evaluation, and testing sets are constructed based on the tissue slide images (WSI) to which the patch data belongs. If the training, evaluation, and testing sets are constructed without considering the relationship between the tissue slide images (WSI) and the patches, patch images generated from the same tissue slide images (WSI) in the training set will be distributed simultaneously in the training data set and the evaluation and testing data sets, which may cause the model to act as a cheat during the testing phase, making accurate evaluation difficult. Finally, to minimize bias generation in the tissue classifier 230, patch data may be randomly sampled from the generated training pool.

[0094] FIG. 5 shows the operation flow and results of the patch classification unit in FIG.

[0095] 3 and 5, the patch classifying step S320 performs learning based on the labeling information of each patch generated by the patch classifying unit 232 from one training slide tissue image WSI.

[0096] The patch classification step S320 uses a CNN-based DNN architecture to train the patch classifier 232. For example, a model can be trained using DenseNet201, which exhibits high performance among DNN architectures, to classify patch images. Set training data can be input to the trained DenseNet201 model. The patch classification model can be trained to accurately infer three correct labels, for example, groups M, D, and N, for each patch image.

[0097] The patch classification results generated by the patch classifier 232 are intended for two purposes: 1) use as elements of inference by the slide tissue image classifier 234, and 2) provision of explanations in a user interface. In particular, slide tissue image WSIs can be trained based on the model learned in this step for use as elements of the slide tissue image classifier 234. The patch classifier 232 can generate patch classification results in the form of distributions for each group, which are used as key information.

[0098] FIG. 6 shows the operation flow and results of the slide tissue image classification unit in FIG.

[0099] 3 and 6, in the step S330 of classifying slide tissue images, the training slide tissue image classifier 234 trains an AI model that generates patch classification information for the slide tissue images WSI by integrating information about the slide tissue images WSI. The AI ​​model training method is equally applied to each AI model of two or more organs to construct the models.

[0100] The slide tissue image classification unit 234 performs learning so as to infer the learning data, which has as elements the patch classification results, which are fragmented information of the slide, as the correct label of the slide tissue image WSI.

[0101] The learning step of the slide tissue image classifier 234 involves training the slide tissue image classifier 234 through a series of processes based on 1) the patchmaker model used in the data preprocessing step and 2) the patch classifier 232 trained in the model training. This is to efficiently utilize the resources required to classify slide tissue images containing gigapixels of information, and each model is designed to function organically as if it were a single model.

[0102] During the learning process of the slide tissue image classification unit 234, the slide tissue image WSI is converted into a patch image by the patch maker, and the converted patch image is reconstructed based on the position information and undergoes a learning process in the slide tissue image classification unit 234.

[0103] The training process of the slide tissue image classifier 234 mainly includes three steps: a patch maker step, a patch classification step, and a slide tissue image training step.

[0104] In the patch maker step, the slide tissue image to be learned is converted into multiple patch images. In order to preserve the position information of each patch image during the conversion process, the index and position information of the converted patch, as well as the slide tissue image from which it was generated, are recorded in the memory unit 240.

[0105] Next, in the patch classification step, the learned patch classifier 232 infers what classification information each patch constituting one slide tissue image has.

[0106] The inference information for each patch image is stored in the memory unit 240 according to the patch index. Therefore, the memory unit 240 records the index, position information, and classification information of each patch image for one slide tissue image in an integrated manner. Finally, to train the slide tissue image, a reconstructed slide tissue image (reconstructed WSI) is generated by integrating the patch images stored in the memory unit 240. The reconstructed slide tissue image contains the feature information of the slide tissue image in a summarized or compressed form, so that the information can be processed by CNN as if it were a single image. Therefore, a CNN group classification network is constructed with the reconstructed slide tissue image as input, and the model is trained to appropriately infer three correct labels, for example, groups M, D, and N.

[0107] Referring to Figure 6, the process is shown in which one slide tissue image is converted into multiple images by the patch maker, distribution information is generated by the patch classification unit 232, and it is converted into a reconstructed slide tissue image.

[0108] It can be seen that the reconstructed slide tissue images are composed of shapes similar to typical rectangular images. The CNN model trained to classify the reconstructed slide tissue images is used as the slide tissue image classifier 234. The above series of processes is performed for each of the gastric biopsy and colon biopsy, and two models are generated.

[0109] FIG. 7 shows the operational flow and results of the tissue classification unit in FIG. 2 that classifies tissues using an artificial intelligence model.

[0110] In this step, the entire prediction framework is operated based on the model trained in the previous step to generate results and additional information. The prediction method can be configured in the same way as the learning sequence described above. The developed models are constructed and trained separately for gastric biopsies and colon biopsies. Each model can be operated in the same sequence.

[0111] The operation process of the trained slide tissue image classifier model consists of the same sequence of operations as the training operation.

[0112] When a new slide tissue image WSI is input into the tissue classification unit 230, the patch maker generates multiple patch images from the input slide tissue image and patch information such as associated indexes and position information, and records them in the memory unit 240.

[0113] Each patch image is input to the patch classifier 232, where a group is inferred by a model, and patch classification results are generated and stored in the memory 240. Finally, the slide tissue image classifier 234 integrates the patch image information with the patch classification information in the memory 240 to generate a reconstructed slide tissue image and classify the slide tissue image.

[0114] The process includes integrating each patch classification information and position information to convert it into a reconstructed slide tissue image, inputting it into the slide tissue image classification unit 234, generating a slide tissue image classification result, and storing it in the memory unit 240.

[0115] Therefore, three types of information are generated and stored in the memory unit 240, which may be patch image information, patch classification information, and slide tissue image classification information. The patch image information includes a patch index, a patch image, patch position information, etc. The patch classification information means a group inference result generated from a patch classification model, and the slide tissue image classification information means group information inferred by the slide tissue image classification model for one slide tissue image.

[0116] Image classification information for routine quality control applications is configured to provide the user with a combination of the above three pieces of information: patch image information, which provides a visualization of the original slide image in a user interface and location information for each patch, and provides visual information about the slide image in the user interface;

[0117] The patch classification information is used to provide classification information (heat map) for each patch on the slide image displayed in the user interface, which helps provide a basis for the classification results and provides lesion location information for quality control. The display unit 250 visually provides the location of the lesion based on the slide tissue image classification information in the memory unit 240 and the patch image prediction information.

[0118] FIG. 8 is a flowchart showing a process of comparing the results of slide reading by a pathologist with the results of classification of slide tissue images by an artificial intelligence model via the histopathology reading support device of FIG.

[0119] The histopathology reading support device 200 in FIG. 2 scans daily reading slides and converts them into slide tissue images, while simultaneously supporting a series of processes from AI reading to visualization and information processing of the results.

[0120] The histopathology reading aid 200 can perform daily scans and artificial intelligence predictions on all slides read after microscopic reading.

[0121] 2 and 8, the pathologist proceeds with the reading under a microscope. The slides are not prepared separately according to specimens such as stomach biopsy and colon biopsy, but are simply mixed in order of reception number without distinguishing between organs and provided to the pathologist. Therefore, the pathologist also reads the slides in order of reception number without distinguishing between organs. The input unit 210 inputs the slide diagnosis results obtained by the pathologist reading the slides corresponding to the slide tissue images under a microscope or reading the slide tissue images through a screen (S810).

[0122] For example, the input unit 210 is an input device such as a keyboard or a mouse, and the analyzed slide reading results can be input by the pathologist typing or by clicking the mouse.

[0123] The read slide is scanned by the scanner 110. The slide tissue image generated by the scanner 110 is a file with a specific extension, for example, an mrxs file, and is stored in a path specified by the scanner program. When a new slide tissue image file is generated in that path, it is copied to the storage unit 250.

[0124] The histopathology reading support device 200 searches the memory unit 240 for the scanned slide name (pathology number), and calls and drives one of two or more artificial intelligence models corresponding to each organ according to the keywords specified in the specimen information of the pathology report (see Table 2). The artificial intelligence model reads the slide tissue image file, predicts the patch classification results and the slide tissue image classification results, and stores each prediction result in the memory unit 240 (S820).

[0125] Table 2 TIFF2025529864000003.tif26144 The display unit 250 may further display whether or not there is a mismatch between the slide tissue image classification result by the tissue classification unit 230 and the slide classification result by the pathological diagnosis, along with the slide pathological diagnosis result by the pathologist (S830).

[0126] Specifically, scanned slides are organized into groups based on pathological diagnosis and groups based on artificial intelligence prediction, and the pathologist can access the tissue pathology reading support device 200 the next day (or a few hours later) to check the agreement between each group and prioritize review of discrepant cases.

[0127] This allows a pathologist to "review" discrepant cases and, if necessary, "correct" the initial results (pathological diagnostic errors due to human error).

[0128] The pathologist determines whether there is a pathological diagnosis error in the discrepancy case (S840), and if the pathological diagnosis is incorrect in the discrepancy case, the pathologist can use the input unit 210 to input the corrected pathological diagnosis and the cause of the created diagnostic error (S850).

[0129] Conversely, in cases of mismatch, if the slide tissue image classification result by the tissue classification unit 230 is incorrect, as described above, a learning slide tissue image including annotations can be created for the slide tissue image, and the artificial intelligence model can be further trained (S860).

[0130] Figure 9 shows a slide tissue image displayed on the display unit of Figure 2. Figure 10 shows the specialist reading results, slide preview image, artificial intelligence reading results, and previous result information displayed on the display unit of Figure 2. Figure 11 shows statistical information displayed on the display unit of Figure 2 and whether there is a discrepancy between the slide reading results and the slide tissue image classification results.

[0131] As shown in Figure 9, the color of the heat map for each patch displayed on the slide tissue image represents the group results inferred from the corresponding patch.

[0132] The histopathology reading support device 200 can search for scanned slide data by receipt date, inspection date, and scan date via the inspection results / statistics page.

[0133] First, the test results page provides a list of slide information for each slide, including the receipt date (Receipt Date), inspection date (Inspect Date), pathology number (slide name), patient name, classification result by pathology diagnosis (Classification by Pathologic Diagnosis), classification result by AI model (Classification by AI Prediction), whether or not there was a match (Concordance), AI model name (AI model by anatomy), and pathologist (reader).

[0134] This list allows users to view the histological image and heat map of the slide. It also provides a single-magnification (0.5x) AI model heatmap thumbnail and text information such as the pathological diagnosis, notes, and previous pathological diagnosis. It is extremely difficult for a pathologist to review 100% of all scanned slide histological images solely for quality control purposes while performing their main daily tasks. Therefore, a filtering function can be implemented to separately search for cases where the classification results based on the pathological diagnosis and the AI ​​prediction do not match.

[0135] When a pathologist reviews a slide tissue image that they have read themselves, the relevant line is highlighted, allowing them to intuitively know whether a review has been performed.

[0136] In addition, the statistics page allows users to check the predicted performance and distribution for each AI model (Gastric / Colorectal), and by clicking on each cell in the table, they can review the corresponding slide tissue image. This allows pathologists to selectively review cases that meet their desired conditions not only through the test results page but also through the statistics page.

[0137] Search for scanned slides and double-click a specific slide row or column to display the slide histology image viewer for that slide. You can drag the screen to move the image position, use the mouse wheel to zoom in / out on the slide histology image, and use the minimap in the upper right corner to see the current position displayed on the screen.

[0138] In addition, if there are multiple slides in one specimen (e.g., recuts, serial or deeper sections, or more than one block), a list of related slides is provided in the upper left corner of the slide histology image viewer so that related slide histology images can be displayed together. In addition, various slide histology image-related functions are available via the function buttons in the lower left corner.

[0139] Because artificial intelligence models are based on black-box technology, it is extremely difficult to understand what features of the data the model has learned and why it has derived its prediction results. Therefore, it is difficult to unconditionally trust the results of artificial intelligence predictions. In practice, proper justification for the results and verification of their validity and reliability are required. This allows the slide tissue image viewer to visualize both slide tissue image prediction information and patch prediction information on a single slide tissue image.

[0140] Referring again to Figure 9, the slide tissue image prediction information is displayed as a text label below the minimap in the upper right corner of the slide tissue image viewer. The patch prediction information is implemented in the form of a heat map, where prediction information is displayed on each patch in the slide tissue image based on the location information of each patch.

[0141] The patches in group M may be marked with red heat, those in group D with blue heat, and those in group N with no heat. The same colors used for slide annotations during model training may be used. In particular, the heat map may be implemented so that it can be simply masked on and off by right-clicking while viewing a slide through the histology image viewer. The basic functions of the histology image viewer, such as zooming and rotation, may be used to simultaneously view the histology image and heat map at all magnifications (0.5x to 40x). Therefore, when reviewing histology images, pathologists can intuitively compare how the AI ​​model inferred each part of a single histology image by frequently masking the heat map at all magnifications.

[0142] Research into the sophistication of AI classification models is still ongoing, and continuous feedback from pathologists on the slide- and patch-level performance of AI models applied in practice is necessary to measure and improve their performance. Therefore, a system can be created that allows users to record qualitative evaluations of the AI ​​model's heat map and predictions in each slide-and-tissue image list. This evaluation information can be used as additional data to enhance future model performance.

[0143] According to the classification in Table 2, if the pathological diagnosis (text information) contains relevant keywords, it is classified into each group, and this classification result is compared 1:1 with the classification result predicted by the artificial intelligence model, which serves as the basis for determining whether it matches or does not match.

[0144] The classification priority for each keyword can be M>U>D>N. However, in the case of "sessile serrated adenoma / polyp," "sessile serrated adenoma," and "sessile serrated lesion," they can take priority over keywords in the D category.

[0145] For example, diagnoses such as "Adenocarcinoma, moderately differentiated," "Neoplastic lesion, suspicious for malignancy," "adenoma, high-grade dysplasia," "Tubulovillous adenoma, low to focal high-grade dysplasia," "with focal carcinomatous change," "adenoma, grade uncertain," "Malignant neoplasm," "Atypical glandular proliferation, favor neoplastic," and "Neuroendocrine carcinoma" include the "italicized keywords" and are ultimately classified with the "bolded keywords." Diagnoses such as "Tubular adenoma, low-grade dysplasia," "Atypical glandular proliferation, favor dysplasia," "glandular proliferation, indefinite for dysplasia," and "glandular proliferation, undetermined significance" are classified as D according to the above rules. "Small cell nests with neuroendocrine features" is also classified as U, along with "Neuroendocrine tumor, grade 1 (carcinoid tumor)."

[0146] According to the classification rules, if the pathological diagnosis of "Tubulovillous adenoma, low to focal high-grade dysplasia" is classified as "M," but the AI ​​prediction based on the slide histology image is "D," and the two cases differ, the case is marked as "discordance" and subject to review by a pathologist. In the case of ambiguous diagnoses such as "Atypical glandular proliferation, favor neoplastic," "Glandular proliferation, undetermined significance," and "Glandular proliferation, indefinite for dysplasia," the pathological diagnosis classification is displayed as "M, D, D, respectively," but the slide histology image prediction may be M, D, or N depending on the findings of each case. These atypical cases are also marked as "discordance" with the AI ​​prediction and can be reconsidered by a pathologist.

[0147] Furthermore, as mentioned above, in atypical cases, serial cut or recut slides are frequently created, and in such cases, the AI ​​prediction results may differ for each slide tissue image. For example, the prediction for the original slide may be D, one of the serial cuts may be D, another may be M, and the final deeper cut may result in the lesion disappearing entirely and resulting in N. In this way, when the AI ​​predictions for the serial slides created from one block are different, the final predictions may be expressed in the order M>U>D>N.

[0148] In the example above, the AI ​​prediction may ultimately be labeled "M." If the pathological diagnosis is "TA, LGD," it would be labeled "D," which would also qualify as a case of "mismatch." If the goal is simply to improve the agreement (accuracy) between the specialist's diagnosis and the AI ​​prediction, the system can be set so that the final prediction result for each cut slide is classified as D, D, M, or N, the highest possible D. However, to meet the purpose of post-analytic quality control and the rapid detection of false negatives, it is possible to prioritize more clinically significant results instead of the AI ​​prediction result with the highest weighting.

[0149] Because of this system configuration, the accuracy of AI predictions is inevitably lower than that of other AI models for diagnostic support. However, because reviewing more cases than necessary is inefficient, it is necessary to ensure an ideal accuracy level that balances efficiency and practical functionality.

[0150] The tissue pathology reading support device 200 of one embodiment can reduce the pathologist's everyday potential human errors, particularly false negatives, without infringing or threatening the pathologist's diagnostic responsibility, while maintaining the optical microscope-based workflow - rather than making a better diagnosis than the pathologist - and can develop an artificial intelligence model with reliable performance, which can be applied in practice as "a tool of daily fast QC" as the most realistic and appropriate method that can be applied to the routine practice of gastrointestinal endoscopic biopsy reading.

[0151] Furthermore, the tissue pathology reading support device 200 according to one embodiment effectively detects false negatives due to potential human error in daily work (or practice) and takes necessary measures immediately, thereby ultimately preventing patients who require treatment from being overlooked.

[0152] Furthermore, the histopathology reading support device 200 according to one embodiment can reflect the artifacts and variables of the actual practice so that it can be directly applied to daily practice.

[0153] Furthermore, the histopathological reading support device 200 according to one embodiment is not intended to provide criteria for accurate differentiation between "invasive cancer and dysplasia" or "neoplastic changes and reactive changes," etc., and can respect the inherent authority of each pathologist.

[0154] FIG. 12 is a flowchart of a method for assisting in histopathological reading according to another embodiment.

[0155] Referring to FIG. 12, the method 1000 for assisting in histopathological reading includes an input step S1010, a preprocessing step S1020, a learning step S1030, a tissue classification step S1040, a storage step S1050, and an output step S1060.

[0156] In an input step S1010, slide information and slide tissue images are input.

[0157] As described with reference to FIG. 4, the pre-processing step S1020 divides the input slide tissue image into a number of patches.

[0158] As described with reference to Figures 5 and 7, in the learning step S1030, if a lesion is present, the artificial intelligence model divides the learning slide tissue image displaying the lesion site into multiple patches, infers the classification of each of the multiple patches, and learns the patch classification results, and then integrates the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image, thereby proceeding with learning of the slide tissue classification results.

[0159] As described with reference to Figure 7, the tissue classification step S1040 generates a patch classification result by using a trained artificial intelligence model to infer the classification of each of the multiple patches, and generates a slide tissue classification result by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image.

[0160] A storage step S1050 stores the slide information, the slide tissue image, the plurality of patches, the patch information about the patches, the patch classification result, and the slide tissue classification result.

[0161] An output step S1060 outputs the patch classification result for each of the plurality of patches and the slide tissue classification result.

[0162] If a lesion is present in the training slide tissue image, the lesion site can be displayed as a single closed curve in a different color for each lesion.

[0163] The training slide tissue images may be slide tissue images scanned by a slide scanner, without any artificial enhancement.

[0164] The trained artificial intelligence models may be two or more, and the slide information may include organs in the slide tissue images. In the tissue classification step S1040, one of the two or more artificial intelligence models may be selected according to the organs in the slide tissue images to classify the tissue images. As described above, the trained artificial intelligence models have been exemplarily described as artificial intelligence models trained on the organs of the stomach and large intestine. However, in addition to the exemplarily described stomach and large intestine models, models of other organs (such as the breast, prostate, and skin) may also be further developed in a similar manner and applied as routine quality control systems in the same manner.

[0165] As described with reference to Figures 7 and 9, in the output step S1060, the patch classification results may be displayed in a unique color depending on the location of the lesion in each patch, and the slide tissue classification results may be displayed using letters, numbers, or colors.

[0166] As described with reference to Figure 7, the reconstructed slide tissue image is reconstructed by combining multiple patches displayed in unique colors according to the location of the lesion, and in output step S1060, the slide tissue analysis results, expressed in letters or numbers, may be displayed together with the reconstructed slide tissue image.

[0167] In input step S1010, a pathologist inputs a pathological diagnosis result obtained by reading a slide corresponding to the slide tissue image through a microscope or by reading the slide tissue image through a screen.

[0168] Then, in the display step S1060, as described with reference to FIG. 10, together with the pathological diagnosis result, in the tissue classification step S1040, it is possible to further display whether or not there is a discrepancy between the slide tissue image classification result and the classification result based on the pathological diagnosis.

[0169] Specifically, what is input in input step S1010 may be the content read by a pathologist after analyzing a slide, i.e., "pathological diagnosis" information (text information). This pathological diagnosis information is "classified" into one of three groups, M, D, or N, according to classification rules. The artificial intelligence model can then independently predict one of the three groups, M, D, or N. In display step S1060, whether the "classification results based on pathological diagnosis information (text information)" and the corresponding "classification results based on the artificial intelligence model (image information)" match or do not match is displayed.

[0170] In the tissue classification step S1040, whether or not there is a discrepancy between the slide tissue image classification result and the classification result based on the pathological diagnosis is additionally displayed. This allows the artificial intelligence model to be used to build an automatic verification system for diagnosis in the daily work of tissue pathology reading, thereby managing the quality of diagnosis in daily work.

[0171] FIG. 13 is a block diagram of a computing system 1100 according to an embodiment of the present invention.

[0172] Referring to FIG. 13, a computing system 1100 may include a memory 1110 and a processor 1120 .

[0173] The memory 1110 may store the slide tissue image of the tissue and the multiple patches into which the slide tissue image is divided, or may store them separately in separate mass storage servers, etc. The memory 1110 may be a volatile memory (e.g., SRAM, DRAM) or a non-volatile memory (e.g., NAND Flash).

[0174] The processor 1120 can, for the artificial intelligence model, divide a training slide tissue image showing the lesion site into a plurality of patches when a lesion is present, infer a classification for each of the plurality of patches, thereby proceeding with learning of the patch classification result, and can integrate the plurality of patches and the patch classification results to infer a classification for the reconstructed slide tissue image, thereby proceeding with learning of the slide tissue classification result. The processor 1120 can generate a patch classification result by inferring a classification for each of the plurality of patches using the trained artificial intelligence model, and can generate a slide tissue classification result by integrating the plurality of patches and the patch classification results to infer a classification for the reconstructed slide tissue image.

[0175] The tissue pathology reading support device 200 may be configured as a computing system 1100 shown in FIG. 13, and may be configured with a storage for storing scan files (WSI images) and a GPU server equipped with a GPU processor and general-purpose memory, but the present invention is not limited to this.

[0176] The tissue pathology reading support device 200 may be implemented by a computing device including at least a processor, memory, a user input device, and a presentation device. The memory is a medium that stores computer-readable software, applications, program modules, routines, instructions, and / or data that are coded to perform specific tasks when executed by the processor. The processor may read and execute the computer-readable software, applications, program modules, routines, instructions, and / or data stored in the memory. The user input device may be a means by which a user inputs instructions to the processor to perform a specific task or data required to perform a specific task. The user input device may include a physical or virtual keyboard or keypad, key buttons, a mouse, a joystick, a trackball, a touch-sensitive input means, a microphone, etc. The presentation device may include a display, a printer, a speaker, a vibrating device, etc.

[0177] The computing device may include a variety of devices such as smartphones, tablets, laptops, desktops, servers, clients, etc. The computing device may be a single standalone device or may include multiple computing devices operating in a distributed environment of multiple computing devices cooperating with each other via a communications network.

[0178] On the other hand, the computing device may not be a classical computing device but a quantum computing device. A quantum computing device performs operations in units of qubits rather than bits. A qubit can have a state in which 0 and 1 are superposed simultaneously, and if there are M qubits, it can represent 2^M states simultaneously.

[0179] To perform quantum operations, quantum computing devices use various types of quantum gates (e.g., Pauli / Rotation / Hadamard / CNOT / SWAP / Toffoli) that receive one or more qubits as input and perform a specified operation, and quantum gates can be combined to form quantum circuits that perform specific functions.

[0180] Quantum computing devices can use quantum artificial neural networks (e.g., QCNN, QGRNN) that can perform functions performed by existing artificial neural networks (e.g., CNN, RNN) at faster speeds using fewer parameters.

[0181] Furthermore, the above-described tissue pathology reading assistance method 1000 may be executed by a computing device having a processor and including memory storing encoded computer-readable software, applications, program modules, routines, instructions, and / or data structures, etc., that, when executed by the processor, can perform the tissue pathology reading assistance method 1000.

[0182] The above-described embodiments can be implemented by various means, for example, by hardware, firmware, software, or a combination thereof.

[0183] When implemented in hardware, the tissue pathology reading assistance method 1000 according to the present embodiment can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, or microprocessors.

[0184] For example, the tissue pathology interpretation support method 1000 according to the embodiment may be realized by an artificial intelligence semiconductor device in which neurons and synapses of a deep neural network are implemented by semiconductor elements. In this case, the semiconductor elements may be currently used semiconductor elements such as SRAM, DRAM, and NAND, or next-generation semiconductor elements such as RRAM, STT MRAM, and PRAM, or a combination thereof.

[0185] When the tissue pathology reading support method 1000 according to the embodiment is implemented using an artificial intelligence semiconductor device, the results (weights) learned by software using a deep learning model can be transferred to synapse mimicking elements arranged in an array structure, or learning can be carried out in the artificial intelligence semiconductor device.

[0186] When implemented by firmware or software, the tissue pathology reading support method 1000 according to this embodiment can be implemented in the form of an apparatus, procedure, function, or the like that performs the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor and can exchange data with the processor by various known means.

[0187] Furthermore, the terms "system," "processor," "controller," "component," "module," "interface," "model," or "unit" may generally refer to a computer-related entity: hardware, a combination of hardware and software, software, or software in execution. For example, such a component may be, but is not limited to, a process run by a processor, a processor, a controller, a control processor, an entity, a thread of execution, a program, and / or a computer. For example, both an application being run by a controller or processor and a controller or processor may be components. One or more components may be within a process and / or thread of execution, and a component may be located on one device (e.g., a system, a computing device, etc.) or distributed across two or more devices.

[0188] Meanwhile, another embodiment provides a computer program stored in a computer recording medium for executing the above-described histopathology reading assistance method 1000. Yet another embodiment provides a computer-readable recording medium having a program for implementing the above-described histopathology reading assistance method 1000 recorded thereon.

[0189] The program recorded on the recording medium can be read, installed, and executed by a computer to perform the steps described above.

[0190] In this way, in order for a computer to read a program recorded on a recording medium and execute the functions realized by the program, the program may include code coded in a computer language such as C, C++, JAVA, machine language, etc. that can be read by the computer's processor (CPU) through a computer device interface (interface).

[0191] Such code may include function code relating to functions that define the aforementioned functions, and may include control code relating to the execution procedures required for the computer processor to execute the functions in a predetermined sequence.

[0192] Furthermore, such code may further include additional information required for the computer's processor to perform the aforementioned functions, or memory reference related code regarding at what location (or address) the media should be referenced in the computer's internal or external memory.

[0193] Furthermore, if the computer processor needs to communicate with other remote computers, servers, etc. to perform the above-mentioned functions, the code may further include communication-related code, such as how the computer processor should communicate with other remote computers, servers, etc. using the computer's communication module, and what information or media should be sent and received during the communication.

[0194] As described above, computer-readable recording media on which the above-mentioned programs are recorded include, by way of example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage device, etc., and may also include those realized in the form of carrier waves (e.g., transmission via the Internet).

[0195] The computer-readable recording medium can also be distributed among computer systems connected via a network, so that the computer-readable code can be stored and executed in a distributed manner.

[0196] Furthermore, functional programs for implementing the present invention and associated codes and code segments can be easily inferred or modified by programmers in the technical field to which the present invention pertains, taking into consideration the system environment of a computer that reads the recording medium and executes the program.

[0197] The tissue pathology reading support method 1000 described with reference to FIG. 12 can be implemented in the form of a recording medium containing computer-executable instructions, such as an application or program module executed by a computer. Computer-readable media may be any available medium accessible by a computer, and can include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media can include all computer storage media. Computer storage media includes all volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable commands, data structures, program modules, or other data.

[0198] The above-described tissue pathology reading support method 1000 may be executed by an application that is pre-installed on the terminal (which may include a program included in a platform or operating system that is pre-installed on the terminal), or by an application (i.e., a program) that a user directly installs on the master terminal via an application providing server such as an application store server, an application, or a web server related to a corresponding service. In this sense, the above-described tissue pathology reading support method 1000 may be implemented in an application (i.e., a program) that is pre-installed on the terminal or that a user directly installs, and may be recorded on a computer-readable recording medium such as a terminal.

[0199] The above description of the present invention is for illustrative purposes only, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not limiting. For example, each component described as a single component can be implemented in a distributed form, and similarly, components described as distributed can be implemented in a combined form.

[0200] The scope of the present invention is indicated by the claims that follow rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

Claims

1. an input unit for inputting slide information and slide tissue images; A preprocessing unit divides the input slide tissue image into multiple patches; a tissue classification unit that generates a patch classification result by inferring a classification of each of a plurality of patches using the trained artificial intelligence model, and generates a slide tissue classification result by inferring a classification of the reconstructed slide tissue image by integrating the plurality of patches and the patch classification results; a storage unit that stores the slide information, the slide tissue image, the plurality of patches, patch information regarding the patches, the patch classification result, and the slide tissue image classification result; and an output unit that outputs patch classification results for each of the plurality of patches and the tissue slide image classification result; The trained artificial intelligence model, when a lesion is present, divides a training slide tissue image displaying the site of the lesion into multiple patches using the pre-processing unit, and proceeds with learning on the patch classification results by inferring the classification of each of the multiple patches, and proceeds with learning on the slide tissue image classification results by integrating the multiple patches and the patch classification results to infer the classification of the reconstructed slide tissue image, a tissue pathology reading support device.

2. 2. The histopathology reading support device according to claim 1, wherein, when the lesion is present in the learning slide tissue image, the site of the lesion is displayed by a single closed curve in a different color for each lesion.

3. 2. The histopathological reading support device according to claim 1, wherein the learning slide tissue images are slide tissue images scanned by a slide scanner without any artificial improvement.

4. The trained artificial intelligence model is two or more, and the slide information includes an organ in the slide tissue image; 2. The tissue pathology reading support device according to claim 1, wherein the tissue classification unit selects one of two or more artificial intelligence models according to the organ in the slide tissue image to classify the tissue image.

5. 2. The tissue pathology reading support device according to claim 1, wherein the output unit displays the patch classification results in a color specific to the location of the lesion in each patch, and displays the slide tissue image classification results in at least one of letters, numbers, and colors.

6. The reconstructed slide tissue image is reconstructed by combining a plurality of patches displayed in a unique color for each lesion position; The histopathology reading support device according to claim 5 , wherein the output unit displays the results of the analysis of the slide tissue image written in letters or numbers together with the reconstructed slide tissue image.

7. The input unit inputs a pathological diagnosis, which is a slide analysis result obtained by a pathologist reading a slide corresponding to the slide tissue image using a microscope or reading the slide tissue image through a screen; 2. The tissue pathology reading support device of claim 1, wherein the display unit further displays, along with the pathology diagnosis resulting from the slide reading by the pathologist, whether or not there is a discrepancy between the slide tissue image classification result by the tissue image classification unit and the classification result by the pathology diagnosis.

8. When the classification result of the slide tissue image and the classification result of the pathological diagnosis are inconsistent, and the classification result of the pathological diagnosis is incorrect, 2. The tissue pathology reading support device according to claim 1, wherein the corrected pathological diagnosis and the created cause of the diagnostic error are input using the input unit.

9. 9. The tissue pathology reading support device according to claim 8, wherein, when the slide tissue image classification result and the classification result based on the pathological diagnosis are inconsistent, if the slide tissue image classification result is incorrect, an annotation is added to the slide tissue image to create a learning slide tissue image, and the artificial intelligence model is additionally trained.

10. 2. The histopathology reading support device of claim 1, wherein when there are multiple slides in one specimen, the output unit provides a related slide list in the upper left corner of the slide tissue image viewer so that related slide tissue images can be viewed together.

11. an input step of inputting slide information and slide tissue images; A pre-processing step of dividing the input slide tissue image into multiple patches; a learning step for learning the tissue slide image classification result by dividing the learning tissue slide image showing the lesion site into a plurality of patches when a lesion is present, and inferring the classification of each of the plurality of patches, and integrating the plurality of patches and the classification results of the patches to infer the classification of the reconstructed tissue slide image, thereby learning the tissue slide image classification result; a tissue classification step of generating a patch classification result by inferring a classification of each of a plurality of patches using the trained artificial intelligence model, and generating a slide tissue image classification result by inferring a classification of a reconstructed slide tissue image by integrating the plurality of patches and the patch classification results; a storage step of storing the slide information, the slide tissue image, the plurality of patches, patch information regarding the patches, the patch classification result, and the slide tissue image classification result; and The tissue pathology reading support method includes an output step of outputting patch classification results for each of the plurality of patches and the slide tissue image classification result.

12. 12. The histopathology reading support method according to claim 11, wherein, when the lesion is present, the learning slide tissue image displays the site of the lesion in a different color for each lesion and as a single closed curve.

13. The histopathology reading support method according to claim 11, wherein the learning slide tissue images are slide tissue images scanned by a slide scanner without any artificial improvement.

14. The trained artificial intelligence model is two or more, and the slide information includes an organ in the slide tissue image; 12. The histopathology reading support method according to claim 11, wherein in the tissue image classification step, one of the two or more artificial intelligence models is selected depending on the organ in the slide tissue image to classify the tissue image.

15. 12. The histopathology reading support method according to claim 11, wherein in the output step, the patch classification results are displayed in a color specific to the location of the lesion in each patch, and the slide tissue image classification results are displayed in one of letters, numbers, or colors.

16. The reconstructed slide tissue image is reconstructed by combining a plurality of patches displayed in a unique color for each lesion position; 16. The histopathology reading support method according to claim 15, wherein the output step displays the slide tissue image classification (or inference) result expressed in letters or numbers together with the reconstructed slide tissue image.

17. In the input step, a pathologist inputs a pathological diagnosis, which is a slide reading result obtained by interpreting the slide tissue image through a microscope or a screen, for the slide corresponding to the slide tissue image; 17. The tissue pathology reading support method according to claim 16, wherein in the display step, together with the pathology diagnosis that is the slide reading result by the pathologist, in the tissue classification step, whether or not there is a discrepancy between the slide tissue image classification result and the classification result by the pathology diagnosis is further displayed.

18. When the classification result of the slide tissue image and the classification result of the pathological diagnosis are inconsistent, and the classification result of the pathological diagnosis is incorrect, The method for assisting in the interpretation of histopathology according to claim 11 , further comprising inputting the corrected pathological diagnosis and the cause of the diagnostic error created in the input step.

19. 20. The method for supporting histopathological reading of tissues according to claim 18, wherein, when the classification result of the tissue slide image and the classification result of the pathological diagnosis are inconsistent, and when the classification result of the tissue slide image is incorrect, annotations are added to the tissue slide image to create a training tissue slide image, and the artificial intelligence model is additionally trained.

20. 12. The method for supporting histopathological reading according to claim 11, wherein, when there are multiple slides in one specimen, the output step provides a list of related slides in the upper left corner of the slide tissue image viewer so that related slide tissue images can be viewed together.

Images