Histopathologic read support apparatus and method thereof

Abstract

The present specification provides a histopathology reading support apparatus and a method thereof, which divide a slide tissue image for learning (indicating a lesion site if a lesion exists) into a plurality of blocks with respect to an artificial intelligence model; inferring a classification of each of the plurality of blocks to learn a block classification result; integrating the plurality of tiles and the tile classification result to infer a classification of the reconstructed slide tissue image, thereby learning a slide tissue image classification result; a trained artificial intelligence model is used to infer a classification of each of the plurality of tiles to generate a tile classification result, and the plurality of tiles and the tile classification result are integrated to infer a classification of the reconstructed slide tissue image to generate a slide tissue image classification result.

Description

Technical Field

[0001] The present invention relates to devices and methods for supporting histopathology readings. Background Art

[0002] Digital pathology is an umbrella term for all technologies that use scanners to convert glass pathology slides into digital slides or slide tissue images (whole slide images (WSIs)), which are then viewed on a monitor via a personal computer or computer network rather than being read using a traditional light microscope.

[0003] The core technology of digital pathology is the rapid conversion of the entire shape of the tissue contained on a glass slide into a high-resolution digital image.

[0004] The ultimate goal of digitizing histopathological images is to achieve “computer-aided diagnosis” by utilizing various automatic image analysis computer algorithms.

[0005] At the same time, the workload of pathologists is steadily increasing, while the number of pathologists is relatively decreasing. The relative shortage of pathologists and the increase in their workload may lead to human errors. Since the pathological diagnosis of histopathological examination actually means "confirmation", if the time for early diagnosis and treatment is missed due to false negatives, there is no chance of identification before the next examination. Unfortunately, false negatives are the most common errors in pathological diagnosis. Therefore, in a repetitive, labor-intensive and habitual reading work environment, tools are needed to reduce false negatives or provide opportunities for early detection and immediate correction of false negatives. Blind review is an important and effective way to strengthen quality control. However, it is actually impossible to carefully check or review all slides read every day.

[0006] A commonly recommended quality control strategy for pathology laboratories is to randomly review a certain percentage of slides each month, but this percentage may vary from institution to institution. Sometimes major inconsistencies occur during this process and are corrected later. However, since the timing is usually 1 to 2 months after the initial diagnostic report date, follow-up measures are often too late to produce clinically meaningful results for patients. If an AI model can help achieve this function, it will be of great help to both patients and pathologists.

[0007] If an AI classification model is used as a screening tool during the pre-reading phase, all target slides should be digitized in advance.

[0008] As a predictable workflow, a glass slide is first created, then digitized and scanned, an AI prediction is made, and positive cases are listed first based on the prediction results. After that, the expert will open the slide tissue image (whole slide image (WSI)) through the viewer, first diagnose through the scanned slide, while referring to the prediction results or heat map shown by the artificial intelligence, and if necessary, directly examine the glass slide through a microscope, otherwise diagnose only through the slide tissue image.

[0009] In recent years, digital pathology has developed rapidly and deeply and has rapidly transitioned to clinical practice, with several advantages over traditional pathology. However, in the field of pathology, it is still difficult to completely replace traditional glass slide-based reading. In particular, there is a concern that microorganisms such as Helicobacter pylori in gastric biopsy tissues are difficult to detect from slide tissue images. In pathology laboratories that regularly report gastritis classification and histological grading through the updated Sydney system, there is a practical burden to directly apply digital pathology to preliminary diagnosis in gastric biopsy reading without using glass slides.

[0010] In particular, in order to apply slide tissue images directly to preliminary diagnosis, high-resolution (x40 or higher) scanning is required to minimize the difference with reading under conventional microscope. In this case, huge costs are incurred in building and operating information computing infrastructure to store and process high-resolution slide tissue images. However, for quality control purposes rather than preliminary diagnosis, slide tissue images scanned at low resolution (e.g., x20) are sufficient.

[0011] In most commercial pathology laboratories, reading begins immediately after slides are made, and the workflow change of postponing the reading step until after the slides are scanned can be a huge burden. Due to these limitations, some researchers have proposed augmented reality microscopes (ARMs) with real-time artificial intelligence integration instead of microscopes based on slide tissue images. However, no matter how high-performance artificial intelligence (ARM) is applied, it cannot detect lesions outside the field of view of the microscope, so it still has limitations in compensating for routine human errors of pathologists. In addition, there are currently no public or commercialized models of ARM solutions for endoscopic biopsy reading.

[0012] Finally, pathologists may be subject to bias from AI reliance when AI models prioritize identifying, visualizing, and providing expected classification results before pathologists read them. Although several AI models have been introduced that demonstrate performance comparable to pathologists’ diagnostic abilities, the legal responsibility and authority for the diagnostic outcome of each case ultimately lies with the pathologist. Bias is a problem in itself, and as AI model performance improves, it may even lead to dependency. This may also be a concern for patients. When pathologists who rely on high-performance AI models make biased diagnoses, which can lead to misdiagnoses (e.g., when both the AI ​​model and the human pathologist miss a lesion, or when the human pathologist ignores the slide tissue image by trusting the AI ​​model’s prediction of a negative result), there is the potential for legal disputes over whether liability exists.

[0013] In recent years, many researchers have developed and proposed artificial intelligence models based on histopathology slide images, and in fact, some studies claim that they show performance similar to or better than that of pathologists. However, ultimately, the full responsibility and authority for each diagnosis lies solely with the pathologist involved, which cannot be fully replaced by artificial intelligence.

[0014] In particular, in gastrointestinal pathology, carcinoma (CA) / high-grade dysplasia (HGD) screening has interobserver variability or inconsistency depending on the group. In gastrointestinal pathology, it is easy to predict that the performance of each developed AI classifier will inevitably vary depending on how the classification is defined. In fact, other similar studies have shown that there are differences in the definition of the classification according to the group, and some studies have excluded diagnoses that fall into the gray area.

[0015] Furthermore, in many cases, pathologists in their daily practice use ambiguous language when reporting endoscopic biopsy (or other small biopsy) specimens, which is different from the excision specimen reading. Some examples are atypical glandular proliferation of undetermined significance, suspected dysplasia, cannot rule out malignancy, favor neoplasm…. In this reality, it is almost fantasy to expect that AI models will address these concerns of pathologists. This is because of the limitations of small biopsy tissue reading itself.

[0016] However, many studies focus on improving the performance of AI models in accurately diagnosing lesions of interest (region of interest (ROI)), which often leads to low reproducibility in clinical practice. This is because the conditions for high accuracy of AI models are very different from the conditions in the reality of daily practice. In order to ensure the high accuracy of AI models, it is very important to prepare for very fine data. In fact, poor scan quality (out of focus, missing tissue, bubbles, etc.) and poor slide quality (poor staining, poor sectioning, tissue artifacts, bubbles, tissue folding, poor dehydration, etc.) are factors that reduce model performance. The performance of the model will be maximized when the model is trained only with very fine data, excluding all artifacts, and scanned clearly without blur from truly well-made slides. However, considering the transition to a "fully digital pathology laboratory", this may be quite contradictory and unrealistic.

[0017] As mentioned earlier, the number of histopathology examinations is gradually increasing, which not only increases the workload of pathologists, but also puts a burden on pathology laboratory technicians who prepare slides. The increase in workload has also affected the work ability of histopathology technicians, which has led to a decline in slide quality. In other words, pathologists cannot control every slide to achieve perfect quality every day, and in fact, they encounter a higher proportion of slides with substandard quality than expected. The same is true for scanning quality. Commercial reports have a scanning error rate of about 1-3%. These slides may be of poor quality and cannot be read completely, or it is common to see very poor quality slide tissue images that can be read to a certain extent but show the various artifacts mentioned above. Although readable, it is unrealistic and inefficient to rescan all low-quality slide tissue images. In terms of developing usable AI models and applying them to practice, it is more appropriate to develop and apply reliable AI models that reflect reality according to routine practice, rather than taking on such inefficient workloads just to maximize the performance of AI models. Summary of the invention

[0018]

Technical issues

[0019] Embodiments provide apparatus and methods for supporting histopathology readings that reduce potential human errors, particularly false negatives, often encountered by pathologists.

[0020]

Technical solution

[0021] The present disclosure provides a tissue pathology reading support device and method, which, when a lesion is present, trains an artificial intelligence model using a patch classification result by dividing a training slide tissue image marked with a lesion site into a plurality of patches and inferring a classification of each of the plurality of patches, and trains the artificial intelligence model using a slide tissue image classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of patches and the classification results of the patches; and a tissue classification step, generates a block classification result by inferring a classification of each of the plurality of patches using the trained artificial intelligence model, and generates a slide tissue image classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of patches and the classification results of the patches.

[0022] One embodiment provides a tissue pathology reading support device, including: an input unit that inputs slide information and a slide tissue image; a preprocessing unit that divides the input slide tissue image into multiple blocks; a tissue classification unit that generates a block classification result by inferring the classification of each of the multiple blocks using a trained artificial intelligence model, and generates a slide tissue image classification result by inferring the classification of a slide tissue image reconstructed by integrating the multiple blocks and the classification results of the blocks; a storage unit that stores the slide information and the slide tissue image, multiple blocks, block information about the blocks, the block classification result, and the slide tissue image classification result; and an output unit that outputs the block classification result and the slide tissue image classification result of each of the multiple blocks.

[0023] When a lesion is present, the trained artificial intelligence model can be trained using the block classification results by using a preprocessing unit to divide the training slide tissue image marked with the lesion site into multiple blocks and by inferring the classification of each of the multiple blocks, and can be trained using the slide tissue image classification results by inferring the classification of the slide tissue image reconstructed by integrating the multiple blocks and the classification results of the blocks.

[0024] Another embodiment provides a tissue pathology reading support method, including: an input step of inputting slide information and a slide tissue image; a preprocessing step of dividing the input slide tissue image into a plurality of blocks; a training step of, when a lesion is present, dividing a training slide tissue image marked with a lesion site into a plurality of blocks and training an artificial intelligence model using a block classification result by inferring a classification of each of the plurality of blocks, and training an artificial intelligence model using a slide tissue image classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks. training the artificial intelligence model; a tissue classification step of generating a block classification result by inferring the classification of each of the plurality of blocks using the trained artificial intelligence model, and generating a slide tissue image classification result by inferring the classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks; a storage step of storing slide information and a slide tissue image, a plurality of blocks, block information about the blocks, block classification results, and slide tissue image classification results; and an output step of outputting the block classification result of each of the plurality of blocks and the slide tissue image classification result.

[0025]

Beneficial Effects

[0026] By means of the apparatus and method for supporting histopathology reading according to the embodiments, potential human errors, especially false negatives, often encountered by pathologists may be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a conceptual diagram showing a histopathology reading support system according to one embodiment;

[0028] Figure 2 is a block diagram showing a histopathology reading support device according to one embodiment;

[0029] Figure 3 Shows the training Figure 2 The process of developing artificial intelligence models;

[0030] Figure 4 It is shown in Figure 3 A flowchart of a data preprocessing process for generating a block image during the data preprocessing process;

[0031] Figure 5 Shows Figure 3 The operation process and results of the block classification unit;

[0032] Figure 6 Shows Figure 3 The operation process and results of the slide tissue image classification unit;

[0033] Figure 7 Shows Figure 2 The operation process and results of the organizational classification unit;

[0034] Figure 8 It is shown by Figure 2 A flowchart of slide analysis results and slide tissue image classification results of a histopathology reading support device;

[0035] Fig. 9 Shown in Figure 2 A slide tissue image displayed on a display unit;

[0036] Fig.10 Shown in Figure 2 The entire reading result, slide preview image, AI reading result and previous result information are displayed on the display unit;

[0037] Fig.11 Shown in Figure 2 whether there is any inconsistency between the slide tissue image classification result displayed on the display unit and the statistical information and slide analysis result;

[0038] Fig.12 is a flowchart illustrating a histopathology reading support method according to another embodiment; and

[0039] Fig.13 is a block diagram illustrating a computing system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] Hereinafter, exemplary embodiments of the inventive concept will be described in detail with reference to the accompanying drawings. However, the inventive concept may be modified in various ways and should not be construed as being limited to the embodiments described herein. Throughout the specification and the accompanying drawings, the same reference numerals may be used to refer to the same or similar elements. However, the present invention may be implemented in various other forms and is not limited to the embodiments described herein. For clarity of the present disclosure, irrelevant parts are removed from the accompanying drawings, and similar reference numerals are used throughout the specification to refer to similar elements.

[0041] In the embodiments of the present invention, when one element is "connected" to another element, the element may be "directly connected" to the other element, or the element may be "electrically connected" to the other element via an intermediate element. When an element "includes" or "contains" another element, the element may also include but not exclude the other element, and the terms "include" and "comprising" should be understood as not excluding the possibility of the existence or addition of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0042] When the measured value of an element is modified by the term "approximately" or "substantially", if a production or material tolerance is provided for the element, the term "approximately" or "substantially" is used to indicate that the element has the same or close value as the measured value, and is used to better understand the present invention, or to prevent any unscrupulous infringement of the present disclosure in the case of referring to exact or absolute numbers. As used herein, "the step of A" or "step A (when performed)" does not necessarily mean that the step is for A.

[0043] As used herein, the term "component" may refer to a unit or device implemented in hardware, software, or a combination thereof. One unit may be implemented by two or more hardware devices or components, or two or more units may be implemented in a single hardware device or component.

[0044] As used herein, some operations or functions described as being performed by a terminal or device may be performed by a server connected to the terminal or device, rather than by the terminal or device. Similarly, some operations or functions described as being performed by a server may be performed by a terminal or device connected to the server, rather than by the server.

[0045] In the present disclosure, the training process may be expressed as training, and the result may be expressed as learning, but the training process or the result may be referred to as one of training or learning.

[0046] Figure 1 is a conceptual diagram illustrating an automatic verification system for diagnosis in routine practice of histopathology reading according to one embodiment.

[0047] Reference Figure 1 , an automatic verification system 100 for histopathology reading diagnoses in routine practice according to one embodiment includes a slide scanner 110 , a server computer 120 , and a display device 130 .

[0048] If the operator inputs a tissue slide 112 , the slide scanner 110 generates a digitized slide tissue image 114 , and if the digitized slide tissue image 114 is uploaded to a server computer 120 , the developed artificial intelligence model 122 generates an output on a display device 130 .

[0049] The output may be, for example, lesion location information 132 including tissue lesion locations and patient reading information 134 including classification results in patient units. In other words, the display device 130 displays the lesion location information 132 including tissue lesion locations and patient reading information 134 including classification results in patient units.

[0050] In addition, the output also shows whether there is a discrepancy between the slide tissue classification result diagnosed by the histopathology support device using the artificial intelligence model and the pathology diagnosis (which is the slide reading result of the pathologist), thereby constructing an automatic verification system 100 for diagnosis in routine practice for histopathology reading by the artificial intelligence model. Therefore, the quality of diagnosis in routine practice can be managed during pathology diagnosis.

[0051] The server computer 120 may include a reference Figures 2 to 10 and Fig.11 The histopathology reading support apparatus 200 is described below. Hereinafter, the histopathology reading support apparatus 200 is described in detail.

[0052] Figure 2 is a block diagram illustrating a histopathology reading support device according to one embodiment.

[0053] Reference Figure 2 , a histopathology reading support device 200 according to one embodiment may include an input unit 210 , a preprocessing unit 220 , a tissue classification unit 230 , a storage unit 240 , and an output unit 250 .

[0054] The input unit 210 inputs slide information, which is information about a tissue slide, and a slide tissue image obtained by digitizing the tissue slide.

[0055] As reference Figure 1 As described above, if the operator inputs a tissue slide 112, the slide scanner 110 generates a digitized slide tissue image 114. The slide tissue image 114 may be a training slide tissue image and a reading slide tissue image. The training slide tissue image is a digitized slide tissue image of the learning slide 112a, and the reading slide tissue image is a digitized slide of the reading slide 112b.

[0056] The input unit 210 may receive a slide tissue image obtained by capturing tissue from the outside of the histopathology reading support apparatus 200. The slide tissue image is an image of the captured tissue.

[0057] The tissue may contain human or animal cells. For example, the tissue may be a biopsy tissue extracted from an organ of the human body (e.g., stomach, colon, small intestine, and liver) to identify the presence of cancer (e.g., stomach or colon cancer) cells.

[0058] The tissue can be captured by various types of capture devices (such as cameras, video cameras or scanners). In addition, the format of the image of the captured tissue can be determined differently as JPG, GIF, PNG, BMP, etc.

[0059] In this case, the tissue may be tissue obtained by hematoxylin and eosin staining. By hematoxylin and eosin staining, the cell nucleus may be stained blue, while the extracellular matrix and cytoplasm may be stained pink.

[0060] Meanwhile, the training slide tissue image may further include annotation information. The annotation information is information indicating which region of the tissue image corresponds to which group. For example, the annotation information may indicate that a specific region of the tissue image is an abnormal region in which a lesion (e.g., cancer) exists, and the remaining region is a normal region in which no lesion exists. For example, as shown below with reference to Figure 4 As described above, if there is a lesion, the training slide tissue image can indicate the lesion site through a single closed curve of a different color for each lesion as annotation information.

[0061] When training the artificial intelligence model 231 for classifying the slide tissue image, the annotation information can be used as information indicating the correct answer.

[0062] The pre-processing unit 220 may divide the slide tissue image 114 input from the input unit 210 into a plurality of blocks 116 .

[0063] The size of each of the plurality of blocks 116 may be smaller than the tissue image. For example, the respective sizes of the plurality of blocks 116 may be the same as n*n pixels (eg, 256*256 or 128*128).

[0064] As another example, the sizes of the plurality of blocks 116 may be different from each other. Some of the plurality of blocks 116 may be blocks of size n*n pixels (e.g., 256×256), while others may be blocks of size m*m pixels (e.g., 128×128). In this case, m is less than n.

[0065] The tissue classification unit 230 generates a block classification result by inferring the classification of each block 116 among the multiple blocks using the trained artificial intelligence model 231, and generates a slide tissue classification result by inferring the classification of the slide tissue image reconstructed by integrating the multiple blocks 116 and the classification results of the blocks.

[0066] When a lesion is present, the trained artificial intelligence model 231 can be trained using the block classification results by using the preprocessing unit 220 to divide the training slide tissue image 114 marked with the lesion site into multiple blocks 116 and inferring the classification of each block 116 in the multiple blocks, and can be trained using the slide tissue image classification results by inferring the classification of the slide tissue image 114 reconstructed by integrating the multiple blocks 116 and the classification results of the blocks.

[0067] The artificial intelligence model 231 may be a deep learning model. In the present disclosure, the deep learning model may be a model in which an artificial neural network is stacked in multiple layers. The deep learning model may be implemented as a model that trains a network in the following manner: the features of each image are automatically obtained by learning a large amount of data in a deep neural network composed of multiple layers of networks, and thereby minimizing the error in the objective function (i.e., prediction accuracy).

[0068] For example, a deep learning model can be a convolutional neural network (CNN). CNN can be implemented in various modes, such as visual geometry group (VGG) network, GoogleNet, ResNet, and DenseNet.

[0069] However, the deep learning model described in the embodiments of the present invention is not limited to CNN, but other types of deep learning models that may be used currently or in the future (for example, a deep hierarchical network (DHN), a convolutional deep belief network (CDBN), a deconvolutional deep network (DDN), or a recurrent neural network (RNN)) may be used.

[0070] The deep learning model can be implemented by a deep learning framework. The role of the deep learning framework is to provide a function library commonly used when developing deep learning models and support the correct use of system software or hardware platforms. In this embodiment, any deep learning framework that is currently available or will be available in the future can be used to implement the deep learning model.

[0071] Training a deep learning model means adjusting the parameters of the deep learning model so that the predicted value for the input of the deep learning model is as close as possible to the actual value (ground truth value). The tissue classification unit 230 can improve the prediction accuracy of the deep learning model by repeatedly inputting the block 116 and the reconstructed slide tissue image (reconstructed WSI) to the deep learning model 231, and adjusting the parameters of the deep learning model 230 by comparing the actual value and the predicted value of the deep learning model.

[0072] In this case, the tissue classification unit 230 may adjust the parameters of the deep learning model 231 in a manner that minimizes the loss function value of the input block 116 and the reconstructed slide tissue image.

[0073] The loss function is a function that calculates how similar the predicted value of the type of the target block is to the actual type of the target block. The loss function can be, for example, mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), binary cross entropy, categorical cross entropy, etc.

[0074] There are two or more trained artificial intelligence models 231, and the slide information may include organs of the slide tissue image. In this case, the tissue classification unit 230 can classify the tissue image by selecting one of the two or more artificial intelligence models according to the organ of the slide tissue image. As an example of a trained artificial intelligence model 231, an artificial intelligence model trained with gastric tract and colon organs is described below. However, in addition to the stomach and colon models shown, models of other organs (breast, prostate, skin, etc.) can also be further developed in a similar manner and applied in the same manner as a conventional quality control system.

[0075] In the embodiments of the present invention, the block classification results and the slide tissue classification results may be classified in various ways.

[0076] For example, the block classification results and the slide tissue classification results may be classified into two groups or categories, such as malignant and non-malignant, or tumor and non-tumor.

[0077] As another example, the block classification results and the slide tissue classification results may be classified into three groups: malignant, dysplastic, and non-tumor.

[0078] As another example, the block classification results and the slide tissue classification results may be classified into four groups: malignant, dysplastic, unclassified, and non-tumor.

[0079] For example, the malignancy group can be defined as a group indicative of malignancy, including adenocarcinoma, suspected adenocarcinoma, suggestive adenocarcinoma, high-grade lymphoma, and other carcinomas.

[0080] The dysplasia group can be defined as the group indicating dysplasia, including tubular adenomas with all grades of dysplasia.

[0081] The non-tumor group can be defined as a group indicating non-tumor benign lesions (such as gastritis, polyps, etc.).

[0082] The unclassified group can be defined to indicate the types of remaining lesions that do not correspond to the above three groups, such as atypical glandular proliferation, neuroendocrine tumors, submucosal tumors, low-grade lymphomas, and stromal tumors.

[0083] In this case, some of the block classification results and the slide tissue classification results may be defined as a normal group, and the remaining results except the normal group may be defined as an abnormal group. For example, if the block classification results and the slide tissue classification results are classified into four groups: malignancy, dysplasia, unclassified, and non-tumor, then malignancy, dysplasia, and unclassified may be defined as an abnormal group, and non-tumor may be defined as a normal group.

[0084] The storage unit 240 can store slide information and slide tissue images, multiple blocks, block information about blocks, block classification results, and slide tissue classification results. The storage unit 240 can temporarily store some of the above information and then automatically delete them, or continuously store them until a deletion request is made.

[0085] The output unit 250 may output the block classification result and the slide tissue classification result of each of the plurality of blocks 116. The output unit 250 may output not only the block classification result and the slide tissue classification result of each of the plurality of blocks 116, but also all information input through the input unit 210 and all information stored in the storage unit 240.

[0086] According to the development intention of the tissue classification model using artificial intelligence model 231, the purpose is to use it for quality control of routine practice in real life. The principle of data preparation is to use the data as it is, retaining the variability between observers and routine artifacts in real life to thoroughly reflect the situation in practice, rather than artificially preparing a very detailed slide tissue image dataset to ensure the high accuracy of model 231.

[0087] Slide quality must not be manipulated to ensure optimal model accuracy. Instead, common artifacts - slide cuts, poor staining, poor sectioning, bubbles, etc. can be used for training as they are not artificially enhanced (e.g., re-preparing high-quality slides specifically for training). However, poor-quality slides that are difficult for pathologists to read can be excluded.

[0088] In other words, the training slide tissue images can be slide tissue images scanned by a slide scanner as is, without artificial enhancement.

[0089] Scan quality cannot be adjusted manually either. Depending on the scanner manufacturer, the defect rate that requires a rescan is an average of 1-3%, and most of the others are insignificant issues that do not affect the read. Examples are defocus in a narrow range, overlap of some tiles, or edge cuts in some scanned tissues. To reflect these common minor issues as is, no artificial enhancement is performed, such as rescanning to obtain high-quality slide tissue images specifically for training. However, as Figure 2As shown, scan failures due to noisy images that are difficult for pathologists to read can be excluded.

[0090] The block classification results can be defined as four groups, as shown below, as a four-classification model that classifies all possible diagnoses (Table 1). However, by definition, “Group U”, which has the largest intra-group disease heterogeneity, may be a factor that causes a significant decrease in model performance, so a three-classification model that does not include “Group U” can be adopted. Cases corresponding to Group U can eventually be excluded from the training data to develop a three-classification model. However, as a next step, it can be redefined to include only NETs (level 1 or 2) of Group U in Table 1 and upgraded to a four-classification model for practical application.

[0091] [Table 1]

[0092]

[0093] The slide tissue image corresponding to each classification can be annotated independently by each pathologist. Group M is drawn as a single red closed curve, group D is drawn as a single blue closed curve, and when the boundary between the lesion part and the normal part is unclear, the outline can be drawn to include only the obvious lesion (including epithelium and stroma). Only the block within the single closed curve is used as the data for each classification, and all other parts can be deleted.

[0094] For group N, a pathologist can review and confirm the slide tissue images that meet the above definition, and the blocks are immediately generated without additional annotation and used as data. In this case, the parts other than the lesions of group M and group D slides may not be used as group N data. It is possible to avoid including non-neoplastic glands or crypts within a closed curve as much as possible, and to avoid including non-tissue components such as necrosis, ulcer debris, extracellular mucin pools, and blood as much as possible. If necessary, the annotations can be reviewed by other pathologists and, in some cases, can be modified or excluded.

[0095] The slide tissue image 114 is input to the artificial intelligence model 231, and the artificial intelligence model 231 is required to group the slide tissue image and predict the result. The slide tissue image 114 is converted and processed into a form recognizable by the artificial intelligence model 231, and the artificial intelligence model is constructed to be processed in 1) the training step and 2) the prediction step without information loss.

[0096] Since the slide tissue image 114 includes information at the gigapixel level, it is not suitable for immediate use in the deep learning model. Therefore, the slide tissue image 114 including information corresponding to the gigapixel level is converted to be processed by the artificial intelligence model 231.

[0097] The model is constructed based on a method of converting the slide tissue image 114 into a block form and processing it. As mentioned in many studies on slide tissue image processing based on convolutional neural networks (CNN), processing a complete slide tissue image at one time may result in loss of information contained in the slide tissue image and loss of resources. Therefore, according to Figure 3 The model is trained in the order of operations shown.

[0098] The output unit 250 may indicate the block classification result with a unique color for each lesion position of each block, and indicate the slide tissue classification result with at least one of letters, numbers, or colors.

[0099] The reconstructed slide tissue image is reconstructed by combining a plurality of blocks displayed in a unique color for each lesion position, and the output unit 250 may display the slide tissue analysis result indicated by letters or numbers together with the reconstructed slide tissue image.

[0100] The input unit 210 can input a pathological diagnosis, which is a slide reading result of a pathologist examining a slide corresponding to a slide tissue image through a microscope or examining a slide tissue image through a screen, and the display unit 250 can additionally display whether the slide tissue image classification result of the tissue classification unit and the pathological diagnosis (which is the slide reading result of the pathologist) are inconsistent and the pathological diagnosis (which is the slide reading result of the pathologist). In this way, the quality of diagnosis in routine practice can be managed by constructing an automatic verification system for diagnosis in routine practice of histopathology reading using the artificial intelligence model 231.

[0101] Specifically, the content input to the input unit 210 can be the content read by the pathologist, that is, the "pathological diagnosis" information (text information). According to the classification rules, the pathological diagnosis information can be "classified" into one of the three categories of M, D and N. The artificial intelligence model can independently predict one of the three categories M, D and N. The display unit 250 displays whether the corresponding "classification result of pathological diagnosis information (text information)" and "classification result of artificial intelligence model (image information)" are consistent or inconsistent. When the display unit 250 additionally displays whether the slide tissue image classification results and pathological diagnosis (which is the pathologist's slide reading result) of the above two or more situations are inconsistent, if one of the slide tissue image classification results and pathological diagnosis (which is the pathologist's slide analysis result) is positive, the display unit 250 can display it preferentially.

[0102] Figure 3 Shows the training Figure 2 The process of developing an artificial intelligence model. Figure 4 It is shown in Figure 3 Flow chart of the data preprocessing process for generating block images during the data preprocessing process.

[0103] Reference Figure 3 and Figure 4 In order to train the artificial intelligence model used by the tissue classification unit 230, it includes a process S310 of preprocessing data, a process S320 of classifying blocks, and a process S330 of classifying slide tissue images.

[0104] In the process of preprocessing data (S310), the data is organized by the preprocessing unit 220, and the artificial intelligence model is converted into several blocks in a trainable form.

[0105] The data preprocessing process S310 trains the model 231 based on the label information shown in Table 1 and the actual value information generated by the above method. The training slide tissue images (WSI) of group N can set data as they are, and for groups M and D, data can be set including annotation information indicating the lesion site by color, for example.

[0106] Two deep neural network (DNN) models are mainly required to learn the glass slide tissue image (WSI) for training. One model can be used for the block classification unit 232, and the other model can be used for the glass slide tissue image classification unit 234. Therefore, for the training of each model, two types of data are prepared: "block image data" and "glass slide tissue image data".

[0107] First, for the slide tissue image data, the scan configuration values ​​of the collected slide tissue images (WSI) are identified, and the slide tissue images with different values ​​are excluded from the data set. This is to prevent the slide tissue images (WSI) taken under different conditions from generating errors in the model. The final selected slide tissue image data set is shown in Table 1.

[0108] For group N, square patches of size 256*256 pixels were generated from the training slide tissue images (WSI). The patch size can be defined by interviews to investigate the trade-off between user convenience and performance. For example, it can be determined that patches with large pixel sizes may have good performance but are not suitable for providing a description of lesion location information provided from the user interface.

[0109] Next, for group M and group D, only the blocks related to the annotation information present in the annotation may be selected and stored, and blocks other than these blocks may not be included in the data set.

[0110] In addition, in order to perform accurate model performance evaluation and feedback, the training, evaluation, and inspection sets are configured based on the slide tissue image (WSI) to which the block data belongs. If the correlation between the slide tissue image (WSI) and the block is not considered when constructing the training, evaluation, and inspection sets, then in the configuration of the training set, the block images generated from the same slide tissue image (WSI) are distributed in the training data set and the evaluation and inspection data set at the same time, so that the model may operate deceptively in the inspection step, making it difficult to perform an accurate evaluation. Finally, in order to minimize the bias generation of the tissue classification unit 230, the block data can be randomly sampled from the generated training pool.

[0111] Figure 5 Shows Figure 3 The operation process and results of the block classification unit.

[0112] Reference Figure 3 and Figure 5 The block classification process S320 trains the label information of each block generated from a training slide tissue image (WSI) by the block classification unit 232.

[0113] The block classification process S320 uses a CNN-based DNN architecture to train the block classification unit 232. For example, the model can be trained by using DenseNet201, which shows high performance in the DNN architecture for block image classification. The set training data can be put into the trained DenseNet201 model. The block classification model can train each block image to infer three correct answer labels, such as group M, group D, and group N.

[0114] The block classification result generated by the block classification unit 232 has two purposes: 1) to be used as an element for the slide tissue image classification unit 234 to make inferences; and 2) to provide a description in the user interface. In particular, in order to be used as an element of the slide tissue image classification unit 234, the slide tissue image (WSI) can be learned based on the model trained in this step. The block classification unit 232 can generate the block classification result in the form of distribution of each group, which is used as key information.

[0115] Figure 6 Shows Figure 3 The operation process and results of the slide tissue image classification unit.

[0116] Reference Figure 3 and Figure 6The slide tissue image classification process S330 trains an artificial intelligence model that integrates information about the slide tissue image (WSI) to generate block classification information for the slide tissue image (WSI) by training the slide tissue image classification unit 234. The training method of the artificial intelligence model is also applied to each artificial intelligence model of two or more organs to construct a model.

[0117] The slide tissue image classification unit 234 performs training to infer training data as a correct answer label of a slide tissue image (WSI), wherein the training data uses a block classification result (which is fragmentation information of a slice) as an element.

[0118] The training step of the slide tissue image classification unit 234 is based on 1) the block maker model used in the data preprocessing step and 2) the block classification unit 232 trained in the model training, and the slide tissue image classification unit 234 is trained through a series of processes. This is to effectively utilize the resources required to classify slide tissue images containing gigapixel information, and each model is designed to operate organically like one model.

[0119] In the training process of the slide tissue image classification unit 234 , the slide tissue image (WSI) is converted into a patch image by the patch maker, and the converted patch image is reconstructed based on the position information and learned by the slide tissue image classification unit 234 .

[0120] The training process of the slide tissue image classification unit 234 mainly includes three steps: a block making step, a block classification step and a slide tissue image learning step.

[0121] In the block making step, the slide tissue image to be learned is converted into a plurality of block images. In order to retain the position information about each block image during the corresponding conversion process, the index and position information about the converted block and the generated slide tissue image are recorded in the storage unit 240.

[0122] Next, in the block classification step, the trained block classification unit 232 is used to infer what classification information each block constituting one slide tissue image has.

[0123] According to the index of the block, the inferred information about each block image is stored in the storage unit 240. Therefore, for one slide tissue image, the index, position information, and classification information about each block image are comprehensively recorded in the storage unit 240. Finally, a reconstructed slide tissue image (WSI) is generated by integrating the storage unit 240 of the block image to learn the slide tissue image. Since the reconstructed slide tissue image includes feature information about the slide tissue image in a summarized or compressed form, the corresponding information can be processed as an image in the CNN. Therefore, a CNN group classification network with a reconstructed slide tissue image in input form is constructed, and the model is trained to appropriately infer three correct answer labels, such as group M, group D, and group N.

[0124] Reference Figure 6 , shows a process where for one slide tissue image, the block maker converts the block into multiple images, and the block classification unit 232 generates distribution information, and this is converted into a reconstructed slide tissue image.

[0125] It can be determined that the reconstructed slide tissue image is configured in the shape of a general square image. A CNN model trained to classify the reconstructed slide tissue image is used as the slide tissue image classification unit 234. The above series of processes are performed on gastric biopsy and colon biopsy, respectively, to generate two models.

[0126] Figure 7 Shows Figure 2 The tissue classification unit uses artificial intelligence models to classify tissues using operational processes and results.

[0127] In this step, the entire prediction framework is operated based on the model trained in the previous step to generate results and additional information. The prediction method can be configured similar to the training sequence described above. For the developed model, a model for each of the gastric biopsy and colon biopsy is constructed and trained separately. Each model can be operated in the same order.

[0128] The operation process of the model of the trained slide tissue image classification unit consists of tasks similar to the sequence of training tasks.

[0129] If a new tissue slide image (WSI) is input to the tissue classification unit 230 , the patch maker generates a plurality of patch images and patch information, such as index and position information, from the inputted slide image and records them in the storage unit 240 .

[0130] Each block image is input to the block classification unit 232, and through the model inference group, a block classification result is generated and stored in the storage unit 240. Finally, the slide tissue image classification unit 234 generates a reconstructed slide tissue image by integrating the block image information and the block classification information of the storage unit 240, and classifies the slide tissue image.

[0131] The process includes converting into a reconstructed slide tissue image by integrating each block classification information and position information, inputting it to the slide tissue image classification unit 234 to generate a slide tissue image classification result, and storing it in the storage unit 240.

[0132] Therefore, three types of information are generated and stored in the storage unit 240, which may be block image information, block classification information, and slide tissue image classification information. The block image information includes block index, block image, block position information, etc., and the block classification information refers to a group inference result generated according to a block classification model, and the slide tissue image classification information refers to group information inferred by the slide tissue image classification model for one slide tissue image.

[0133] Image classification information for quality management purposes in routine practice is configured to comprehensively provide the above three types of information to the user. The block image information provides visualization for the original image and provides location information about each block in the user interface, providing visual information about the slide image in the user interface.

[0134] The block classification information is used to provide classification information (heat map) related to each block of the slide image appearing in the user interface, which helps to provide a basis for the classification result and the lesion location information for quality control. The display unit 250 intuitively provides the location of the lesion based on the slide tissue image classification information and the block image prediction information on the storage unit 240.

[0135] Figure 8 The pathologist's slide reading results and Figure 2 Flowchart of the slide tissue image classification results of the histopathology read-in support device artificial intelligence model.

[0136] Figure 2 The histopathology reading support device 200 supports a series of processes in which daily reading slides are scanned and converted into slide tissue images, while artificial intelligence reading and visualization and information processing of the results are performed.

[0137] The histopathology reading support device 200 can perform daily scanning and artificial intelligence prediction on all slides read after microscope reading.

[0138] Reference Figure 2 and Figure 8 , the pathologist reads under a microscope. The slides are not prepared individually according to specimens such as gastric biopsy and colon biopsy, but are simply mixed in the order of receiving numbers without distinguishing organs and provided to the pathologist. Therefore, the pathologist also reads in the order of receiving numbers without distinguishing organs. The input unit 210 inputs the slide diagnosis result obtained by the pathologist reading the slide corresponding to the slide tissue image under a microscope or through a screen (S810).

[0139] The input unit 210 is an input device such as a keyboard or a mouse, for example, and can input analysis slide reading results typed by a pathologist or clicked by a mouse.

[0140] The scanner 110 scans the read slide. The slide tissue image generated by the scanner 110 is a file with a specific extension, such as a file with an mrxs extension, and is stored in a path specified by the scanner program. If a new slide tissue image file is generated in the path, it is copied to the storage unit 250.

[0141] The histopathology reading support device 200 searches the scanned slide name (pathology number) in the storage unit 240, and calls and drives one of two or more artificial intelligence models corresponding to each organ according to the specified keyword (see Table 2). The artificial intelligence model reads the slide tissue image file, predicts the block classification result and the slide tissue image classification result, and stores each prediction result in the storage unit 240 (S820).

[0142] [Table 2]

[0143]

[0144] The display unit 250 may additionally display whether the slide tissue image classification result of the tissue classification unit 230 and the slide classification result of the pathological diagnosis are inconsistent and the slide pathological diagnosis result of the pathologist (S830).

[0145] Specifically, the scanned slides are divided into one group according to the pathological diagnosis and one group according to the artificial intelligence prediction, and the pathologist can access the histopathology reading support device 200 the next day (or a few hours later) to determine whether the groups are consistent and prioritize the review of inconsistent cases.

[0146] This allows the pathologist to "review" discrepant cases and, if necessary, "correct" the initial results (pathology diagnostic errors caused by human error).

[0147] The pathologist may determine whether there is a pathological diagnosis error in the inconsistent case (S840), and if the pathological diagnosis in the inconsistent case is incorrect, the pathologist may input a corrected pathological diagnosis and the reason for the diagnosis error using the input unit 210 (S850).

[0148] In contrast, when the slide tissue image classification result of the tissue classification unit 230 is incorrect in the inconsistent case, as described above, a training slide tissue image having annotations included in the corresponding slide tissue image is created, and the artificial intelligence model may be additionally trained (S860).

[0149] Fig. 9 Shown in Figure 2 The slide tissue image is displayed on the display unit. Fig.10 Shown in Figure 2 The entire reading result, slide preview image, AI reading result and previous result information are displayed on the display unit. Fig.11 Shown in Figure 2 The statistical information displayed on the display unit and the slide reading results and the slide tissue image classification results are inconsistent.

[0150] like Fig. 9 As shown, the heat map color of each block displayed on the slide tissue image represents the result of the group inferred from the corresponding block.

[0151] The histopathology reading support apparatus 200 can search for scanned slide data for each reception date, examination date, and scan date through the examination result / statistics page.

[0152] First, the examination results page provides a list of slide information for each slide including the receipt date, examination data, pathology number (slide name, patient name, pathology diagnosis classification, AI predicted classification, consistency, anatomical AI model or pathologist reader).

[0153] From this list, the corresponding slide tissue images and heat maps can be identified. In addition, text information such as a single magnification (0.5x) AI model heat map thumbnail and pathology diagnosis, annotations, and previous pathology diagnosis are also provided. It is difficult for pathologists to review 100% of all read slide tissue images for quality control purposes only when performing critical tasks in their daily lives. Therefore, a filtering function can be performed to search only for cases where the classification results of the pathology diagnosis and the classification results predicted by the artificial intelligence are inconsistent.

[0154] Furthermore, if a pathologist examines the slide tissue image he or she reads, the corresponding row is highlighted so that it is intuitively known whether it has been examined.

[0155] Furthermore, on the statistics page, the predictive performance and distribution of each AI model (stomach / colorectal) can be identified, and since the corresponding slide tissue image can be reviewed by clicking on each cell in the table, pathologists can even selectively review cases that meet desired conditions through the statistics page as well as through the examination results page.

[0156] If you search for scanned slides and double-click a specific slide row or column, the slide tissue image viewer for the corresponding slide will be displayed. The image position can be moved by dragging the screen, the mouse wheel can be operated to support zooming in / out of the slide tissue image, and the position currently displayed on the screen can be identified by the mini map at the upper right end.

[0157] In addition, a list of related slides is provided at the upper left end of the slide tissue image viewer so that in the case where one specimen includes multiple slides (for example, when there are recut, continuous or deeper sections, or when there are two or more blocks), related slide tissue images can be bundled and viewed. In addition, various functions related to slide tissue images can be used through the function buttons at the lower and left ends.

[0158] Since the AI ​​model is based on black box technology, it is difficult to determine which features of the data the model has learned and for what reasons the prediction results are obtained. Therefore, it is difficult to unconditionally trust the results predicted by AI, and in fact, a proper basis is required to judge the results and verify their validity and reliability. Therefore, a function of visualizing on a slide tissue image viewer to jointly grasp the slide tissue image prediction information and block prediction information about a single slide tissue image is provided.

[0159] Return to reference Fig. 9 , the slide tissue image prediction information is displayed as a text label at the lower end of the mini map at the upper right end of the slide tissue image viewer. The block prediction information is implemented in the form of a heat map, and the prediction information is represented on each block in the slide tissue image based on the position information about each block.

[0160] The location of the group M block is red hot, the group D block is blue hot, and the group N block is no hot, and the same colors can be used for slide annotation when training the model. In particular, it can be implemented as a simple "masking on and off" heat map by right-clicking when viewing the slide through the slide tissue image viewer. Through the basic functions of the slide tissue image viewer, such as zoom in / out and rotation, it can allow the slide tissue image and heat map to be viewed simultaneously at all magnifications (0.5x to 40x). Therefore, when viewing the slide tissue image, the pathologist can intuitively compare which part the artificial intelligence model infers through a single slide tissue image and how each part is inferred, while frequently masking on / off the heat map at all magnifications.

[0161] Research on the progress of AI classification models is ongoing, and ongoing feedback from pathologists on the slide-level and block-level performance of AI models applied in practice is needed to measure and improve the performance of AI models. Therefore, it is possible to allow the heatmaps and qualitative evaluations of the predictions of AI models to be recorded in each slide tissue image list. This evaluation information will be used as additional data to improve future model performance.

[0162] According to the classification in Table 2 above, if the corresponding keyword is included in the pathology diagnosis (text information), it is classified into each group, and the classification result is compared 1:1 with the classification result predicted by the artificial intelligence model, which becomes a reference for determining whether they are consistent or inconsistent.

[0163] The priority of applying classification to each keyword may be M>U>D>N. However, "sessile serrated adenoma / polyp", "sessile serrated adenoma", "sessile serrated lesion" may take precedence over the keyword of item D.

[0164] For example, "adenocarcinoma, moderately differentiated", "neoplastic lesion, suspected of malignancy", "adenoma, high-grade dysplasia", "tubulovillous adenoma, low to focal high-grade dysplasia", "focal carcinoma", "adenoma, uncertain degree", "malignancy", "atypical glandular hyperplasia, tending to neoplasm", "neuroendocrine carcinoma" or such diagnoses include "italic keywords" and are ultimately classified by "bold keywords". According to the above rules, "tubular adenoma, low-grade dysplasia", "atypical glandular hyperplasia, tending to dysplasia", "glandular hyperplasia, uncertain whether dysplasia" or "glandular hyperplasia, undetermined significance" or such diagnoses are classified as D. "Small cell nests with neuroendocrine features" is also classified as U together with "neuroendocrine tumor, degree 1 (neoplastic)".

[0165] According to the classification rules, when "tubulovillous adenoma, low to focal high-grade dysplasia" is classified as "M" by pathological diagnosis, but is classified as "D" by slide tissue image artificial intelligence prediction, making them different from each other, the case can be indicated as "inconsistent" and to be re-examined by a pathologist. For ambiguous diagnosis cases such as "atypical glandular hyperplasia, tending to be neoplastic", "glandular hyperplasia, significance unknown", and "glandular hyperplasia, uncertain whether dysplasia", their classification based on pathological diagnosis is indicated as "M, D, and D", respectively, but can be predicted as M, D, or N by slide tissue image prediction according to the performance of each case. If these atypical cases are also indicated as "inconsistent" with the prediction of artificial intelligence, they can be re-examined by a pathologist.

[0166] Furthermore, in such atypical cases, serial sections or recut slides are often generated, in which case the AI ​​prediction results may differ for each slide tissue image. For example, there may be a case where the prediction for the original slide is D, the prediction for one of the serial sections is D, the prediction for another of the serial sections is M, and the prediction for the last deeper section is N as all lesions disappear. Therefore, when the AI ​​predictions of serial slides generated as one block differ from each other, the final predictions may be labeled in the order of M>U>D>N.

[0167] In the above example, the AI ​​prediction can ultimately be labeled as "M". If the pathology diagnosis is "TA, LGD", it will be labeled as "D", so this can also be considered an "inconsistent" case. If the goal is just to improve the agreement rate (accuracy) between the expert diagnosis and the AI ​​prediction, it is sufficient to set it up as follows: based on the prediction results for each slice of the slide, it will ultimately be classified as D, D, M, and N, with the most D. However, for post-analysis quality control and rapid detection of false negatives, clinically more critical results can take precedence over the AI ​​prediction results with the largest weight.

[0168] Because it is composed of such a system, in fact, the accuracy of AI predictions is necessarily lower than that of other AI models used for diagnostic assistance purposes. However, it is inefficient to review unnecessary excessive cases, so it is necessary to ensure the ideal level of accuracy that can perform efficient and practical functions.

[0169] The histopathology reading support device 200 according to one embodiment can develop an artificial intelligence model with reliable performance that can reduce pathologists' daily potential human errors, especially false negatives, while maintaining an optical microscope-based workflow without infringing or threatening the pathologist's diagnostic responsibility, rather than diagnosing better than the pathologist, and can be applied as a "daily rapid QC tool" as the most realistic and appropriate approach for routine practice of gastrointestinal endoscopic biopsy reading.

[0170] Furthermore, the histopathology reading support device 200 according to one embodiment can prevent patients who need treatment from being missed, effectively detect false negatives due to potential human errors in routine work (practice), and immediately take necessary measures.

[0171] Furthermore, the histopathology reading support device 200 according to one embodiment can reflect artifacts and variables of the field to be directly applied to regular practice.

[0172] Furthermore, the histopathology reading support apparatus 200 according to one embodiment is not intended to provide standards for accurately screening "invasive cancer and dysplasia," "neoplastic changes and reactive changes," which are subject to and respected by the inherent authority of each pathologist.

[0173] Fig.12 is a flowchart illustrating a histopathology reading support method according to another embodiment.

[0174] Reference Fig.12 The histopathology reading support method 1000 includes an input step S1010, a preprocessing step S1020, a training step S1030, a tissue classification step S1040, a storage step S1050 and an output step S1060.

[0175] The input step S1010 inputs the slide information and the slide tissue image.

[0176] As reference Figure 4 As described above, the preprocessing step S1020 divides the inputted slide tissue image into a plurality of blocks.

[0177] like Figure 5 and Figure 7 As shown, when a lesion is present, the training step S1030 performs the following operations: the artificial intelligence model is trained using the block classification results by dividing the training slide tissue image marked with the lesion site into multiple blocks and by inferring the classification of each of the multiple blocks, and the artificial intelligence model is trained using the slide tissue classification results by inferring the classification of the slide tissue image reconstructed by integrating the multiple blocks and the classification results of the blocks.

[0178] like Figure 7 As shown, the tissue classification step S1040 generates a block classification result by using a trained artificial intelligence model to infer the classification of each of the multiple blocks, and generates a slide tissue classification result by inferring the classification of a slide tissue image reconstructed by integrating the multiple blocks and the classification results of the blocks.

[0179] The storing step S1040 stores the slide information and the slide tissue image, a plurality of blocks, block information about the blocks, block classification results, and slide tissue classification results.

[0180] The output step S1050 outputs the block classification result and the slide tissue classification result of each of the plurality of blocks.

[0181] When lesions are present, the training slide tissue image may indicate the lesion site by a single closed curve of a different color for each lesion.

[0182] The training slide tissue images may be slide tissue images scanned by a slide scanner using them as is without any artificial enhancement.

[0183] There are two or more trained artificial intelligence models, and the slide information includes an organ of the slide tissue image, and in the tissue classification step S1030, the tissue image can be classified by selecting one of the two or more artificial intelligence models according to the organ of the slide tissue image. As described above, as a trained artificial intelligence model, an artificial intelligence model trained for the stomach and colon organs has been described as an example. However, in addition to the stomach and colon models shown, models of other organs (breast, prostate, skin, etc.) can also be further developed in a similar manner and applied in the same manner as a conventional quality control system.

[0184] As reference Figure 7 and Fig. 9 As described above, in the output step S1060, the block classification result may be displayed in a unique color for each lesion location of each block, and the slide tissue classification result may be displayed as one of letters, numbers, and colors.

[0185] As reference Figure 7 As described above, the reconstructed slide tissue image is reconstructed by combining a plurality of blocks displayed in a unique color for each lesion location, and the output step S1060 may display the slide tissue analysis result indicated by letters or numbers together with the reconstructed slide tissue image.

[0186] The input step S1010 inputs a pathological diagnosis result obtained by a pathologist reading a slide corresponding to a slide tissue image under a microscope or through a screen.

[0187] Then, in the display step S1060, as shown in FIG. Fig.10 As described above, in the tissue classification step S1040, whether the slide tissue image classification result and the pathological diagnosis classification result are inconsistent can be displayed separately together with the pathological diagnosis result.

[0188] Specifically, the content input in the input step S1010 can be the content analyzed and read by the pathologist, that is, the "pathological diagnosis" information (text information). According to the classification rules, the pathological diagnosis information can be "classified" into one of the three groups M, D and N. The artificial intelligence model can independently predict one of the three groups M, D and N. The display step S1060 displays whether the corresponding "classification result of pathological diagnosis information (text information)" and "classification result of artificial intelligence model (image information)" are consistent or inconsistent.

[0189] By additionally displaying whether the slide tissue image classification result and the classification result of the pathological diagnosis are inconsistent in the tissue classification step S1040, the diagnostic quality of routine practice can be managed by establishing an automatic verification system for the diagnosis of artificial intelligence models in tissue pathology reading routine practice.

[0190] Fig.13 is a block diagram illustrating a computing system 1100 according to an embodiment of the present invention.

[0191] Reference Fig.13 , the computing system 1100 may include a memory 1110 and a processor 1120 .

[0192] The memory 1110 may store the slide tissue image in which the tissue has been captured and the plurality of blocks into which the slide tissue image is segmented, or may store them separately in, for example, a separate high-capacity storage server. The memory 1110 may be a volatile memory (e.g., SRAM or DRAM) or a non-volatile memory (e.g., NAND flash memory).

[0193] When a lesion is present, the processor 1120 may perform the following operations: train an artificial intelligence model using block classification results by dividing a training slide tissue image marked with a lesion site into a plurality of blocks and by inferring a classification of each of the plurality of blocks, and train the artificial intelligence model using slide tissue classification results by inferring a classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks; the processor 1120 may generate a block classification result by using the trained artificial intelligence model to infer a classification of each of the plurality of blocks, and generate a slide tissue classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks.

[0194] The histopathology reading support device 200 may be configured as Fig.13 The computing system 1100 shown may be configured as a GPU server, including a memory for storing scanned documents (WSI images), a GPU processor, and a general memory, but the present disclosure is not limited thereto.

[0195] The above-mentioned histopathology reading support device 200 can be realized by a computing device including at least some of a processor, a memory, a user input device and a presentation device. The memory is a medium for storing computer-readable software, applications, program modules, routines, instructions and / or data that are encoded to perform a specific task when executed by a processor. The processor can read and execute computer-readable software, applications, program modules, routines, instructions and / or data stored in the memory. The user input device can be a device for allowing a user to input a command to the processor to perform a specific task or input the data required for performing a specific task. The user input device can include a physical or virtual keyboard or keypad, a button, a mouse, a joystick, a trackball, a touch-sensitive input device or a microphone. The presentation device can include, for example, a display, a printer, a loudspeaker or a vibrator.

[0196] The computing device may include various devices, such as smartphones, tablets, laptops, desktops, servers, clients, etc. The computing device may be a single standalone device, and may include multiple computing devices operating in a distributed environment consisting of multiple computing devices cooperating with each other through a communication network.

[0197] At the same time, the computing device can be a quantum computing device rather than a classical computing device. Quantum computing devices perform operations in units of quantum bits (rather than bits). A quantum bit can have a state where 0 and 1 are superimposed at the same time, and if there are M quantum bits, 2^M states can be represented at the same time.

[0198] Quantum computing devices can use various types of quantum gates (e.g., Pauli / Rotation / Hadamard / CNOT / SWAP / Toffoli), which receive one or more quantum bits to perform quantum operations and perform designated operations, and can configure quantum circuits with special functions by combining quantum gates.

[0199] Quantum computing devices can use quantum artificial neural networks (such as QCNN or QGRNN), which can perform functions performed by traditional artificial neural networks (such as CNN or RNN) at higher speeds using fewer parameters.

[0200] Furthermore, the above-described histopathology reading support method 1000 may be performed by a computing device comprising a processor and a memory storing computer-readable software, applications, program modules, routines, instructions and / or data structures encoded to perform the above-described histopathology reading support method 1000 when executed by the processor.

[0201] The above-mentioned embodiment can be implemented in various ways. For example, the embodiment can be implemented in various ways, such as hardware, firmware, software or a combination thereof.

[0202] When implemented in hardware, the histopathology reading support method 1000 according to the present embodiment can be implemented by, for example, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, or microprocessors.

[0203] For example, the histopathology reading support method 1000 according to the embodiment can be implemented by an artificial intelligence semiconductor device, wherein neurons and synapses of a deep neural network are implemented using a semiconductor device. In this case, the semiconductor device can be a currently available semiconductor device such as SRAM, DRAM, or NAND, or can be a next-generation semiconductor device such as RRAM, STT MRAM, or PRAM, or can be a combination thereof.

[0204] When the histopathology reading support method 1000 according to the embodiment is implemented using an artificial intelligence semiconductor device, the results (weights) of training a deep learning model with software can be transferred to a synapse simulation device arranged in an array structure, or learning can be performed in the artificial intelligence semiconductor device.

[0205] When implemented in firmware or hardware, the histopathology reading support method 1000 according to the present embodiment can be implemented in the form of a device, program or function that performs the above functions or operations. The software code can be stored in a memory unit and driven by a processor. The memory unit can be located inside or outside the processor to exchange data with the processor in various known ways.

[0206] The above terms, such as "system", "processor", "controller", "component", "module", "interface", "model" or "unit" described above, can generally refer to computer-related physical hardware, a combination of hardware and software, software, or software being executed. For example, the above components can be but are not limited to processes driven by a processor, multiple processors, a controller, a control processor, an entity, an execution thread, a program and / or a computer. For example, an application executed by a controller or a processor and the controller or processor can both be components. One or more components can reside within a process and / or execution thread, and these components can be located in one device (e.g., a system, a computing device, etc.) or distributed in two or more devices.

[0207] Meanwhile, another embodiment provides a computer program stored in a computer recording medium for executing the above-mentioned histopathology reading support method 1000. Furthermore, another embodiment provides a computer-readable recording medium storing a program for implementing the above-mentioned histopathology reading support method.

[0208] The program recorded on the recording medium can be read, installed, and executed by a computer to perform the above-described steps.

[0209] Therefore, in order for a computer to read a program recorded on a recording medium and use the program to execute the implemented functions, the above-mentioned program may include codes encoded in a computer language (such as C, C++, JAVA or machine language), and the computer's processor (CPU) may read the codes through a computer device interface.

[0210] Such codes may include function codes related to functions that define the above functions, or may include control codes related to execution procedures required for a processor of a computer to execute the above functions according to a predetermined procedure.

[0211] In addition, the code may also include additional information required by the computer's processor to perform the above functions, or code related to a memory reference, which refers to the location (or address) in the computer's internal or external memory that the medium should refer to.

[0212] In addition, when the computer's processor needs to communicate with, for example, another computer or server at a remote site to perform the above-mentioned functions, the code may also include communication-related code, which involves how the computer's processor should use the computer's communication module to communicate with the remote computer or server, and what information or media should be sent / received during communication.

[0213] The above-mentioned computer-readable recording medium may include a ROM, RAM, CD-ROM, magnetic tapes, floppy disks, or optical data storage devices, for example, or may also include a carrier wave type implementation (for example, transmission via the Internet).

[0214] Furthermore, the computer-readable recording medium can be distributed to computer systems connected via a network, and computer-readable codes can be stored and executed in a distributed manner.

[0215] A programmer skilled in the art to which the present invention pertains can easily infer or modify a functional program for implementing the present invention and codes and code segments related thereto, taking into account, for example, a system environment of a computer that reads and executes the program.

[0216] Combination Fig.12 The described histopathology reading support method 1000 can be implemented in the form of a recording medium including computer executable instructions, such as an application or program module. Computer readable media can be available media that is accessible to a computer. Computer readable storage media can include volatile media, non-volatile media, detachable media, and / or inseparable media. Computer readable media can include computer storage media. Computer storage media can include volatile media, non-volatile media, detachable media, and / or inseparable media implemented in any method or scheme to store computer readable commands, data architectures, program modules, or other data or information.

[0217] The above-mentioned histopathology reading support method 1000 can be executed by an application installed on a terminal, which includes a platform equipped in the terminal or a program included in the operating system of the terminal, or can be executed by an application (or program) installed by a user on a main terminal via an application providing server (such as a network server), an application or an application store server associated with a service or method. In this sense, the above-mentioned histopathology reading support method 1000 can be implemented in an application or program installed by default on the terminal or directly installed by the user, and can be recorded in a recording medium or storage medium readable by the terminal or a computer.

[0218] Although the embodiments of the present invention have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that the present invention can be implemented in various other specific forms without changing the essence or technical spirit of the present invention. Therefore, it should be noted that the above-mentioned embodiments are provided only as examples and should not be construed as limitations. Each component can be divided into two or more units or modules to perform its (one or more) functions or (one or more) operations, and two or more components can be integrated into a single unit or module to perform its functions or operations.

[0219] It should be noted that the scope of the present invention is defined by the appended claims rather than the description of the described embodiments, and includes all modifications or changes made to the claims or equivalents of the claims.

Claims

1. A histopathology reading support device, comprising: An input unit for inputting slide information and slide tissue images; A preprocessing unit, which divides the input slide tissue image into a plurality of blocks; a tissue classification unit that generates a block classification result by inferring a classification of each of the plurality of blocks using a trained artificial intelligence model, and generates a slide tissue image classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks; a storage unit, storing the glass slide information and the glass slide tissue image, the plurality of blocks, block information about blocks, the block classification result, and the glass slide tissue image classification result; as well as an output unit, outputting the block classification result and the slide tissue image classification result of each of the plurality of blocks, Wherein, when a lesion is present, the training slide tissue image of the site marked with the lesion is divided into a plurality of blocks by using the preprocessing unit and the classification of each of the plurality of blocks is inferred to train the trained artificial intelligence model using the block classification results, and the trained artificial intelligence model is trained using the slide tissue image classification results by inferring the classification of the slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks.

2. The histopathology reading support device according to claim 1, wherein: When the lesions are present, the training slide tissue image indicates the location of the lesions by a single closed curve of a different color for each lesion.

3. The histopathology reading support device according to claim 1, wherein: The training slide tissue images use slide tissue images scanned by a slide scanner as they are without artificial enhancement.

4. The histopathology reading support device according to claim 1, wherein: Two or more trained artificial intelligence models are provided, wherein the slide information includes an organ of the slide tissue image, and Wherein, the tissue classification unit classifies the tissue image by selecting one of the two or more artificial intelligence models according to the organ of the slide tissue image.

5. The histopathology reading support device according to claim 1, wherein: The output unit indicates the block classification result with a unique color for each lesion position of each block, and indicates the slide tissue image classification result with at least one of letters, numbers or colors.

6. The histopathology reading support device according to claim 5, wherein: The reconstructed slide tissue image is reconstructed by combining a plurality of blocks indicated with a unique color for each lesion location, and The output unit displays the classification result of the glass slide tissue image indicated by the letters or the numbers together with the reconstructed glass slide tissue image.

7. The histopathology reading support device according to claim 1, wherein: The input unit inputs a pathological diagnosis, which is a slide analysis result obtained by a pathologist reading a slide corresponding to the slide tissue image through a microscope or reading the slide tissue image through a screen, and Wherein, the display unit also displays the consistency or inconsistency between the classification result of the slide tissue image by the tissue classification unit and the classification result of the pathological diagnosis, and the pathological diagnosis as the slide reading result of the pathologist.

8. The histopathology reading support device according to claim 1, wherein: When the slide tissue image classification result is inconsistent with the pathological diagnosis classification result, if the pathological diagnosis classification result is incorrect, the input unit is used to input a corrected pathological diagnosis and a reason for the diagnosis error.

9. The histopathology reading support device according to claim 8, wherein: When the slide tissue image classification result is inconsistent with the classification result of the pathological diagnosis, if the slide tissue image classification result is incorrect, the training slide tissue image with the annotations included in the corresponding slide tissue image is created, and the artificial intelligence model is additionally trained.

10. The histopathology reading support device according to claim 1, wherein: When there are multiple slides in one specimen, the output unit provides a related slide list at the upper left end of the slide tissue image viewer so that related slide tissue images can be bundled and viewed.

11. A histopathology reading support method, comprising: An input step, inputting slide information and slide tissue image; A preprocessing step, where the input slide tissue image is divided into multiple blocks; a training step, when there is a lesion, training the artificial intelligence model using block classification results by dividing a training slide tissue image of a site marked with the lesion into a plurality of blocks and by inferring the classification of each of the plurality of blocks, and training the artificial intelligence model using slide tissue image classification results by inferring the classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks; a tissue classification step of generating a block classification result by inferring a classification of each of the plurality of blocks using a trained artificial intelligence model, and generating a slide tissue image classification result by inferring a classification of a slide tissue image reconstructed by integrating the plurality of blocks and the classification results of the blocks; A storing step of storing the glass slide information and the glass slide tissue image, the plurality of blocks, block information about blocks, the block classification result, and the glass slide tissue image classification result; as well as The output step is to output the block classification result and the slide tissue image classification result of each of the plurality of blocks.

12. The histopathology reading support method according to claim 11, wherein: When the lesions are present, the training slide tissue image indicates the location of the lesions by a single closed curve of a different color for each lesion.

13. The histopathology reading support method according to claim 11, wherein: The training slide tissue images use slide tissue images scanned by a slide scanner as they are without artificial enhancement.

14. The histopathology reading support method according to claim 11, wherein: Two or more trained artificial intelligence models are provided, wherein the slide information includes an organ of the slide tissue image, and Wherein, the tissue classification step classifies the tissue image by selecting one of the two or more artificial intelligence models according to the organ of the slide tissue image.

15. The histopathology reading support method according to claim 11, wherein: The outputting step indicates the block classification result with a unique color for each lesion position of each block, and indicates the slide tissue image classification result with at least one of letters, numbers or colors.

16. The histopathology reading support method according to claim 15, wherein: The reconstructed slide tissue image is reconstructed by combining a plurality of blocks indicated with a unique color for each lesion location, and Wherein, the output step displays the classification (or inference) result of the glass slide tissue image indicated by the letters or the numbers together with the reconstructed glass slide tissue image.

17. The histopathology reading support method according to claim 16, wherein: The input step inputs a pathological diagnosis, which is a slide reading result obtained by a pathologist analyzing a slide corresponding to the slide tissue image through a microscope or analyzing the slide tissue image through a screen, and Wherein, the display step further displays the consistency or inconsistency between the classification result of the slide tissue image of the tissue classification step and the classification result of the pathological diagnosis, and the pathological diagnosis as the slide reading result of the pathologist.

18. The histopathology reading support method according to claim 11, wherein: When the slide tissue image classification result is inconsistent with the pathological diagnosis classification result, if the pathological diagnosis classification result is incorrect, a corrected pathological diagnosis and the reason for the diagnosis error are input through the input step.

19. The histopathology reading support method according to claim 18, wherein: When the slide tissue image classification result is inconsistent with the classification result of the pathological diagnosis, if the slide tissue image classification result is incorrect, the training slide tissue image with the annotations included in the corresponding slide tissue image is created, and the artificial intelligence model is additionally trained.

20. The histopathology reading support method according to claim 11, wherein: When there are multiple slides in one specimen, the output step provides a related slide list at the upper left end of the slide tissue image viewer so that related slide tissue images can be bundled and viewed.

Images