Patch-level severity determination method, slide-level severity determination method, and computing system for performing these

Abstract

A patch-level severity determination method, a slide-level severity determination method, and a computing system for performing these are disclosed. According to one aspect of the present invention, a slide-level severity determination method performed in a computing system including a first deep learning model that is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a pathological slide image into a predetermined unit size is input, includes: determining a valid grade for each of a plurality of determination target partial images obtained by dividing a predetermined determination target pathological slide image into the unit size based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input; and determining a slide-level severity grade for the entire determination target pathological slide image based on the valid grades for each of the plurality of determination target partial images constituting the determination target pathological slide image. A slide-level severity determination method is provided that includes these steps.

Description

Technical Field

[0001] The present invention relates to a patch-level severity determination method, a slide-level severity determination method, and a computing system for performing these methods. More specifically, the present invention relates to a method for determining an effective grade by utilizing the output of a deep learning model that is trained to input a pathological image and output grade information of histological severity, and a method for determining the slide-level severity for the entire tissue included in a pathological slide image by utilizing the output of a deep learning model that is trained to analyze a predetermined partial image including the tissue to determine the histological severity of the disease, and a computing system for performing these methods.

Background Art

[0002] After removing a part or all of a tissue for the treatment of a severe disease such as cancer, in order to predict the prognosis of the patient and determine the treatment method, a histopathological examination is performed on the excised tissue to diagnose the severity of the disease.

[0003] There is a consensus system for reporting histological severity for each disease, and this system grades and reports the severity based on the morphological characteristics of the tissue. For example, in the case of breast cancer, there is a grading system called the Nottingham system, which measures how well the tissue structure is maintained (tubule formation), how uniform the cell nuclei are (nuclear pleomorphism), and the level of mitotic activity, grades them from grade 1 to grade 3 respectively, and then sums them up to grade the severity of the tissue. In the case of ovarian cancer, there is a grading system called the Silverberg system, which is similar to the Nottingham system. Similarly, after grading from grade 1 to grade 3 for each category of tissue structure, cell nuclei, and mitosis, the severity is graded based on this. In the case of prostate cancer, there is a grading system called the Gleason grading system, which classifies tissue patterns into grades from grade 1 to grade 5 based on the morphological characteristics of glandular tissue, and grades the overall severity based on the proportion of areas for each grade in the whole tissue.

[0004] The biggest problem in such grading systems is that it is very difficult to accurately determine the grade of a lesion according to the criteria of the system. The morphology of biological tissue changes gradually according to the progression of the disease, and there may be tissues within a single lesion that can be classified into various grades, so it often occurs that it is difficult to determine as any one grade.

[0005] In recent years, due to the development of deep learning technology, it has become possible to develop deep learning models with accuracy exceeding human cognitive ability in the field of image recognition. As a result, diagnostic techniques for diseases using deep learning have been developed and commercialized in the medical field. Whole slide images generated by scanning biological tissue slides via a digital slide scanner are very large in scale. When analyzing these images to diagnose diseases or their severity, a method is usually used in which the whole slide image is divided into small-scale partial images (patches or tiles), each of which is analyzed by a deep learning model, and the results are combined to generate a diagnostic result for the entire slide.

[0006] When analyzing tissue images with a deep learning model to recognize the severity grade of the tissue, as described above, since the distinction between grades is ambiguous, there is a problem that the accuracy of the recognition result of the model for tissues corresponding to the boundary between grades is not high. By applying a dual-class annotation method (Korean Registered Patent No. 10-2162895; Disease diagnosis system and method for assisting dual classes) that allows ambiguity in the determination of tissues on the boundary, etc., it is possible to make the analysis result of the deep learning model for tissues with ambiguous grades be on the boundary between grades, but it is very difficult to train it to strongly support any one of the grades.

[0007] On the one hand, when the analysis results of the deep learning model do not strongly support any one grade in this way, the histological severity of the organization can be determined to be at the boundary between the two grades. By calculating the histological severity for the entire slide considering such boundary information, it is considered that a more detailed severity grading becomes possible compared to the conventional histological severity reporting system. Nagpal et al. mentioned in their research on the development of histological severity determination technology for prostate cancer (Nagpal, K., Foote, D., Liu, Y. et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. npj Digit. Med. 2, 48(2019). https: / / doi.org / 10.1038 / s41746-019-0112-2) a method of assigning a value corresponding to the boundary to tissues with ambiguous grade determination. However, in this research, only the possibility of such a method and an example of assigning a value corresponding to the boundary according to the degree of grade agreement among a large number of pathologists were mentioned, and a method of assigning a value corresponding to the grade boundary to lesions using a deep learning model and a method of grading the severity of the entire tissue by integrating the finely assigned severity grades for each lesion were not mentioned.

[0008] On the other hand, in the existing histological severity reporting systems for each disease, the severity of the entire tissue was mainly determined in the following ways. For example, in the case of the Gleason grading system for prostate cancer, the overall severity is determined by the sum of the top two severity grades based on the area size. When examining this in detail, it can be seen that the average severity of the entire area is displayed as follows.

[0009]

Table 1

[0010] Also, in the case of breast cancer, for each region observed, an average grade is assigned to each category of tissue structure, cell nucleus, and cell division, and these are totaled to assign an overall grade. Thus, the severity of the entire tissue is considered to be determined in the form of the average severity of the entire region. However, in some cases, when there is a lesion with a high-grade severity, regardless of its size, a high severity may have to be assigned. Therefore, when histological severity information for each partial image is output, methods such as using the maximum value or the average value in the method of integrating this information can be considered. However, as seen in the "rare cases" of the prostate cancer severity groups described above, simply determining the overall severity using only the general maximum / minimum / average may not be sufficient in some cases, and it may be possible to develop a more accurate method for determining the severity at the slide level by utilizing a machine learning (deep learning) model or the like.

Summary of the Invention

Problems to be Solved by the Invention

[0011] The technical problem to be achieved by the present invention is to provide a method for determining the severity at the slide level that can more finely classify the severity of the whole slide image of a biological tissue using deep learning technology.

[0012] Also, to provide a method for effectively determining the effective grade of a biological tissue image.

Means for Solving the Problems

[0013] According to one aspect of the present invention, there is provided a slide-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a pathological slide image into a predetermined unit size is input. The method includes: determining, for each of a plurality of determination target partial images obtained by dividing a predetermined determination target pathological slide image into the unit size, a valid grade for the determination target partial image based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input; and determining a slide-level severity grade for the entire determination target pathological slide image based on the valid grade for each of the plurality of determination target partial images constituting the determination target pathological slide image. The determination result for the partial image output by the first deep learning model is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system. The step of determining the valid grade for the determination target partial image based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input includes determining the valid grade for the determination target partial image based on the likelihood numerical value for each grade on the severity grade system for the determination target partial image and the grade score for each grade on the severity grade system. A slide-level severity determination method is provided.

[0014] In one embodiment, the step of determining the effective grade for the partial image to be determined based on the probability values for each grade on the severity grade system output by the first deep learning model into which the partial image to be determined is input, and the grade scores for each grade on the severity grade system includes calculating a weighted average of the grade scores for each grade on the severity grade system, using the probability values for each grade on the severity grade system as weights, to determine the effective grade for the partial pathological image to be determined; calculating a weighted average of the grade scores for a plurality of upper grades with high probability values, using the probability values of each of the plurality of upper grades as weights, to determine the effective grade for the partial pathological image to be determined; or determining the grade score of the topmost grade with the highest probability value among the grades on the severity grade system as the effective grade for the partial pathological image to be determined, and may include a severity determination method at the slide level.

[0015] According to another aspect of the present invention, there is provided a slide-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a pathological slide image into a predetermined unit size is input. The method includes: determining a valid grade for each of a plurality of determination target partial images obtained by dividing a predetermined determination target pathological slide image into the unit size, based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input; and determining a slide-level severity grade for the entire determination target pathological slide image, based on the valid grades for each of the plurality of determination target partial images constituting the determination target pathological slide image. The determination result for the partial image output by the first deep learning model includes a determination result for each pixel constituting the partial image, and the determination result for each pixel is a likelihood numerical value for each grade in a predetermined histological severity grade system for a predetermined disease. Each grade in the severity grade system is pre-assigned a predetermined grade score. The step of determining a valid grade for the determination target partial image based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input includes: determining a valid grade for each determination target pixel constituting the determination target partial image, based on the likelihood numerical value for each grade in the severity grade system for the determination target pixel and the grade score for each grade in the severity grade system; and determining a valid grade for the determination target partial image, based on the valid grades for each determination target pixel constituting the determination target partial image. A slide-level severity determination method is provided.

[0016] In one embodiment, the step of determining the effective grade for the pixel to be determined based on the probability values for each grade on the severity grade system of the pixel to be determined output by the first deep learning model and the grade scores for each grade on the severity grade system includes calculating a weighted average of the grade scores for each grade on the severity grade system, using the probability values for each grade on the severity grade system of the pixel to be determined as weights, to determine the effective grade of the pixel to be determined; calculating a weighted average of the grade scores for each of a plurality of upper-level grades with high probability values, using the probability values of each of the plurality of upper-level grades as weights, to determine the effective grade of the pixel to be determined; or determining the grade score of the topmost grade with the highest probability value as the effective grade of the pixel to be determined.

[0017] In one embodiment, the step of determining the effective grade for the partial image to be determined based on the effective grades of the pixels to be determined that make up the partial image to be determined may include calculating a representative value of the effective grades of the pixels to be determined that make up the partial image to be determined and determining the effective grade for the partial image to be determined.

[0018] In one embodiment, the step of calculating a representative value of the effective grades of the pixels to be determined that make up the partial image to be determined and determining the effective grade for the partial image to be determined may include calculating any one of the maximum value, minimum value, average value, median value, and average value within the IQR (InterQuartile Range) of the effective grades of the pixels to be determined that make up the partial image to be determined as the representative value.

[0019] In one embodiment, the step of determining the severity grade at the slide level for the entire pathological slide image to be determined based on the effective grade for each of the plurality of partial images to be determined that constitute the pathological slide image to be determined may include the step of calculating a representative value of the effective grades of each partial image to be determined that constitutes the pathological slide image to be determined, and determining the severity grade at the slide level for the entire pathological slide image to be determined.

[0020] In one embodiment, the step of calculating a representative value of the effective grades of each partial image to be determined that constitutes the pathological slide image to be determined, and determining the severity grade at the slide level for the entire pathological slide image to be determined may include the step of calculating, as the representative value, any one of the maximum value, minimum value, average value, median value, and average value within the IQR (InterQuartile Range) of the effective grades of each partial image to be determined that constitutes the pathological slide image to be determined.

[0021] In one embodiment, when a data set including the effective grades for each of the plurality of partial images obtained by dividing the pathological slide image into the unit size is input, the computing system further includes a second deep learning model which is an artificial neural network pre-trained to output an output value for determining the severity grade for the pathological slide image. The step of determining the severity grade for the pathological slide image to be determined based on the effective grades for each of the plurality of partial images to be determined that constitute the pathological slide image to be determined may include the step of determining the severity grade for the pathological slide image to be determined based on the output value output by the second deep learning model when the data set including the effective grades for each of the plurality of partial images to be determined that constitute the pathological slide image to be determined is input.

[0022] In one embodiment, the second deep learning model is an artificial neural network pre-trained by a second deep learning model learning method. The second deep learning model learning method includes steps of: obtaining a plurality of training pathological slide images; for each of the plurality of training pathological slides, generating a data set corresponding to the training pathological slide image; and inputting each of the data sets corresponding to the plurality of training pathological slide images into the second deep learning model to train the second deep learning model. The step of generating a data set corresponding to the training pathological slide image may include steps of: for each of a plurality of partial images for generating a data set obtained by dividing the training pathological slide image into the unit size, determining a validity grade for the partial image for generating a data set based on a determination result output by the first deep learning model into which the partial image for generating a data set is input; and generating a data set including the validity grade for each of the plurality of partial images for generating a data set.

[0023] According to another aspect of the present invention, there is provided a patch-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when a partial image obtained by dividing a pathological slide image into a predetermined unit size is input. The method includes: inputting a predetermined determination target partial image into the first deep learning model, and obtaining a determination result for the determination target partial image output by the first deep learning model (the determination result for the determination target partial image is a likelihood numerical value for each grade on a predetermined severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system); and determining a valid grade for the determination target partial image based on the likelihood numerical value for each grade on the severity grade system output by the first deep learning model into which the determination target partial image is input, and the grade score for each grade on the severity grade system.

[0024] According to another aspect of the present invention, there is provided a patch-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when a partial image obtained by dividing a pathological slide image into a predetermined unit size is input. The method includes: inputting a predetermined target partial image into the first deep learning model, and obtaining a determination result for the target partial image output by the first deep learning model (the determination result for the target partial image output by the first deep learning model includes determination results for each pixel constituting the target partial image, and the determination result for each pixel is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system); determining, for each target pixel constituting the target partial image, a valid grade for the target pixel based on the likelihood numerical value for each grade on the severity grade system of the target pixel and the grade score for each grade on the severity grade system; and determining a valid grade for the target pathological image based on the valid grades of each target pixel constituting the target partial image.

[0025] According to another aspect of the present invention, there is provided a computer program installed in a data processing device and recorded on a medium for performing the above-described method.

[0026] According to another aspect of the present invention, there is provided a computer-readable recording medium having recorded thereon a computer program for performing the above-described method.

[0027] According to another aspect of the present invention, there is provided a computing system including a processor and a memory. The memory stores a first deep learning model which is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a computer program and a pathological slide image into a predetermined unit size is input. When the computer program is executed by the processor, the computing system is controlled to perform the above-described method.

[0028] In one embodiment, the memory may further store a second deep learning model which is an artificial neural network pre-trained to output an output value for determining the severity grade for the pathological slide image when a data set including the effective grade for each of a plurality of partial images obtained by dividing the pathological slide image into the unit size is input.

Advantages of the Invention

[0029] According to the technical idea of the present invention, by analyzing a partial image (for example, a patch or a tile) which is a part of the whole slide image divided into a predetermined unit size and utilizing the output of a deep learning model trained to determine the histological severity of a predetermined disease, it is possible to provide a slide-level severity determination method capable of more finely classifying the severity for the entire tissue (in other words, the whole slide image).

[0030] Further, by utilizing the output of a deep learning model trained to output grade information of the histological severity for a biological tissue image, it is possible to provide a method capable of effectively determining the effective grade of the partial image itself.

Brief Description of the Drawings

[0031] To more fully understand the drawings cited in the detailed description of the present invention, a brief description of each drawing is provided.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5a

Figure 5b

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0032] Since the present invention can be subjected to various transformations and can have various embodiments, specific embodiments will be shown in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood to include all transformations, equivalents, and alternatives included in the spirit and technical scope of the present invention. When explaining the present invention, if it is determined that a specific description of related known technologies will obscure the gist of the present invention, the detailed description thereof will be omitted.

[0033] Terms such as first and second can be used to describe various components, but the above components should not be limited by the above terms. The above terms are only used for the purpose of distinguishing one component from another.

[0034] The terms used in this application are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0035] In this specification, terms such as "including" or "having" are used to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and it should be understood that they do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0036] Also, in this specification, when a certain component "transmits" data to another component, it means that the above component can directly transmit the above data to the above other component, or can also transmit the above data to the other component via at least one other component. Conversely, when a certain component "directly transmits" data to another component, it means that the above data is transmitted from the above component to the above other component without passing through another component.

[0037] Hereinafter, with reference to the accompanying drawings, the present invention will be described in detail centering on embodiments of the present invention. The same reference numerals presented in each drawing indicate the same members.

[0038] The patch-level severity determination method and the slide-level severity determination method according to the technical idea of the present invention may be performed by a computing system. The computing system may be a data processing device having computing capabilities, and may include not only a server which is generally a data processing device to which a client can be connected via a network, but also computing devices such as a personal computer and a mobile terminal.

[0039] FIG. 1 is a diagram schematically showing an example of the configuration of a computing system according to an embodiment of the present invention. The computing system 100 may physically have a configuration as shown in FIG. 1. The computing system 100 may include a memory 120 in which a computer program 123 and a first deep learning model 121 for realizing the technical idea of the present invention are stored, and a processor 110 for executing the computer program 123 stored in the memory 120. According to an embodiment, a second deep learning model 122 may be further stored in the memory 120.

[0040] It can be easily inferred by an average expert in the technical field of the present invention that the processor 110 can be named with various names such as a CPU, an APU, a microprocessor, an ASIC, etc. according to the embodiment of the computing system 100. Further, the computing system 100 may be realized by organically combining a plurality of physical devices. In such a case, at least one processor 110 is provided for each physical device, and it can be easily inferred by an average expert in the technical field of the present invention that the computing system 100 of the present invention can be realized. The processor 110 may further include a GPU used for learning the first deep learning model 121 and / or the second deep learning model 122.

[0041] The memory 120 stores the program 123 and may be implemented as any type of storage device accessible by the processor 110 to drive the program 123. Also, depending on the hardware embodiment, the memory 120 may be implemented as a plurality of storage devices rather than a single storage device. Further, the memory 120 may include not only a main memory but also a temporary memory. Also, it may be implemented as a volatile memory or a non-volatile memory. The memory 120 may include, for example, flash memory, ROM, RAM, EEROM, EPROM, EEPROM, a hard disk, and registers. Alternatively, the memory 120 may be defined to include any form of information storage means in which the program 123 is stored and can be driven by the processor 110.

[0042] Depending on the embodiment, the computing system 100 may further include various peripheral devices (peripheral devices 1 to peripheral device M, 130-1 to 130-M). For example, an average expert in the technical field of the present invention would easily be able to infer that a keyboard, a display device, a graphics card, a network device, a storage device, etc. may also be further included in the computing system 100 as peripheral devices.

[0043] The computing system 100 may be implemented as a single physical device, but an average expert in the technical field of the present invention would easily be able to infer that a plurality of physical devices can be organically combined as needed to realize the computing system 100 according to the technical idea of the present invention.

[0044] Hereinafter, in this specification, that a predetermined module performs a certain function means that the processor 110 drives the program 123 provided in the memory 120 to perform the above function, which an average expert in the technical field of the present invention would easily be able to infer.

[0045] In this specification, a deep learning model may include a multi-layer perceptron model and may mean a set of information representing a series of design matters that define an artificial neural network.

[0046] In one embodiment, the deep learning model may be a convolutional neural network. As is well known, a convolutional neural network may include an input layer, a plurality of hidden layers, and an output layer. Each of the plurality of hidden layers may include a convolutional layer and a pooling layer (or a subsampling layer). A convolutional neural network can be defined by functions, filters, strides, weight coefficients, etc. for defining such each layer. Also, the output layer can be defined as a fully connected feed-forward layer.

[0047] The design matters for each layer constituting a convolutional neural network are widely known. For example, for each of the number of layers included in the plurality of layers, the convolutional function, the pooling function, and the activation function for defining the plurality of layers, known functions may be used, or functions separately defined to implement the technical idea of the present invention may be used.

[0048] Examples of convolution functions include discrete convolution sums. As examples of pooling functions, max pooling, average pooling, etc. may be used. Examples of activation functions include sigmoid, tangent hyperbolic (tanh), ReLU (rectified linear unit), etc. When such design matters of the convolutional neural network are defined, the convolutional neural network with the defined design matters can be stored in a storage device. Also, when the convolutional neural network is learned, the weight coefficients corresponding to each layer can be specified. That is, the learning of the convolutional neural network can mean a process in which the weight coefficients of each layer are determined. Also, when the convolutional neural network is learned, the learned convolutional neural network can input input data through an input layer and output output data through a predefined output layer.

[0049] The neural network according to an embodiment of the present invention may be defined by selecting any one or more of the widely known design matters as described above, or unique design matters may be defined for the neural network.

[0050] The neural network may be a classification neural network used for classification of input data. The classification neural network may be a neural network for classifying a determination result for input data into any one of a plurality of predefined results.

[0051] According to an embodiment, the neural network may be a segmentation neural network. The segmentation neural network is a neural network for identifying regions (e.g., regions of disease onset) that satisfy specific conditions in an input image, and may also be referred to as a pixel-level classification neural network that performs classification on a pixel-by-pixel basis.

[0052] On the other hand, the first deep learning model 121 may be an artificial neural network that has been pre-trained to output a determination result for an input partial image when a partial image obtained by dividing a pathological slide image into a predetermined unit size is input. The pathological slide image may be an entire slide image obtained by scanning a slide of a pathological specimen with a digital scanner.

[0053] The digital slide image of the pathological specimen can be generated by slicing the pathological specimen to produce a glass slide, staining this with a predetermined stain, and digitizing it. The pathological specimen may be a biopsy taken from various organs of the human body and a biological tissue excised by surgery.

[0054] The partial image input to the first deep learning model 121 is a part of a pathological slide image divided into a predetermined unit size, and may be referred to as a patch or a tile.

[0055] In one embodiment, the first deep learning model 121 may be a patch-level classification neural network. In this case, the first deep learning model 121 may be an artificial neural network that has been pre-trained to output a determination result for an input pathological image when a partial image is input. The determination result for the pathological image output by the first deep learning model is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease. A predetermined grade score may be pre-assigned to each grade on the severity grade system. For example, the severity grade system may be the Nottingham system, the Silverberg system, the Gleason grading system, or the like.

[0056] FIG. 2 is a diagram for explaining the first deep learning model 121 according to an embodiment of the present invention.

[0057] Referring to FIG. 2, the first deep learning model 121 may receive the partial image 1 via an input layer and output a determination result 2 for the input partial image 1 via an output layer. The determination result 2 may be, for example, a likelihood numerical value for each grade of the severity grade of prostate cancer according to the Gleason grade system. At this time, the sum of the likelihood numerical values for each grade may be 1.

[0058] On the other hand, a grade score 3 may be pre-assigned to each grade on the Gleason grade system.

[0059] FIG. 3 is a flowchart showing an example of a patch-level severity determination method performed by the computing system 100. As shown in FIG. 2, the computing system 100 can use the first deep learning model 121 to determine the severity of a partial image (more precisely, the valid grade on a predetermined histological severity grade system for a predetermined disease), in other words, the patch-level severity.

[0060] Referring to FIG. 3, the computing system 100 can input a predetermined partial image to be determined into the first deep learning model 121 and obtain the determination result for the partial image to be determined output by the first deep learning model 121 (S100). At this time, as described above, the determination result for the partial image to be determined is a probability value for each grade on a predetermined severity grade system for a predetermined disease, and a predetermined grade score may be pre-assigned to each grade on the severity grade system.

[0061] Thereafter, the computing system 100 can determine the effective grade for the partial image to be determined based on the determination result for the partial image to be determined (S110).

[0062] In one embodiment, the computing system 100 can calculate the weighted average of the grade scores for each grade on the severity grade system using the probability value for each grade on the severity grade system as a weight, and determine the effective grade for the partial image to be determined. Taking the case of FIG. 2 as an example, the computing system 100 can input the partial image 1 in FIG. 2 into the first deep learning model 121 to obtain the probability value 2 for each Gleason grade, and calculate the weighted average of the grade scores for each grade on the Gleason grade system using the probability value for each grade on the Gleason grade system as a weight. At this time, "benign" can be excluded when calculating the weighted average, and the effective grade calculated in the case of FIG. 2 is as follows (rounded off from the fourth decimal place).

[0063] (3×0.1 + 4×0.5 + 5×0.3) / (0.1 + 0.5 + 0.3)=3.8 / 0.9 = 4.222

[0064] In another embodiment, the computing system 100 can calculate a weighted average of grade scores for a plurality of top grades, each with a high probability value among the grades on the severity grade system, using the probability value of each of the plurality of top grades as a weight, and determine the effective grade for the partial image to be determined. Assuming that the computing system 100 calculates the weighted average of the corresponding grades with the probability values for the top two grades as the weights as the effective grade, the computing system 100 can calculate the effective grade in the case of FIG. 2 as follows.

[0065] (4 × 0.5 + 5 × 0.3) / (0.5 + 0.3) = 3.5 / 0.8 = 4.375

[0066] In another embodiment, the computing system 100 can determine, as the effective grade for the partial image to be determined, the grade score of the topmost grade with the highest probability value among the grades on the severity grade system. In the case of FIG. 2, the computing system 100 can determine the grade score 4 of grade (G4) with the largest probability of 0.5 as the effective grade.

[0067] On the other hand, the first deep learning model 121 is a pre-trained artificial neural network, and FIG. 4 is a flowchart showing the process of training the first deep learning model 121.

[0068] Referring to FIG. 4, the computing system 100 can obtain a plurality of partial images for training (S200). At this time, any one of the grades on the histological severity grade system may be tagged to each of the plurality of partial images for training. The plurality of partial images for training can be generated by dividing one or more whole slide images for training into unit sizes.

[0069] The computing system 100 can input each of a plurality of partial images for learning into the first deep learning model 121 to learn the first deep learning model 121.

[0070] To schematically explain the deep learning process based on the learning data tagged with the correct labels, when the deep learning model input with the learning data outputs a result, the error between the output result and the correct label is reflected in the deep learning model through the backpropagation process. In the backpropagation process, techniques such as the gradient descent method can be applied. Since the specific process by which the model is learned from the learning data through such a deep learning process is very widely known, further detailed explanation is omitted.

[0071] The embodiments of the present invention described with reference to FIGS. 2 to 4 relate to the case where the first deep learning model 121 is a classification neural network, that is, a neural network for determining to which class (grade) the image input to the first deep learning model 121 belongs.

[0072] By the way, according to another embodiment of the present invention, the first deep learning model 121 may be a segmentation neural network or a pixel-level classification neural network, which will be described in more detail below with reference to FIGS. 5 and 6.

[0073] FIGS. 5a and 5b are diagrams for explaining the first deep learning model 121-1 which is the segmentation neural network of the present invention.

[0074] First, referring to FIG. 5a, the first deep learning model 121-1 may receive the partial image 10 through the input layer, and the partial image 10 may be composed of a plurality of pixels (e.g., 11). In the example of FIG. 5a, a case where the partial image 10 is composed of 4×4 pixels is shown. On the other hand, the first deep learning model 121-1 may output a determination result 20 for the input partial image 10, and the determination result 20 for the partial image 10 may include determination results (e.g., 21) for each pixel constituting the partial image 10. In the example of FIG. 5a, the determination result 20 for the partial image 10 may include determination results for each of the 4×4 pixels.

[0075] FIG. 5b is a diagram showing a determination result for any one of the pixels 11 constituting the partial image 10 in FIG. 5a. Referring to FIG. 5b, the determination result 21 of the pixel 11 may be, for example, a probability value for each grade of the severity grade of prostate cancer by the Gleason grading system. At this time, the sum of the probability values for each grade may be 1. On the other hand, a grade score 3 may be pre-assigned for each grade on the Gleason grading system.

[0076] FIG. 6 is a flowchart showing another example of a patch-level severity determination method performed by the computing system 100. The computing system 100 can determine the severity (more precisely, the effective grade on a predetermined histological severity grading system for a predetermined disease) for the partial image using the first deep learning model 121-1 in FIG. 5a.

[0077] Referring to FIG. 6, the computing system 100 can input a predetermined partial image to be determined into the first deep learning model 121-1 and obtain a determination result for the partial image to be determined output by the first deep learning model 121-1 (S150). At this time, the determination result for the partial image to be determined may include, as described above, the determination result for each determination target pixel constituting the partial image to be determined. The determination result for each determination target pixel is a probability value for each grade on a predetermined severity grade system for a predetermined disease, and a predetermined grade score may be assigned in advance to each grade on the severity grade system.

[0078] Thereafter, the computing system 100 can determine the effective grade of the determination target pixel p based on the determination result of the first deep learning model 121-1 for the determination target pixel p for each determination target pixel p constituting the partial image to be determined (S160).

[0079] There can be various methods for determining the effective grade of the determination target pixel p.

[0080] In one embodiment, the computing system 100 can calculate a weighted average of the grade scores for each grade on the severity grade system, using the probability value for each grade on the severity grade system as a weight, to determine the effective grade for the determination target pixel p. Taking the case of FIG. 5b as an example, the computing system 100 can obtain the probability value 21 for each Gleason grade for pixel 11 in FIG. 5b and calculate the weighted average of the grade scores for each grade on the Gleason grade system, using the probability value for each grade on the Gleason grade system as a weight. At this time, benign can be excluded when calculating the weighted average, and the effective grade calculated in the case of FIG. 5b is as follows.

[0081] (3×0.1 + 4×0.3 + 5×0.4) / (0.1 + 0.3 + 0.4) = 3.5 / 0.8 = 4.375

[0082] In another embodiment, the computing system 100 can calculate a weighted average of the grade scores for a plurality of top grades, each with a high probability value among the grades on the severity grade system, using the probability value of each of the plurality of top grades as a weight, and determine the effective grade for the pixel p to be determined. Assuming that the computing system 100 calculates the weighted average effective grade of the corresponding grade using the probability values for the top two grades with the highest probabilities, the computing system 100 can calculate the effective grade of pixel 11 in the case of Fig. 5b as follows (rounded off from the fourth decimal place).

[0083] (4×0.3 + 5×0.4) / (0.3 + 0.4) = 3.2 / 0.7 = 4.571

[0084] In another embodiment, the computing system 100 can determine the grade score of the topmost grade with the highest probability value among the grades on the severity grade system as the effective grade for the pixel p to be determined. In the case of Fig. 5b, the computing system 100 can determine the grade score 5 of the grade (G5) with the highest probability of 0.4 as the effective grade.

[0085] Thereafter, the computing system 100 can determine the effective grade for the partial image to be determined based on the effective grades of the pixels constituting the partial image to be determined (S170).

[0086] In one embodiment, the computing system 100 can calculate the average value of the effective grades of the pixels constituting the partial image to be determined and determine this as the effective grade for the partial image to be determined.

[0087] However, it is not necessarily the case that the average value of the effective grades of the pixels is determined as the effective grade of the partial image to be determined. This is just an example for convenience of understanding. Depending on the embodiment, the computing system 100 can calculate a representative value of the effective grades of the pixels constituting the partial image to be determined through various statistical techniques, and determine this as the effective grade for the partial image to be determined. The representative value of two or more values may be the average value of these values, but is not limited thereto, and may be any one of these values or a value calculated by at least a part of these values. For example, the representative value of two or more values may be, in addition to the average value, the maximum value, the minimum value, the median value, the n-th quantile value, the average value within the IQR (interquartile range), or the average value of only the values that satisfy a predetermined condition (for example, the average value of the remaining values excluding the values outside a certain range), the mode of the histogram, etc.

[0088] The first deep learning model 121-1 shown in FIG. 5a is also a pre-trained artificial neural network. The process of training the first deep learning model 121-1 shown in FIG. 5a is substantially the same as the process shown in FIG. 4, except that the information tagged to the training image is information about the lesion area or information about the severity grade for each pixel, so a detailed description is omitted.

[0089] On the other hand, the computing system 100 can comprehensively determine the effective grade of the partial image determined by the above-described patch-level severity determination method, and determine the slide-level severity of the pathological slide image composed of the partial images. FIG. 7 is a flowchart showing an example of the slide-level severity determination method performed by the computing system 100.

[0090] Referring to FIG. 7, the computing system 100 can divide the pathological slide image S to be determined into a predetermined unit size (S300). Assuming that the pathological slide image S to be determined can be divided into N partial images, the divided partial images are P 1 from PN It may be. The unit size may be the same as the size of the input data input to the first deep learning model 121, and the size may be a pixels × b pixels (a and b are integers of 1 or more, respectively).

[0091] The computing system 100 has a plurality of partial images P to be determined 1 from P N For each of them, the partial image P to be determined i (where the integer i is from 1 to N) is input to the first deep learning model 121 (S310, S320), and the partial image P to be determined i (where the integer i is from 1 to N) is output by the first deep learning model 121 to which it is input, and based on the determination result for the partial image P i The effective grade D for the partial image P to be determined i can be determined (S330). i

[0092] In step S330, the computing system 100 can use the patch-level severity determination method described with reference to FIG. 3 or FIG. 6 to determine the effective grade D for the partial image P to be determined i i

[0093] Thereafter, the computing system 100 can determine the slide-level severity grade for the pathological slide image S to be determined based on the effective grades (D 1 to D N ) for each of the plurality of partial images to be determined (S340).

[0094] FIG. 8 is a diagram showing a specific example to which the slide-level severity determination method according to an embodiment of the present invention is applied.

[0095] Referring to FIG. 8, the computing system 100 can obtain a predetermined pathological slide image 10 to be determined and divide it into a plurality of partial images 20 having a predetermined unit size (S300). FIG. 8 shows an example of dividing into 15 partial images 20-1 to 20-15.

[0096] On the other hand, the computing system 100 inputs each of the divided plurality of partial images 20-1 to 20-15 into the first deep learning model 121 (S310, S320), and based on the determination results for each of the plurality of partial images 20-1 to 20-15 output by the first deep learning model 121, an effective grade 30 for each of the plurality of partial images 20-1 to 20-15 can be determined (S330).

[0097] Thereafter, the computing system 100 can determine the severity grade at the slide level for the pathological slide image 10 to be determined based on the effective grade 30 for each of the plurality of partial images 20-1 to 20-15 (S340).

[0098] The method for determining the severity grade at the slide level in step S340 of FIG. 7 can be various.

[0099] In one embodiment, the computing system 100 can calculate a representative value of the effective grade of each of the plurality of partial images to be determined that make up the pathological slide image to be determined, and determine the calculated representative value as the severity grade at the slide level for the entire pathological slide image to be determined.

[0100] For example, the computing system 100 can calculate, as the representative value, the maximum value, minimum value, average value, median value, average value within the IQR (InterQuartile Range), or the average value of only the values that satisfy a predetermined condition (for example, the average value of the remaining values excluding the values outside a certain range), the mode of the histogram, etc. of the effective grade of each of the partial images to be determined that make up the pathological slide image to be determined.

[0101] When considering more specifically through the example of FIG. 8, when the computing system 100 uses the average value as the representative value of the effective grade of each of the plurality of partial images to be determined, in the example of FIG. 8, the computing system 100 can determine 3.46, which is the average value of the effective grades of the 15 partial images 20-1 to 20-15 as a whole, as the severity grade at the slide level of the pathological slide image 10 to be determined.

[0102] Alternatively, when the computing system 100 uses the maximum value as the representative value of the effective grade of each of the plurality of partial images to be determined, in the example of FIG. 8, the computing system 100 can determine 4.2, which is the maximum value among the effective grades of the 15 partial images 20-1 to 20-15, as the severity grade at the slide level of the pathological slide image 10 to be determined.

[0103] Alternatively, when the computing system 100 uses the minimum value as the representative value of the effective grade of each of the plurality of partial images to be determined, in the example of FIG. 8, the computing system 100 can determine 2.9, which is the minimum value among the effective grades of the 15 partial images 20-1 to 20-15, as the severity grade at the slide level of the pathological slide image 10 to be determined.

[0104] Alternatively, when the computing system 100 uses, as the representative value of the effective grade of each of the plurality of partial images to be determined, the average value of the upper part of the effective grades with larger values among the effective grades for each of the plurality of partial images to be determined, in the example of FIG. 8, the computing system 100 can determine 4.1(=(4.2 + 4.1 + 4) / 3), which is the average value of the top 3 among the effective grades of the 15 partial images 20-1 to 20-15, as the severity grade at the slide level of the pathological slide image 10 to be determined.

[0105] In another embodiment, the computing system 100 may also determine, as the severity grade at the slide level for the entire pathological slide image to be determined, the average value of the effective grades that are equal to or higher than a predetermined threshold among the effective grades for each of the plurality of partial images to be determined. Alternatively, when the computing system 100 uses, as the representative value of the effective grades of each of the plurality of partial images to be determined, the average value of the effective grades that are equal to or higher than a predetermined threshold among the effective grades for each of the plurality of partial images to be determined, in the example of FIG. 8, the computing system 100 may determine 3.9, which is the average value of the effective grades equal to or higher than the threshold value of 3.5 among the effective grades of the 15 partial images 20-1 to 20-15, as the severity grade at the slide level of the pathological slide image 10 to be determined.

[0106] In addition to this, it goes without saying that there may be various methods for determining the severity grade at the slide level in step S340 of FIG. 7.

[0107] In one embodiment, the computing system 100 may determine the severity grade at the slide level using deep learning, and at this time, the second deep learning model 122 may be used.

[0108] The second deep learning model 122 may be an artificial neural network that is pre-trained to output an output value for determining the severity grade for the pathological slide image when a data set including the effective grades for each of the plurality of partial images obtained by dividing the pathological slide image into unit sizes is input. For example, the second deep learning model 122 may be a neural network that inputs the data set 30 composed of the effective grades of each of the plurality of partial images shown in FIG. 8 and outputs an output value for determining the severity 40 at the slide level for the pathological slide image 10.

[0109] According to this embodiment, in step S340, the computing system 100 determines the severity grade at the slide level for the determination target pathological slide image S based on the output values output by the second deep learning model 122 for which a data set including the effective grades D 1 to D N for each of the plurality of determination target partial images constituting the determination target pathological slide image S is input.

[0110] The specific process of determining the severity grade at the slide level for the determination target pathological slide image S based on the output values output by the second deep learning model 122 may be the same as the determination process of the effective grades shown in FIG. 3 or FIG. 6. For example, the second deep learning model 122 can output the probability values for each grade on the severity grade system in the same process as shown in FIG. 3, and the computing system 100 can calculate the weighted average of the grade scores for each grade on the severity grade system with the probability values for each grade on the severity grade system as weights, the weighted average of the grade scores for each of the plurality of upper grades with high probability values among the grades on the severity grade system as weights, the grade score of the topmost grade with the highest probability value among the grades on the severity grade system, etc., as the severity grade at the slide level.

[0111] The second deep learning model 122 may be a neural network that has been pre-trained before being used to determine the severity grade at the slide level. FIG. 9 is a diagram showing the process of training the second deep learning model 122.

[0112] Referring to FIG. 9, the computing system 100 can acquire the learning pathological slide image R and divide the learning pathological slide image R into unit sizes (S400). Assuming that the learning pathological slide image R is divided into N partial images, each of the divided partial images for data set generation is Q 1 to Q N and so on.

[0113] After that, for each of the plurality of partial images Q 1 to Q N for data set generation, the partial image Q i (where the integer i is from 1 to N) is input to the first deep learning model 121 (S410, S420), and based on the determination result output by the first deep learning model 121 into which the partial image Q i is input, the effective grade E i for the partial image Q i can be determined (S430).

[0114] After that, the computing system 100 can input a data set including the effective grades E 1 to E N for each of the plurality of partial images for data set generation into the second deep learning model 122 and can train the second deep learning model 122.

[0115] On the other hand, the method according to the embodiment of the present invention is realized in the form of program instructions readable by a computer and can be stored in a computer-readable recording medium. The control program and the target program according to the embodiment of the present invention can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data readable by a computer system is stored.

[0116] The program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those that can be used as known to those skilled in the software field.

[0117] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Further, the computer-readable recording medium may be distributed in a computer system connected by a network and can store and execute computer-readable code in a distributed manner.

[0118] Examples of program instructions include not only machine language code created by a compiler but also high-level language code executable by a computer, such as an apparatus that electronically processes information using an interpreter or the like.

[0119] The above-described hardware device can be configured to operate as one or more software modules for performing the operations of the present invention, and vice versa.

[0120] The above description of the present invention is for illustrative purposes, and those having ordinary knowledge in the technical field to which the present invention pertains will understand that it can be easily transformed into other specific forms without changing the technical idea and essential features of the present invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. For example, each component described as a single form may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

[0121] The scope of the present invention is indicated by the claims described below rather than the above detailed description, and all changes or modified forms derived from the meaning and scope of the claims and the equivalent concept thereof should be construed as being included in the scope of the present invention.

Industrial Applicability

[0122] The present invention can be used for a patch-level severity determination method, a slide-level severity determination method, and a computing system that performs these methods.

Claims

1. A slide-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a pathological slide image into a predetermined unit size is input, comprising: determining a valid grade for each of a plurality of determination target partial images obtained by dividing a predetermined determination target pathological slide image into the unit size, based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input; determining a slide-level severity grade for the entire determination target pathological slide image, based on the valid grades for each of the plurality of determination target partial images constituting the determination target pathological slide image; including: the determination result for the partial image output by the first deep learning model is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system; the step of determining a valid grade for the determination target partial image based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input, includes determining a valid grade for the determination target partial image, based on the likelihood numerical value for each grade on the severity grade system of the determination target partial image and the grade score for each grade on the severity grade system. A slide-level severity determination method.

2. The step of determining a valid grade for the determination target partial image based on the likelihood numerical value for each grade on the severity grade system output by the first deep learning model to which the determination target partial image is input and the grade score for each grade on the severity grade system, Calculating a weighted average of the grade scores for each grade on the severity grade system, using the likelihood numerical values for each grade on the severity grade system as weights, and determining an effective grade for the pathological image to be determined; Calculating a weighted average of the grade scores for each of a plurality of upper grades with high likelihood numerical values among the grades on the severity grade system, using the likelihood numerical values of each of the plurality of upper grades as weights, and determining an effective grade for the pathological image to be determined, or Determining the grade score of the uppermost grade with the highest likelihood numerical value among the grades on the severity grade system as the effective grade for the pathological image to be determined The slide-level severity determination method according to claim 1, comprising the above steps.

3. A slide-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input partial image when each partial image obtained by dividing a pathological slide image into a predetermined unit size is input, Based on the determination result for the determination target partial image output by the first deep learning model with the determination target partial image input, determining an effective grade for the determination target partial image for each of the plurality of determination target partial images obtained by dividing a predetermined pathological slide image to be determined into the unit size; Based on the effective grades for each of the plurality of determination target partial images constituting the pathological slide image to be determined, determining a slide-level severity grade for the entire pathological slide image to be determined; including The determination result for the partial image output by the first deep learning model includes the determination result for each pixel constituting the partial image, and the determination result for each pixel is the likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system. Based on the determination result of the determination target partial image output by the first deep learning model to which the determination target partial image is input, the step of determining the effective grade for the determination target partial image is as follows: For each determination target pixel constituting the determination target partial image, Based on the possibility value for each grade of the determination target pixel on the severity grade system and the grade score for each grade on the severity grade system, the step of determining the effective grade for the determination target pixel, and Based on the effective grades of each determination target pixel constituting the determination target partial image, the step of determining the effective grade for the determination target partial image, including a severity determination method at the slide level.

4. Based on the possibility value for each grade of the determination target pixel on the severity grade system output by the first deep learning model and the grade score for each grade on the severity grade system, the step of determining the effective grade for the determination target pixel is as follows: Calculating a weighted average of the grade scores for each grade on the severity grade system, with the possibility value for each grade of the determination target pixel on the severity grade system as the weight, to determine the effective grade of the determination target pixel, Calculating a weighted average of the grade scores for a plurality of upper grades with high possibility values, with the possibility value of each of the plurality of upper grades as the weight, among the grades of the determination target pixel on the severity grade system, to determine the effective grade of the determination target pixel, or Determining the grade score of the topmost grade with the highest possibility value among the grades of the determination target pixel on the severity grade system as the effective grade of the determination target pixel Including the severity determination method at the slide level according to claim 3.

5. Based on the effective grades of each determination target pixel constituting the determination target partial image, the step of determining the effective grade for the determination target partial image is as follows: The method for determining the severity at the slide level according to claim 3, comprising the step of calculating a representative value of the effective grades of each determination target pixel constituting the determination target partial image and determining the effective grade for the determination target pathological image.

6. The step of determining the severity grade at the slide level for the entire determination target pathological slide image based on the effective grades for each of the plurality of determination target partial images constituting the determination target pathological slide image is: The method for determining the severity at the slide level according to claim 1 or claim 3, comprising the step of calculating a representative value of the effective grades of each determination target partial image constituting the determination target pathological slide image and determining the severity grade at the slide level for the entire determination target pathological slide image.

7. The step of calculating a representative value of the effective grades of each determination target partial image constituting the determination target pathological slide image and determining the severity grade at the slide level for the entire determination target pathological slide image is: The method for determining the severity at the slide level according to claim 6, comprising the step of calculating any one of the maximum value, minimum value, average value, median value, and average value within the IQR (InterQuartile Range) of the effective grades of each determination target partial image constituting the determination target pathological slide image as the representative value.

8. When a data set including the effective grades for each of the plurality of partial images obtained by dividing the pathological slide image into the unit size is input, the computing system further includes a second deep learning model which is an artificial neural network pre-trained to output an output value for determining the severity grade for the pathological slide image. The step of determining the severity grade for the determination target pathological slide image based on the effective grades for each of the plurality of determination target partial images constituting the determination target pathological slide image is: The method for determining the severity at the slide level according to claim 1 or claim 3, comprising the step of determining the severity grade for the determination target pathological slide image based on the output value output by the second deep learning model when the data set including the effective grades for each of the plurality of determination target partial images constituting the determination target pathological slide image is input.

9. The second deep learning model is: An artificial neural network pre-trained by a second deep learning model learning method, wherein the second deep learning model learning method includes steps of obtaining a plurality of training pathological slide images, generating a data set corresponding to each of the plurality of training pathological slides for each of the training pathological slide images, inputting each of the data sets corresponding to the plurality of training pathological slide images into the second deep learning model to train the second deep learning model, The step of generating a data set corresponding to the training pathological slide image includes, for each of a plurality of sub-images for generating a data set obtained by dividing the training pathological slide image into the unit size, determining a validity grade for the sub-image for generating a data set based on a determination result output by the first deep learning model into which the sub-image for generating a data set is input, and generating a data set including the validity grade for each of the plurality of sub-images for generating a data set. The slide-level severity determination method according to claim 8.

10. A patch-level severity determination method performed in a computing system including a first deep learning model, which is an artificial neural network pre-trained to output a determination result for an input sub-image when a sub-image obtained by dividing a pathological slide image into a predetermined unit size is input, including steps of inputting a predetermined target sub-image into the first deep learning model and obtaining a determination result for the target sub-image output by the first deep learning model (the determination result for the target sub-image is a likelihood numerical value for each grade on a predetermined severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system), and determining a validity grade for the target sub-image based on the likelihood numerical value for each grade on the severity grade system output by the first deep learning model into which the target sub-image is input and the grade score for each grade on the severity grade system. A method for determining the severity of a patch level, including

11. A method for determining the severity of a patch level performed in a computing system including a first deep learning model which is an artificial neural network pre-trained to output a determination result for an input partial image when a partial image obtained by dividing a pathological slide image into a predetermined unit size is input, a step of inputting a predetermined determination target partial image into the first deep learning model and obtaining a determination result for the determination target partial image output by the first deep learning model (the determination result for the determination target partial image output by the first deep learning model includes a determination result for each pixel constituting the determination target partial image, and the determination result for each pixel is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system), and for each determination target pixel constituting the determination target partial image, a step of determining an effective grade for the determination target pixel based on the likelihood numerical value for each grade on the severity grade system of the determination target pixel and the grade score for each grade on the severity grade system, and a step of determining an effective grade for the determination target partial image based on the effective grades of the determination target pixels constituting the determination target partial image, and A method for determining the severity of a patch level, including

12. A computer program installed in a data processing device and recorded on a medium for performing the method according to any one of Claims 1, 3, 10, and 11.

13. A computer-readable recording medium on which a computer program for performing the method according to any one of Claims 1, 3, 10, and 11 is recorded.

14. A computing system, including a processor and a memory, wherein the memory is When a computer program and each partial image obtained by dividing a pathological slide image into a predetermined unit size are input, a first deep learning model, which is an artificial neural network pre-trained to output a determination result for the input partial image, is stored. The computer program is When performed by the processor, controls the computing system to perform a severity determination method at the slide level. The severity determination method at the slide level is For each of a plurality of determination target partial images obtained by dividing a predetermined determination target pathological slide image into the unit size, based on the determination result for the determination target partial image output by the first deep learning model into which the determination target partial image is input, determining an effective grade for the determination target partial image; Based on the effective grade for each of the plurality of determination target partial images constituting the determination target pathological slide image, determining a severity grade at the slide level for the entire determination target pathological slide image; including The determination result for the partial image output by the first deep learning model is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system. Based on the determination result for the determination target partial image output by the first deep learning model into which the determination target partial image is input, the step of determining an effective grade for the determination target partial image is A computing system including the step of determining an effective grade for the determination target pathological image based on the likelihood numerical value for each grade on the severity grade system of the determination target partial image and the grade score for each grade on the severity grade system.

15. Based on the likelihood numerical value for each grade on the severity grade system output by the first deep learning model into which the determination target partial image is input and the grade score for each grade on the severity grade system, the step of determining an effective grade for the determination target pathological image is Calculating a weighted average of the grade scores for each grade on the severity grade system, where the possible numerical values for each grade on the severity grade system are used as weights, and determining the effective grade for the pathological image to be determined; Calculating a weighted average of the grade scores for each of a plurality of upper grades with high possible numerical values among the grades on the severity grade system, where the possible numerical values for each of the plurality of upper grades are used as weights, and determining the effective grade for the pathological image to be determined, or Determining the grade score of the topmost grade with the highest possible numerical value among the grades on the severity grade system as the effective grade for the pathological image to be determined The computing system according to claim 14, comprising the above steps.

16. A computing system, Comprising a processor and a memory, The memory stores, A first deep learning model which is an artificial neural network pre-trained to output a determination result for an input partial image when a computer program and each partial image obtained by dividing a pathological slide image into a predetermined unit size are input; The computer program, When executed by the processor, controls the computing system to perform a severity determination method at the slide level. The severity determination method at the slide level includes: For each of a plurality of determination target partial images obtained by dividing a predetermined pathological slide image to be determined into the unit size, based on the determination result for the determination target partial image output by the first deep learning model to which the determination target partial image is input, determining an effective grade for the determination target partial image; Based on the effective grades for each of the plurality of determination target partial images constituting the pathological slide image to be determined, determining a severity grade at the slide level for the entire pathological slide image to be determined. The determination result for the partial image output by the first deep learning model includes the determination result for each pixel constituting the partial image. The determination result for each pixel is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease. A predetermined grade score is pre-assigned to each grade on the severity grade system. Based on the determination result for the partial image to be determined output by the first deep learning model to which the partial image to be determined is input, the step of determining the effective grade for the partial image to be determined is for each determination target pixel constituting the partial image to be determined determining an effective grade for the determination target pixel based on the likelihood numerical value for each grade of the determination target pixel on the severity grade system and the grade score for each grade on the severity grade system; and determining an effective grade for the partial image to be determined based on the effective grades of each determination target pixel constituting the partial image to be determined. A computing system including

17. The step of determining an effective grade for the determination target pixel based on the likelihood numerical value for each grade of the determination target pixel on the severity grade system and the grade score for each grade on the severity grade system is calculating a weighted average of the grade scores for each grade on the severity grade system, with the likelihood numerical value for each grade of the determination target pixel on the severity grade system as the weight, to determine the effective grade of the determination target pixel; calculating a weighted average of the grade scores for a plurality of upper grades with high likelihood numerical values among the grades of the determination target pixel on the severity grade system, with the likelihood numerical value of each of the plurality of upper grades as the weight, to determine the effective grade of the determination target pixel, or determining the grade score of the topmost grade with the highest likelihood numerical value among the grades of the determination target pixel on the severity grade system as the effective grade of the determination target pixel The computing system according to claim 16, comprising

18. The step of determining the effective grade for the target partial image based on the effective grades of the respective target pixels constituting the target partial image The computing system according to claim 16, comprising the step of calculating a representative value of the effective grades of the respective target pixels constituting the target partial image and determining the effective grade for the target pathological image

19. The step of determining the severity grade at the slide level for the entire target pathological slide image based on the effective grades for each of the plurality of target partial images constituting the target pathological slide image The computing system according to claim 14 or claim 16, comprising the step of calculating a representative value of the effective grades of the respective target partial images constituting the target pathological slide image and determining the severity grade at the slide level for the entire target pathological slide image

20. The step of calculating a representative value of the effective grades of the respective target partial images constituting the target pathological slide image and determining the severity grade at the slide level for the entire target pathological slide image The computing system according to claim 19, comprising the step of calculating any one of the maximum value, minimum value, average value, median value, and average value within the IQR (InterQuartile Range) of the effective grades of the respective target partial images constituting the target pathological slide image as the representative value

21. When a data set including the effective grades for each of the plurality of partial images obtained by dividing the pathological slide image into the unit size is input, the memory further stores a second deep learning model which is an artificial neural network pre-trained to output an output value for determining the severity grade for the pathological slide image, The step of determining the severity grade for the target pathological slide image based on the effective grades for each of the plurality of target partial images constituting the target pathological slide image Based on the output value output by the second deep learning model into which a data set including the effective grade for each of the plurality of determination target partial images constituting the determination target pathological slide image is input, determining a severity grade for the determination target pathological slide image, the computing system according to claim 14 or claim 16.

22. The second deep learning model is an artificial neural network pre-trained by a second deep learning model learning method, The second deep learning model learning method is including the steps of obtaining a plurality of learning pathological slide images, for each of the plurality of learning pathological slides, generating a data set corresponding to the learning pathological slide image, inputting each of the data sets corresponding to the plurality of learning pathological slide images into the second deep learning model to learn the second deep learning model, The step of generating a data set corresponding to the learning pathological slide image includes for each of the plurality of partial images for data set generation obtained by dividing the learning pathological slide image into the unit size, based on the determination result output by the first deep learning model into which the partial image for data set generation is input, determining an effective grade for the partial image for data set generation, generating a data set including the effective grade for each of the plurality of partial images for data set generation, the computing system according to claim 21.

23. A computing system, including a processor and a memory, The memory stores a first deep learning model which is an artificial neural network pre-trained to output a determination result for an input partial image when a computer program and each partial image obtained by dividing a pathological slide image into a predetermined unit size are input, The computer program when performed by the processor, controls the computing system to perform a severity determination method at the patch level, The severity determination method at the patch level is Input a predetermined partial image to be determined into the first deep learning model, and obtain a determination result for the partial image to be determined output by the first deep learning model (the determination result for the partial image to be determined is a likelihood numerical value for each grade on a predetermined severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system), and Based on the likelihood numerical value for each grade on the severity grade system output by the first deep learning model into which the partial image to be determined is input and the grade score for each grade on the severity grade system, determine an effective grade for the partial image to be determined. A computing system comprising.

24. A computing system, Including a processor and a memory, The memory is When a computer program and each partial image obtained by dividing a pathological slide image into a predetermined unit size are input, it stores a first deep learning model which is an artificial neural network pre-trained to output a determination result for the input partial image, The computer program is When performed by the processor, controls the computing system to perform a severity determination method at the patch level, The severity determination method at the patch level is Input a predetermined partial image to be determined into the first deep learning model, and obtain a determination result for the partial image to be determined output by the first deep learning model (the determination result for the partial image to be determined output by the first deep learning model includes a determination result for each pixel constituting the partial image to be determined, and the determination result for each pixel is a likelihood numerical value for each grade on a predetermined histological severity grade system for a predetermined disease, and a predetermined grade score is pre-assigned to each grade on the severity grade system), and For each determination target pixel constituting the partial image to be determined, Determining a valid grade for the pixel to be determined based on the probability value for each grade on the severity grade system of the pixel to be determined and the grade score for each grade on the severity grade system; A computing system including: determining a valid grade for the partial image to be determined based on the valid grades of the pixels to be determined that make up the partial image to be determined.

Images