Method and system for predicting prognosis using disease diagnosis results via neural network

Abstract

A prognosis prediction method and a system thereof using a diagnosis result of a disease via a neural network are disclosed. The prognosis prediction method includes a step of inputting a biological image to the system, a step of the system generating a disease region diagnosis result in which a disease manifestation region in the biological image is determined for the input biological image, and a step of the system determining first information corresponding to the size of all tissues on the biological image and second information corresponding to the size of the disease region on the biological image based on the diagnosis result, and generating prognosis prediction information based on the determination result, and the step of the system generating a disease region diagnosis result in which a disease manifestation region in the biological image is determined for the input biological image includes generating a disease region diagnosis result based on a determination result of whether or not a disease manifestation occurs for each of predetermined patches obtained by dividing the biological image into a predetermined size, or generating a disease region diagnosis result based on a result of segmenting a disease manifestation region in each patch.

Description

[Technical field]

[0001] The present invention relates to a method and system for predicting prognosis using a diagnosis result of a disease via a neural network, more particularly to a method and system for deriving predictive factors for predicting the prognosis of a disease using a diagnosis result of a disease diagnosis system capable of dividing an onset region of the disease via a neural network, and for predicting prognosis using the predictive factors. [Background technology]

[0002] One of the main tasks performed by pathology or pathology departments is to read the images of a patient's body and perform a diagnosis to determine the condition or symptoms of a particular disease. Such a diagnosis is a process that relies on the experience and knowledge of skilled medical personnel over many years.

[0003] Recently, with the development of machine learning, there have been many attempts to automate tasks such as image recognition or classification using computing systems. In particular, attempts have been made to automate diagnoses that were previously performed by skilled medical personnel using neural networks (e.g., deep learning methods using convolution neural networks (CNNs)), which are a type of machine learning.

[0004] In particular, diagnosis through deep learning using neural networks (e.g., CNN) does not simply automate the experience and knowledge of traditionally experienced medical professionals, but rather finds characteristic elements through its own learning and derives the desired answer, and in some cases can even find characteristic disease factors from images that even experienced medical professionals are unaware of.

[0005] In general, disease diagnosis through a neural network using a biological image uses fragments of a biological image, i.e., patches (also called tiles). That is, for each tile, a skilled medical professional annotates the state of a specific disease (e.g., the presence or absence of cancer), and a neural network is trained using a plurality of such annotated tiles as training data. In this case, a convolutional neural network can be used as the neural network.

[0006] On the other hand, for disease prognosis prediction, information on how large an area the disease manifestation site occupies in all tissues to be diagnosed, i.e., expression rate, can be used as an important factor. The progression of the disease can be predicted based on such expression rate, and appropriate treatment methods, whether or not surgery is performed, and drug selection, etc. can be performed, so the expression rate for prognosis prediction can be very important information.

[0007] However, in the past, pathologists would visually examine glass slides containing tissue samples under a microscope, mark the diseased areas they determined on the glass slide with a designated tool (such as a felt-tip pen), and then only derive a rough expression rate based on the marking information.

[0008] Such a conventional method for determining the expression ratio is shown in FIG.

[0009] FIG. 1 is a diagram for explaining a conventional method for determining the incidence rate of a disease by a pathologist.

[0010] Referring to FIG. 1, as shown in the upper part of FIG. 1, conventionally, at least one biological tissue is stained in a predetermined manner and then placed on a slide.

[0011] Then, as shown in the lower part of Figure 1, a pathologist will visually check the tissues arranged on such slides one by one through a microscope and mark the areas where the disease has appeared in each tissue. At this time, since the disease-affected areas must be marked while visually checking, it is practically impossible to mark in extremely minute area units (for example, in the case of prostate cancer, in units of a ground), and as shown in the lower part of Figure 1, only the minimum information that allows for an approximate expression rate to be determined will be marked, such as a method of marking the endpoints of one direction and the other direction of the disease-affected area.

[0012] An example of a pathologist directly marking a slide (glass slide) with a red writing implement is shown at the bottom of Figure 1. In addition, Figure 1 illustrates an example of a disease being prostate cancer, but an average expert in the technical field of the present invention can easily deduce that the scope of the present invention is not limited thereto and that the technical idea of ​​the present invention can be applied to various diseases.

[0013] In the case of the lower part of Figure 1, the information regarding the expression rate for the first tissue obtained from the pathology expert's judgment may be that the total length of the tissue is 8.5 mm, the length of the affected area determined based on the marked information is 7 mm, and the expressed area contains a mixture of Gleason scores of 3 and 4.

[0014] In addition, the information regarding the expression rate obtained from the pathology expert's assessment of the second tissue may be that the total length of the tissue is 8 mm, the length of the affected area determined based on the marked information is 6.5 mm, and the expressed area contains a mixture of Gleason scores of 3 and 4.

[0015] In addition, the information regarding the expression rate for a third tissue obtained from a pathology expert's assessment may be that the total length of the tissue is 9.6 mm, the length of the affected area determined based on the marked information is 1.5 mm, and the affected area only has a Gleason score of 3.

[0016] In this way, when obtaining information on the expression rate for prognostic prediction based on the judgment of a pathologist, it is inevitably influenced by the method of marking the disease onset area, the thickness of the writing implement, and the physical condition of the pathologist at the time of judgment, and there is probably a considerable amount of inaccuracy.

[0017] In addition, as shown in Figure 1, pathologists generally mark only the boundaries of the disease-affected area (e.g., both ends) and predict the incidence rate by assuming that the disease occurs in all areas within the boundaries. However, in reality, there are areas within the boundaries where the disease occurs and areas where it does not occur, making it difficult to determine the accurate incidence rate.

[0018] Furthermore, as explained in Figure 1, when the disease manifests but the stage of the disease (e.g., Gleason score for prostate cancer) is classified, even if the incidence rate for each stage of the disease can be important information (e.g., the incidence rate of only Gleason score 3 in all tissues, or the incidence rate of only Gleason score 4), there is a problem in that it is practically difficult to obtain such information.

[0019] Therefore, there is an urgent need for a method and system that can accurately obtain information for prognosis prediction using the diagnostic results obtained using a neural network and use this information to predict the prognosis of a disease. [Prior art documents] [Patent documents]

[0020] [Patent Document 1] Korean Patent Application Publication No. 10-2019-0084814 "Disease diagnosis system and method for segmentation using neural network and non-local block" Summary of the Invention [Problem to be solved by the invention]

[0021] The technical problem that the present invention aims to achieve is to provide a technical idea that enables accurate information for prognosis prediction to be obtained using diagnostic results obtained using a neural network, and that enables prediction of the prognosis of a disease to be performed using this information.

[0022] In particular, the object of the present invention is to provide a technical idea that makes it possible to obtain more accurate prognostic prediction information by using diagnostic results obtained by performing segmentation to divide areas where a disease has occurred even within a patch, rather than determining whether a disease has occurred on a patch-by-patch basis. [Means for solving the problem]

[0023] According to one aspect of the present invention, a prognosis prediction method realized by a system including a processor and a storage device that stores a neural network includes the steps of: receiving a biological image as input to the system; generating a disease region diagnosis result in which a disease region in the input biological image is determined by the system; and determining first information corresponding to the size of all tissue on the biological image and second information corresponding to the size of the disease region on the biological image based on the diagnosis result, and generating prognosis prediction information based on the determination result. The step of generating a disease region diagnosis result in which a disease region in the input biological image is determined by the system generates the disease region diagnosis result based on a determination result of whether or not a disease has been manifested for each of predetermined patches obtained by dividing the biological image into a predetermined size, or generates the disease region diagnosis result based on a result of segmenting a disease region in each of the patches.

[0024] The step in which the system generates an onset area diagnosis result for an input biological image in which an onset area where a disease is manifested in the biological image is determined includes a step in which, when the disease manifests and the state of the manifested disease is divided into a plurality of classes, the system generates an onset area diagnosis result for each class, and the step in which the system determines first information corresponding to the size of all tissue on the biological image and second information corresponding to the size of the onset area on the biological image based on the diagnosis result, and generates prognosis prediction information based on the determination result can be characterized in that the system determines second information corresponding to the size of the onset area on the biological image for each class based on the onset area diagnosis result.

[0025] The prognosis prediction information may include information regarding an expression ratio corresponding to the size of the affected area relative to the size of the entire tissue.

[0026] A system for prognosis prediction according to another aspect includes a processor, a program driven by the processor, and a memory storing a neural network, and the processor, by driving the program, inputs a biological image, generates a disease area diagnosis result in which an onset area in the biological image where a disease is manifested is determined for the biological image, determines first information corresponding to the size of all tissue on the biological image and second information corresponding to the size of the onset area on the biological image based on the diagnosis result, generates prognosis prediction information based on the determination result, and generates the onset area diagnosis result based on the determination result of whether or not a disease has manifested for each of specified patches obtained by dividing the biological image into a specified size, or generates the onset area diagnosis result for each of the patches based on the result of segmenting the area of ​​the patch where a disease is manifested.

[0027] The processor can be characterized in that, by driving the program, when the disease manifests and the state of the manifested disease is divided into multiple classes, it generates a diseased area diagnosis result for each class, and determines second information corresponding to the size of the diseased area on the biological image for each class based on the diseased area diagnosis result.

[0028] The above method can be realized by a program installed in a data processing device. Effect of the Invention

[0029] According to the technical concept of the present invention, prognosis prediction information that enables relatively accurate prognosis prediction can be obtained by using the diagnostic results obtained using a neural network, and by using this information to predict the prognosis of a disease, it is possible to make a prognosis prediction based on accurate and objective information.

[0030] In addition, by using diagnostic results that have been segmented to divide areas where a disease has manifested within a patch, rather than determining whether a disease has manifested on a patch-by-patch basis, it is possible to obtain more accurate prognostic prediction information. [Brief description of the drawings]

[0031] In order to more fully understand the drawings referred to in the detailed description of the invention, a brief description of each of the drawings is provided. [Figure 1] FIG. 1 is a diagram for explaining a conventional method for determining the incidence rate of a disease by a pathologist. [Diagram 2] FIG. 1 is a diagram showing a schematic system configuration for realizing a method for predicting prognosis using disease diagnosis results via a neural network according to the technical idea of ​​the present invention. [Diagram 3] FIG. 2 is a diagram for explaining a hardware configuration of a system according to an embodiment of the present invention. [Figure 4] FIG. 2 is a diagram for explaining a logical configuration of a system according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example in which a system according to an embodiment of the present invention is divided into a diagnostic system and a prognosis prediction system. [Figure 6] 13 is a diagram showing a diagnosis result of a disease region in a biological image to explain a prognosis prediction method using a disease diagnosis result via a neural network according to one embodiment of the present invention. FIG. [Figure 7] FIG. 2 is a diagram for explaining the overall structure of a patch-level segmentation neural network according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Since the present invention can be modified in various ways and can have various embodiments, a specific embodiment is shown in the drawings and described in detail. However, it is not intended to limit the present invention to the specific embodiment, and it should be understood that it includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. In the description of the present invention, if it is determined that a specific description of related publicly known technology makes the gist of the present invention unclear, the detailed description will be omitted.

[0033] Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another.

[0034] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless otherwise clearly meant in the context.

[0035] As used herein, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and are to be understood as not precluding the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0036] Also, in this specification, when a component "transmits" data to another component, it means that the component can transmit data directly to the other component, or can transmit data to the other component via at least one other component. Conversely, when a component "transmits" data to another component, it means that the component also transmits data to the other component without via the other component.

[0037] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, in which the same reference numerals indicate the same elements.

[0038] Fig. 2 is a diagram showing a schematic system configuration for realizing a method for predicting prognosis using a disease diagnosis result via a neural network according to the technical idea of ​​the present invention. Fig. 3 is a diagram for explaining a hardware configuration of a system according to an embodiment of the present invention. Fig. 4 is a diagram for explaining a logical configuration of a system according to an embodiment of the present invention. Fig. 5 is a diagram showing an example in which a system according to one embodiment of the present invention is divided into a diagnosis system and a prognosis prediction system.

[0039] 2 to 5, in order to realize the technical idea of ​​the present invention, a system 100 for predicting prognosis using a diagnosis result of a disease via a neural network can be provided.

[0040] The system 100 according to the technical idea of ​​the present invention can be installed in a predetermined server 10 and realize the technical idea of ​​the present invention. The server 10 means a data processing device having a computing capability for realizing the technical idea of ​​the present invention, and an average expert in the technical field of the present invention can easily deduce that not only a data processing device to which a client can be connected via a network, but also any device capable of performing a specific service, such as a personal computer, a mobile terminal, etc., can be defined as a server.

[0041] The server 10 may include a processor 11 and a storage device 12 (memory) as shown in Fig. 3. The processor 11 means a computing device capable of running a program 12-1 for realizing the technical idea of ​​the present invention, and the processor 11 can perform diagnosis using the program 12-1 and a neural network 12-2 defined by the technical idea of ​​the present invention.

[0042] The neural network 12-2 can determine the region of the biological image where a predetermined disease occurs. In one example, the neural network 12-2 may be trained to determine whether a disease occurs in a patch obtained by dividing the biological image into a predetermined size. Such a neural network 12-2 can be defined as a patch-level classification neural network that performs patch-level diagnosis as described below.

[0043] In some embodiments, neural network 12-2 may include not only a patch-level classification neural network, but also a neural network for classifying and identifying only disease-affected areas of each patch, i.e., a patch-level segmentation neural network, which may also perform patch-level segmentation to identify areas of a patch where disease is present.

[0044] Depending on the embodiment, the neural network 12-2 may further include a neural network that performs slide-level diagnosis. Depending on the embodiment, the configuration for performing slide-level diagnosis may be realized by various machine learning techniques, not just a neural network. According to the technical idea of ​​the present invention, the diagnosis engine for performing slide-level diagnosis uses the well-known XGBoost, but it goes without saying that a diagnosis engine using various types of machine learning techniques may be realized, and such a diagnosis engine may be stored in the storage device 12.

[0045] 3, the server 10 may include a processor 11 and a storage device 12. The processor 11 means a computing device capable of running a program 12-1 for realizing the technical idea of ​​the present invention, and the processor 11 can perform diagnosis using the program 12-1 and a neural network 12-2 defined by the technical idea of ​​the present invention.

[0046] The neural network 12-2 may include a neural network for patch-level diagnosis, as described below. The neural network for patch-level diagnosis can determine whether a disease exists in a patch, which is a part of a divided slide. The neural network for patch-level diagnosis can also perform segmentation to identify an area of ​​the patch where a disease occurs. Such a neural network is hereinafter referred to as a patch-level segmentation neural network.

[0047] The storage device 12 means a data storage means capable of storing the program 12-1, the neural network 12-2, and / or the diagnostic engine for performing slide-level diagnosis, and may be realized by a plurality of storage means depending on the embodiment. The storage device 12 may mean not only a main storage device included in the server 10, but also a temporary storage device or memory that may be included in the processor 11.

[0048] Although system 100 is shown in FIG. 2 or FIG. 3 as being realized by any one physical device, it will be readily apparent to one of ordinary skill in the art of the present invention that multiple physical devices may be organically combined to realize system 100 according to the technical concept of the present invention, as necessary.

[0049] As used herein, when the system 100 performs a diagnosis, it may refer to a series of processes in which a biological image representing biological tissue, i.e., a patch which is an entire slide or a portion of a slide, is input, and output data defined in this specification is output.

[0050] In one example, the system 100 can perform a two-phase diagnosis. The first phase can be a patch-level diagnosis process, in which the system 100 receives input for each patch of a slide and outputs the presence or absence of a disease in the patch and / or identifies the area where a disease has developed in the patch. Needless to say, this can be achieved by learning a neural network.

[0051] In the second phase, it is possible to output whether or not a disease has appeared in the slide based on the diagnosis result in the first phase. This process can use a neural network or a certain machine learning technique.

[0052] That is, even if some patches are judged to have a disease according to the diagnosis result for each patch, there may be a possibility that the disease is not judged to have occurred in the biological tissue corresponding to the entire slide including the patch. For example, if the patches judged to have a disease are scattered in the slide or the number of patches is small, or if the physical characteristics (e.g., position, size, density, etc.) of the patches judged to have a disease may have a significant meaning in judging whether or not the disease has occurred in the actual corresponding slide. Therefore, in the second phase, the presence or absence of the disease in the slide is judged based on the diagnosis result for each patch and the characteristics of the patch judged based on such diagnosis result (i.e., the patch diagnosed to have a disease), thereby making it possible to perform an effective and highly accurate diagnosis.

[0053] On the other hand, the system 100 may include a patch-level segmentation neural network for performing segmentation for each patch. This is because, when the presence or absence of a disease is determined for each patch based on the diagnosis result for each patch (i.e., classification for each patch), and a disease is determined to have occurred for each patch, there may be a problem that the disease is determined to have occurred for the entire patch even if there is an area within the patch where the disease is not present. Therefore, segmentation that can distinguish diseased areas within each patch is required so that the tissue parts diagnosed as diseased can be clearly understood, and for this purpose, a patch-level segmentation neural network can be realized.

[0054] An example of a technical idea for such segmentation is disclosed in Korean Patent Publication No. 10-2019-0084814, “Disease diagnosis system and method for performing segmentation using neural network and non-local blocks” (hereinafter, the “previous application”), and the prior application may be incorporated by reference into the present invention.

[0055] In any case, the neural network 12-2 determines the presence or absence of disease on a patch-by-patch basis and, depending on the embodiment, is able to identify only the diseased areas within each patch.

[0056] That is, the neural network 12-2 may simply determine the disease-affected area for each patch, and may further identify only the disease-affected area within a patch.

[0057] Furthermore, the neural network 12-2 may determine not only the presence or absence of a specific disease, but also condition information indicating the degree of progression of the specific disease (or the probability of corresponding to the degree of progression).

[0058] For example, when the technical idea of ​​the present invention is used for diagnosing prostate cancer, the status information output by the neural network may include a Gleason pattern or a Gleason score, which is an index showing the progression of prostate cancer. For example, the Gleason score has a value from 2 to 5, and the higher the number, the greater the degree of onset of prostate cancer. Therefore, the status information may mean the probability that the biological tissue corresponding to the patch to be diagnosed corresponds to a specific Gleason score value (e.g., 3, 4, or 5).

[0059] In other words, the neural network 12-2 can not only determine whether or not a disease is present for each patch, but also, if a disease is present, determine which of multiple classes (e.g., Gleason score 3, 4, or 5) the patch is classified into.

[0060] When the system 100 is implemented by being included in a given server 10, the system 100 may communicate with at least one client (e.g., 20, 20-1) that can be connected to the server 10. In such a case, the client (e.g., 20, 20-1) may transmit a biometric image to the system 100, and the system 100 may perform a diagnosis on the transmitted biometric image according to the technical idea of ​​the present invention. The system 100 may also transmit a diagnosis result to the client (e.g., 20, 20-1). The system 100 may also generate and output prognosis prediction information based on the diagnosis result of the biometric image.

[0061] To this end, the system 100 may include a diagnostic system and a prognostic system, as shown in FIG.

[0062] The diagnostic system may be configured to perform diagnosis by inputting a biological image as described above. Performing a diagnosis may mean outputting information that can identify an area where a disease has appeared in a biological image, that is, a diseased area diagnosis result. The diseased area diagnosis result is sufficient as long as it can identify an area where a disease has appeared in a biological image. For example, image information itself that displays an area where a disease has appeared in a biological image so as to be separated from other areas may be the diseased area diagnosis result, or information (coordinates) regarding pixels included in the area where a disease has appeared may be the diseased area diagnosis result. Alternatively, identification information of a patch where a disease has appeared may be the diseased area diagnosis result. There may be various embodiments for realizing the diseased area diagnosis result.

[0063] The prognosis prediction system can generate information used for prognosis prediction. To this end, the prognosis prediction system can receive a disease region diagnosis result from the diagnosis system.

[0064] Then, the prognosis prediction system can determine the first information corresponding to the size of the whole tissue on the biological image, and the second information corresponding to the size of the diseased area on the biological image based on the diagnosis result. Also, the prognosis prediction system can generate prognosis prediction information based on the determination result. As described above, the prognosis prediction information may include information on the expression ratio, but is not limited thereto, and any information that can be derived from the diseased area diagnosis result and has significance for prognosis prediction can be included in the prognosis prediction information.

[0065] The first information may indicate information regarding the size of the tissue, and the first information may be defined in various ways, such as length, area, number of pixels, etc.

[0066] To determine the first information, the prognosis prediction system can classify the stained color of the tissue and the background in a predetermined manner to calculate the first information corresponding to the size of the tissue.

[0067] The second piece of information may be information indicating the size of the area of ​​tissue that is actually diseased.

[0068] In addition, when the diagnostic system is realized to classify the disease into one of a plurality of classes according to the state of the disease when the disease is manifested, the second information indicating the size of the diseased area for each class may be calculated by the prognosis prediction system. For example, in the case of prostate cancer, the second information corresponding to Gleason score 3, the second information corresponding to Gleason score 4, and the second information corresponding to Gleason score 5 may be determined individually.

[0069] On the other hand, as described above, the diagnostic system may simply diagnose a disease on a patch-by-patch basis, and in such a case, for a patch determined to have a disease present, the entire patch may be treated as having the corresponding disease (or a corresponding class of diseases) present.

[0070] Even in such a case, it may be possible to calculate prognostic information that is far more accurate than in the conventional case where the second information is determined by a pathology expert, as described above with reference to FIG.

[0071] In some embodiments, the diagnostic system can segment, for each patch, the diseased areas of the patch, such that only the diseased areas within any patch can be identified, making the second information more accurate and, when the second information is determined based on the results, more accurate prognostic information can be calculated.

[0072] The division into the diagnostic system and the prognosis prediction system as shown in Fig. 5 above is a division of their logical functions, and does not necessarily mean that the system 100 according to the technical idea of ​​the present invention is realized in a physically divided configuration. In other words, the configurations disclosed in Figs. 3 and 4 above can realize both the diagnostic system and the prognosis prediction system.

[0073] This technical idea will be described with reference to FIG.

[0074] FIG. 6 is a diagram showing a diagnosis result of an onset region in a biological image, illustrating a method for predicting a prognosis using a diagnosis result of a disease via a neural network according to an embodiment of the present invention.

[0075] FIG. 6 shows an example in which the tissue shown in FIG. 1 is diagnosed by the diagnostic system according to the technical concept of the present invention, and the affected region diagnosis result is displayed on a living body image.

[0076] The areas displayed in yellow in Figure 6 are areas that have been determined to be a predetermined first class of disease state (e.g., Gleason score 3), and the areas displayed in brown are areas that have been determined to be a predetermined second class of disease state (e.g., Gleason score 4).

[0077] 6 and 1, as shown in FIG. 6, unlike a pathologist marking the boundary of a diseased area and treating all tissue within the boundary as diseased, according to the technical idea of ​​the present invention, it is possible to distinguish and determine, even within the boundary, areas into those that are actually diseased and those that are not, down to the pixel level or close to that level of detail.

[0078] Therefore, compared with the conventional method in which only a rough expression rate was estimated, the technical idea of ​​the present invention makes it possible to obtain relatively highly accurate information regarding the size of the disease-affected area.

[0079] 6, it may be possible to calculate not only the disease manifestation area but also the disease state, i.e., accurate area for each class. Therefore, the system 100 can set various indices for prognosis prediction, such as the ratio of the disease manifestation area to the whole tissue, the ratio of the first class (e.g., Gleason score 3) manifestation area to the disease manifestation area, the ratio of the second class manifestation area to the disease manifestation area (e.g., Gleason score 4), the ratio of the second class manifestation area to the first class manifestation area, and so on.

[0080] Being able to utilize such a variety of prognostic prediction information may mean that a patient's current condition can be classified in more detail, enabling treatment and prescriptions to be made accordingly.

[0081] On the other hand, the neural network 12-2 can diagnose diseases at the patch level and determine the disease onset area on the biological image as described above. Of course, a process of training the neural network to perform such a diagnosis may be performed in advance. Also, as described above, a certain neural network can be used for slide-level diagnosis.

[0082] Therefore, the system 100 may be a system that performs diagnosis by receiving from the outside a neural network trained according to the technical idea of ​​the present invention and a program for performing diagnosis using the neural network, or may be a system that performs neural network training. Also, the system 100 may be realized by a dedicated device manufactured for realizing the technical idea of ​​the present invention, rather than a general-purpose data processing device, and in such a case, may further include a means for scanning a biological image, etc.

[0083] A system for realizing such a technical idea can have a logical configuration as shown in FIG.

[0084] 4, the system includes a control module 110 and a neural network module 120 in which a neural network and / or a slide diagnostic engine is stored. The system 100 may further include a pre-processing module 130. Depending on the embodiment of the present invention, some of the above-mentioned components may not necessarily be essential components for realizing the present invention, and it goes without saying that the system 100 may include more components depending on the embodiment. For example, the system 100 may further include a communication module (not shown) for communicating with a client (e.g., 20, 20-1).

[0085] The system 100 refers to a logical configuration having hardware resources and / or software necessary to realize the technical idea of ​​the present invention, and does not necessarily refer to one physical component or one device. In other words, the system 100 refers to a logical combination of hardware and / or software provided to realize the technical idea of ​​the present invention, and may be realized as a set of logical configurations that are installed in devices separated from each other and perform their respective functions to realize the technical idea of ​​the present invention, as necessary. The system 100 may also refer to a set of configurations that are realized separately for each function or role to realize the technical idea of ​​the present invention. For example, each of the control module 110, the neural network module 120, and / or the pre-processing module 130 may be located in different physical devices or may be located in the same physical device. In addition, depending on the embodiment, the combination of software and / or hardware constituting each of the control module 110, the neural network module 120, and / or the pre-processing module 130 may also be located in different physical devices, and the configurations located in the different physical devices may be organically combined with each other to realize each module.

[0086] In addition, in this specification, a module may refer to a functional and structural combination of hardware for carrying out the technical idea of ​​the present invention and software for driving the hardware. For example, a module refers to a logical unit of a certain code and a hardware resource for carrying out the certain code, and does not necessarily refer to physically connected code or one type of hardware, which can be easily inferred by an average expert in the technical field of the present invention.

[0087] The control module 110 may control other components (eg, the neural network module 120 and / or the pre-processing module 130) included in the system 100 to realize the technical concept of the present invention.

[0088] The control module 110 can also receive a biological image and perform diagnosis according to the technical idea of ​​the present invention using the stored neural network module 120 and / or slide diagnosis engine for the input biological image. It can also output a disease region diagnosis result. The control module 110 can also generate prognosis prediction information.

[0089] The control module 110 may receive input data, i.e., input for each patch, to a patch-level neural network, i.e., a trained neural network, stored in the neural network module 120. The control module 110 may perform an operation defined by the neural network to output output data, i.e., a feature value corresponding to a disease occurrence probability corresponding to the patch. In some embodiments, the control module 110 may identify an area of ​​the patch where a disease has occurred. In some embodiments, the control module 110 may output whether a patch has a disease occurrence or not depending on whether a feature value is a predetermined threshold value for slide-level diagnosis as described below.

[0090] The neural network module 120 may include a patch diagnostic engine that performs patch-level diagnostics, and a slide diagnostic engine that performs slide-level diagnostics.

[0091] The patch level diagnosis engine can be realized through a neural network based on deep learning according to the technical idea of ​​the present invention as described above. The slide diagnosis engine may use a neural network based on deep learning, or may use a predetermined machine learning engine (e.g., XGBoost) instead of a neural network.

[0092] A neural network may refer to a collection of information representing a set of design considerations that define the neural network. As used herein, a neural network may be a convolutional neural network.

[0093] As is well known, a convolutional neural network may include an input layer, multiple hidden layers, and an output layer. Each of the multiple hidden layers may include a convolutional layer and a pooling layer (or a subsampling layer). The convolutional neural network may also be designed to include a normalization layer, such as a batch normalization (BN) layer.

[0094] A convolutional neural network can be defined by functions, filters, strides, weighting coefficients, etc. for defining each such layer. Also, the output layer can be defined as a fully connected FeedForward layer.

[0095] The design items for each layer constituting a convolutional neural network are widely known. For example, the number of layers included in the multiple layers, the convolution function for defining the multiple layers, the pooling function, and the activation function may be publicly known functions, or functions defined separately to realize the technical idea of ​​the present invention may be used.

[0096] An example of the convolution function is discrete convolution sum. An example of the pooling function is max pooling, average pooling, etc. An example of the activation function may be sigmoid, tangent hyperbolic (tanh), ReLU (rectified linear unit), Swish, ELU (exponential linear unit), etc.

[0097] As mentioned above, the neural network that performs the patch-level diagnosis may be a patch-level segmentation neural network that not only determines whether a disease is present in a patch, but also performs segmentation to identify the disease-affected areas of the patch.

[0098] The patch level segmentation neural network according to an embodiment of the present invention can be realized in a form in which a separate architecture for segmentation is combined with a neural network ("patch level classification neural network" described below) that performs classification to determine whether a disease exists in a patch. The structure of such a patch level segmentation neural network is shown in Figure 7. The patch level segmentation neural network shown in Figure 7 has been disclosed in detail in a previous application, and will be briefly described as follows.

[0099] As shown in FIG. 7, a patch level segmentation neural network 400 according to one embodiment of the present invention may include a patch level classification neural network 200 and a patch level segmentation architecture 500.

[0100] The patch-level classification neural network 200 can input a patch, which is a part of a divided slide, to an input layer and output a patch-level classification result (for example, a score as shown in FIG. 7) regarding whether or not a disease exists in the patch. This is called classification, and in the classification process, the patch-level classification neural network 200 can generate features for the input (i.e., the patch) as an intermediate product in some hidden layers included therein. In particular, when a matrix of two or more dimensions such as an image is received as an input, the generated features are in the form of a two-dimensional matrix, so the term feature map may be used. Meanwhile, hereinafter, a layer that generates a feature map among the hidden layers included in the patch-level classification neural network 200 is called a feature map extraction layer.

[0101] Meanwhile, the patch-level segmentation architecture 500 can receive as input feature maps (e.g., f1, f2, and f3 shown in FIG. 4 of the previous application) generated in each of two or more feature map extraction layers among the hidden layers included in the patch-level classification neural network 200, and identify and output areas of the patch where a disease exists.

[0102] FIG. 7 shows an example in which the patch-level classification neural network 200 generates three feature maps (low-level feature map f1, mid-level feature map f2, and high-level feature map f3) during classification, but it goes without saying that more or less feature maps can be generated depending on the embodiment.

[0103] Meanwhile, according to an embodiment of the present invention, the patch-level classification neural network 200 for performing patch-level classification can be, but is not limited to, the well-known densenet. Various other neural networks can be used, and in any case, the patch-level classification neural network 200 can be defined to receive a specific patch as an input and output a feature value corresponding to the disease occurrence probability of the specific patch.

[0104] As described above, in the classification process performed by the patch-level classification neural network 200, the patch-level segmentation architecture 500 generates feature maps f1, f2, and f3 by the respective feature extraction layers, and can identify the region of the patch where the disease exists based on the feature maps. For this purpose, non-local correlation can be used, as disclosed in the previous application.

[0105] As described above, once the neural network module 120 has identified the disease-affected region, ie, once the disease-affected region diagnosis result has been generated, the control module 110 can generate prognosis prediction information.

[0106] The control module 110 may determine the first information corresponding to the size of the entire tissue on the biological image that is the subject of diagnosis, as described above. For example, the first information may be determined based on the number of pixels and / or the number of patches.

[0107] Also, second information corresponding to the size of the diseased area on the biological image can be determined based on the diagnosis result, for example, the second information can be determined based on the number of pixels and / or patches determined to be diseased.

[0108] The control module 110 can then generate prognosis prediction information based on the first information and the second information.

[0109] In addition, when the neural network module 120 classifies diseases into a plurality of classes by determining the presence or absence of the disease on a pixel-by-pixel basis and / or on a patch-by-patch basis, the control module 110 can determine second information corresponding to the size of the disease-affected area on the biological image for each class. In such a case, it goes without saying that the prognosis prediction information can include an expression ratio corresponding to the size of the disease-affected area for each class relative to the size of the entire tissue.

[0110] The pre-processing module 130 can perform pre-processing of the biometric image as required before performing a diagnosis using a neural network. For example, pre-processing of the biometric image may include patching the biometric image into patches of a predefined size and calculating the gray value of the pixels for each patch as described above. It can be easily deduced by an average person skilled in the art that suitable image processing can also be performed in a manner suitable for a neural network, if necessary.

[0111] However, depending on the embodiment, system 100 may include a processor and memory that stores programs executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include high-speed random access memory and may include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory by the processor and other components may be controlled by a memory controller.

[0112] Meanwhile, the method according to the embodiment of the present invention may be realized in the form of computer readable program instructions and stored in a computer readable recording medium, and the control program and the target program according to the embodiment of the present invention may also be stored in a computer readable recording medium. The computer readable recording medium includes any kind of recording device in which data readable by a computing system is stored.

[0113] The program instructions contained on the recording media may be those specially designed and constructed for the purposes of the present invention, or they may be of the type well known and available to those having skill in the software arts.

[0114] Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Also, computer readable recording media can be distributed among computing systems connected through a network to store and execute computer readable code in a distributed manner.

[0115] Examples of program instructions include not only machine code produced by a compiler, but also higher level language codes that may be executed by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0116] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0117] The above description of the present invention is for illustrative purposes only, and a person having ordinary skill in the art to which the present invention pertains can easily understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described in a single form can be implemented in a distributed manner, and similarly, components described as being distributed can be implemented in a combined form.

[0118] The scope of the present invention is indicated by the claims set forth below, rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims, as well as their equivalent concepts, should be construed as being included within the scope of the present invention. [Industrial Applicability]

[0119] INDUSTRIAL APPLICABILITY The present invention can be used in a method and system for predicting prognosis using a disease diagnosis result via a neural network.

Claims

1. A method for predicting prognosis, the method being implemented by a system including a processor and a storage device storing a neural network, the method comprising: receiving a biometric image from said system; The system generates an onset region diagnosis result in which an onset region in the input biological image where a disease is manifested is determined; The system determines first information corresponding to a size of all tissues on the biological image and second information corresponding to a size of an affected area on the biological image based on the diagnosis result, and generates prognosis prediction information based on the determination result; Including, The step of generating a disease region diagnosis result in which a disease region in the input biological image is determined in the biological image by the system includes: Generate the onset region diagnosis result based on a result of determining whether or not a disease has appeared for each of the predetermined patches obtained by dividing the biological image into a predetermined size, or generate the onset region diagnosis result for each of the patches based on a result of segmenting a region of the patch in which a disease has appeared; When the disease is manifested, if the manifested disease state is classified into a plurality of classes, the system includes a step of generating a disease domain diagnosis result for each class; The system determines first information corresponding to a size of the entire tissue on the biological image and second information corresponding to a size of an affected area on the biological image based on the diagnosis result, and generates prognosis prediction information based on the determination result, A method for predicting prognosis using disease diagnosis results via a neural network, characterized in that the system determines second information corresponding to the size of the affected area on the biological image for each class based on the affected area diagnosis result.

2. The prognostic prediction information is A method for predicting prognosis using a disease diagnosis result via a neural network as described in claim 1, which includes information on the expression ratio corresponding to the size of the affected area relative to the size of the entire tissue.

3. A computer program recorded on a medium for being installed in a data processing device and for carrying out the method according to any one of claims 1 or 2.

4. A system for prognostic prediction comprising a processor and a memory device storing a neural network, A processor; a memory for storing a program and a neural network operated by the processor; Including, The processor runs the program, A living body image is input, and an onset region diagnosis result is generated in which an onset region in the living body image where a disease is manifested is determined for the living body image, first information corresponding to the size of all tissue on the living body image and second information corresponding to the size of the onset region on the living body image based on the diagnosis result are determined, and prognosis prediction information is generated based on the determination result. Generate the onset region diagnosis result based on a result of determining whether or not a disease has appeared for each of the predetermined patches obtained by dividing the biological image into a predetermined size, or generate the onset region diagnosis result for each of the patches based on a result of segmenting a region of the patch in which a disease has appeared; When the disease is manifested, if the manifested disease state is classified into a plurality of classes, a diagnosis result of the disease region is generated for each class; A system for predicting prognosis using disease diagnosis results via a neural network, characterized in that second information corresponding to the size of the affected area on the biological image for each class is determined based on the affected area diagnosis result.

Images