Method and apparatus for determining cancer metastasis tissue on basis of linked multi-neural network model

Abstract

In an embodiment, a cancer metastasis tissue determination apparatus may comprise the operations of: acquiring a pathological image including a tissue subject to determination for cancer; generating a plurality of patches by dividing the pathological image into patches of a predetermined size; inputting the plurality of patches to a first neural network model trained to distinguish whether tumor tissue is included in a pathological image, and determining a probability that each of the plurality of patches includes tumor tissue; on the basis of the probability, selecting a patch to be observed from among the plurality of patches; and inputting the selected patch to a second neural network model trained to determine, on the basis of a plurality of patches, whether tumor tissue is included, and determining whether tumor tissue is included in the pathological image or a location of the tumor tissue.

Description

Method and device for identifying cancer metastasis tissue based on a linked multi-neural network model

[0001] The present invention relates to a technology for determining cancer metastatic tissue by linking two or more neural network models having different characteristics.

[0002]

[0003] Cancer diagnosis and treatment are among the most critical challenges in the medical field. While numerous studies and technologies have contributed to cancer diagnosis and treatment, they remain a major challenge for humanity.

[0004] Traditional cancer tissue diagnosis techniques have relied primarily on histopathological examinations, relying on the subjective judgment and experience of experts. Recently, advances in deep learning and machine learning have led to the development of cancer diagnosis methods utilizing computer vision and pattern recognition. The use of neural network models, driven by advances in machine learning, is opening up new possibilities in the fields of medical imaging and diagnosis.

[0005] In particular, computer vision and pattern recognition using deep learning techniques are attracting significant attention in the analysis of pathological tissue images. These technologies can be used to extract features from high-resolution tissue images, detect lesions, and classify diseases.

[0006] Meanwhile, most studies utilizing neural network models have focused primarily on attempts to diagnose cancer tissue using a single neural network, but there are several limitations to diagnosing cancer tissue using a single neural network.

[0007] First, the complexity and diversity of tissue images can make it difficult for a single neural network to accurately distinguish or classify all types of cancer tissue. In particular, the diverse shapes and characteristics of cancer tissues can limit the generalization ability of a single neural network model.

[0008] Second, extracting and analyzing diverse features considering the diverse characteristics of cancer tissue is a complex task. Sufficiently learning these complex features with a single neural network can require extremely large datasets and complex architectures. This increases the computational and resource requirements for model training and execution, potentially reducing practicality.

[0009] Finally, the potential problem of overfitting when using a single neural network to diagnose cancer tissue must also be considered. In particular, when the training dataset is small or imbalanced, the model may become overly reliant on specific features or patterns, reducing its generalization ability.

[0010] Accordingly, this paper proposes a technique to overcome the limitations of cancer tissue diagnosis using such a single neural network.

[0011]

[0012] The problem to be solved by the present invention is to provide a more accurate and reliable diagnostic technology by linking two or more neural networks trained in different ways and using them for cancer diagnosis.

[0013] Meanwhile, the technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below.

[0014]

[0015] In one embodiment, a method performed by a cancer metastasis tissue determination device operated by a processor may include: acquiring a pathological image including a tissue to be determined as cancer; dividing the pathological image into a preset size to generate a plurality of patches; inputting the plurality of patches into a first neural network model trained to distinguish whether tumor tissue is included in the pathological image to determine a probability that each of the plurality of patches includes tumor tissue; selecting a patch to be observed from among the plurality of patches based on the probability; and inputting the selected patch into a second neural network model trained to determine whether tumor tissue is included based on the plurality of patches to determine whether tumor tissue is included or a location of tumor tissue in the pathological image.

[0016] In addition, the first neural network model is configured as a neural network with a MIL (Multiple Instance Learning) structure, and is trained based on labeled learning data with a single BAG class that only specifies whether an instance corresponding to tumor tissue is included in a pathological image, and can output the probability that the input data includes the instance.

[0017] In addition, the operation of selecting the patches to be judged may include an operation of classifying patches judged to have a probability higher than a predetermined threshold; and an operation of arranging the classified patches in order of increasing probability.

[0018] In addition, the second neural network model is configured as a neural network with a RNN (Recurrent Neural Networks) structure and is trained to determine whether tumor tissue is included by identifying changes in the order of input patches and the spatial relationship of the input patches, so that when the classified patches are input in the order in which they are arranged, the probability that the classified patches include tumor tissue can be output.

[0019] In addition, the second neural network model is configured as a neural network with an autoencoder structure including an encoder and a decoder, and is trained to encode and decode input data based on learning data of a pathological image containing only normal tissue to restore the input data, and when encoding and decoding the selected patch is performed by inputting the selected patch, a patch having a reconstruction error greater than a preset value can be determined as the location of tumor tissue in the pathological image.

[0020] In addition, the second neural network model is configured as a neural network having an autoencoder structure including two encoders and one decoder trained based on different training data, and is trained to encode and decode input data based on training data of a pathological image containing only normal tissue to restore the input data, and when encoding and decoding are performed by inputting the selected patch, if the standard deviation of the difference value of the reconstruction error by each of the two encoders is greater than a preset value, it can be determined that the pathological image contains tumor tissue.

[0021] In addition, the second neural network model is configured as a neural network having an autoencoder structure including two encoders and one decoder trained based on different training data, wherein the first encoder of the two encoders is trained to encode and decode input data based on training data of a pathological image including only normal tissue to restore the input data, and the second encoder of the two encoders is trained to encode and decode input data based on training data of a pathological image including abnormal tissue to restore the input data, and when the selected patch is input and encoding and decoding are performed, it is possible to determine whether the pathological image includes tumor tissue based on a comparison of the size of the reconstruction error by each of the two encoders.

[0022] In addition, the operation of generating the plurality of patches may include an operation of determining a border of a tissue included in the pathological image; an operation of removing data of an outer region of the border of the tissue; and an operation of generating a patch by dividing an inner region of the border of the tissue into a preset size.

[0023] In addition, the operation of generating the plurality of patches may include, after the operation of generating the patches, an operation of making the tissue area included in the patches symmetrical left-right or up-down within the patches when the tissue area included in the patches is 30% or more and 50% or less.

[0024] Additionally, the operation of generating the plurality of patches may include, after the operation of generating the patch, an operation of copying the tissue area included in the patch and pasting it into a blank area if the tissue area included in the patch is less than 30%.

[0025] In one embodiment, a device includes a memory including a command; and a processor that performs a predetermined operation based on the command, wherein the operation of the processor may include: an operation of acquiring a pathological image including a tissue to be determined as cancer; an operation of dividing the pathological image into a preset size to generate a plurality of patches; an operation of inputting the plurality of patches into a first neural network model trained to distinguish whether tumor tissue is included in the pathological image and determining a probability that each of the plurality of patches includes tumor tissue; an operation of selecting a patch to be observed from among the plurality of patches based on the probability; and an operation of inputting the selected patch into a second neural network model trained to determine whether tumor tissue is included based on the plurality of patches and determining whether tumor tissue is included or a location of tumor tissue in the pathological image.

[0026]

[0027] The present invention can more effectively handle the diversity and complexity of cancer tissues by linking two or more neural networks learned in different ways and using them for cancer diagnosis, and can alleviate the overfitting problem of neural network models and improve generalization ability.

[0028] In addition, since the neural network model of the present invention can be trained in different ways for the same training data set, the computation and resources required for training and executing the model can be reduced, thereby improving practicality, and since neural network models with different special features are linked, a diagnosis can be made by considering various characteristics of cancer tissue.

[0029] Therefore, the present invention can contribute significantly to the advancement of medical technology by achieving practical application of deep learning and machine learning technologies in the field of histopathological examination and, at the same time, significantly improving the accuracy and efficiency of cancer tissue diagnosis.

[0030] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other technical effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0031]

[0032] Figure 1 is a schematic diagram of a cancer metastasis tissue determination device according to one embodiment.

[0033] Figure 2 is a flowchart showing the steps of an operation performed by a cancer metastasis tissue determination device according to one embodiment.

[0034] FIG. 3 is an example diagram of an operation of removing data other than an organization through recognition of the organization's border according to one embodiment.

[0035] FIG. 4 is an example of an operation for filtering and removing the border recognition of outliers such as bubbles that have occurred in an organization in the embodiment of FIG. 3.

[0036] FIG. 5 is an example diagram of an operation of generating multiple patches by dividing an area corresponding to a tissue in a pathological image into preset sizes according to one embodiment.

[0037] Figure 6 is an example of modifying a patch when the tissue area included in the patch is 30% or more and 50% or less.

[0038] Figure 7 is an example of modifying a patch when the tissue area included in the patch is less than 30%.

[0039] FIG. 8 is an example diagram of a first neural network model that determines the probability that a specific instance is included in a patch according to one embodiment.

[0040] FIG. 9 is an example diagram showing a second neural network model configured and operating in the form of a recurrent neural network (RNN) according to one embodiment.

[0041] FIG. 10 and FIG. 11 are exemplary diagrams showing a second neural network model configured and operated in the form of an autoencoder with one encoder and one decoder according to one embodiment.

[0042] FIGS. 12 to 14 are exemplary diagrams showing a second neural network model configured and operating in the form of an autoencoder including two encoders and one decoder according to one embodiment.

[0043]

[0044] The purpose, technical configuration, and resulting operational effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. Reference will now be made to the accompanying drawings, which will further describe embodiments of the present invention.

[0045] The embodiments disclosed herein should not be construed or used to limit the scope of the present invention. Those skilled in the art will readily appreciate that the descriptions herein, including the embodiments, have a wide range of applications. Therefore, any embodiments described in the detailed description of the present invention are intended to serve as illustrative examples to better illustrate the present invention and are not intended to limit the scope of the present invention to the embodiments.

[0046] The functional blocks depicted in the drawings and described below are merely examples of possible implementations. Other implementations may utilize other functional blocks without departing from the spirit and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are depicted as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0047] Additionally, the expression “including certain components” is an “open” expression, simply indicating the presence of those components, and should not be understood as excluding additional components.

[0048] Furthermore, when it is said that a component is “connected” or “connected” to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0049] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of the present invention are included.

[0050] FIG. 1 is a configuration diagram of a cancer metastasis tissue determination device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0051] Referring to FIG. 1, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0052] The memory (110) can store data acquired from an external device or data generated by the processor (120). The memory (110) can store commands that can perform operations of the processor (120). For example, the memory (110) can store pathological images of a specific tissue of a patient, as well as the first neural network model and the second neural network model described below.

[0053] The processor (120) is a computing device that controls overall operations. The processor (120) can execute instructions stored in the memory (110). The operation of the device (100) according to the embodiment of this document can be understood as an operation performed by the processor (120).

[0054] The input / output interface (130) may include a hardware interface or software interface for inputting or outputting information.

[0055] The communication interface (140) enables information to be transmitted and received via a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0056] The device (100) may be implemented in various forms of devices capable of performing operations through a processor (120) and transmitting and receiving information through a network. For example, the device may be implemented in the form of a server, a computer device, a portable communication device, a smart phone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0057] FIG. 2 is a flowchart of operations performed by a device (100) according to one embodiment. The operations of the device (100) according to the embodiment of FIG. 2 can be understood as operations performed by a processor (120).

[0058] Each step disclosed in FIG. 2 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and one step may be included and performed within another step. The order of each operation disclosed in FIG. 3 is merely an order arranged for ease of understanding, and this order is not limited to a chronological order, and the order may be changed and performed differently depending on the designer's choice.

[0059] Referring to FIG. 2, at step S1010, the device (100) may acquire a pathological image. For example, the device (100) may acquire the pathological image from an external device or a linked device (e.g., a database, a photographing device, etc.). For example, the pathological image may include a tissue that is a target for determining renal cancer, bladder cancer, or thyroid cancer, and the tissue may be stained with a predetermined dye to be distinguished from other objects in the image.

[0060] In step S1020, the device (100) can generate multiple patches by dividing the pathological image into preset sizes. Examples of generating multiple patches are as shown in the following Figures 3 to 7.

[0061] FIG. 3 is an example diagram of an operation of removing data other than an organization through recognition of the organization's border according to one embodiment.

[0062] Referring to FIG. 3, in step S1020, the device (100) can determine the border of a tissue included in a pathological image, remove data from an area outside the border of the tissue, and divide the inner area of ​​the border of the tissue into a preset size to generate a patch. For example, the device (100) can extract the border of the foreground, not the background, from a pathological image through the GrabCut algorithm, and can remove data from an area outside the border to make it null. Accordingly, the device (100) can generate a patch so that only an area corresponding to the tissue is included in the patch.

[0063] If pathological images were segmented directly to create patches without the process illustrated in Figure 3, areas outside the tissue would also be created as patch images, potentially resulting in unnecessary computations for neural network learning or utilization. Therefore, the embodiment of this document preprocesses pathological images through the process illustrated in Figure 3 to use only the data essential for determining the presence or absence of cancerous tissue, thereby reducing resource consumption for neural network utilization.

[0064] FIG. 4 is an example of an operation for filtering and removing the border recognition of outliers such as bubbles that have occurred in an organization in the embodiment of FIG. 3.

[0065] Referring to FIG. 4, when recognizing a border, an unnecessary area for examination may be recognized as a border due to bubbles generated during a tissue examination. In this case, the device (100) may additionally remove an area within the border of the tissue that does not include staining color information. For example, the device (100) may extract a border of the foreground, not the background, in a pathology image through the GrabCut algorithm, and may extract color information for the area within the border. At this time, the interior of the area corresponding to the tissue includes staining information for the tissue, but the interior of the area corresponding to the bubble does not include staining information for the tissue. Therefore, the device (100) may extract color information for the interior of the recognized border, recognize a border that does not include staining information as an outlier (e.g., a bubble), and remove data for the area corresponding to the outlier.

[0066] If a tissue image is segmented and patches are generated without the process illustrated in Figure 4, the region corresponding to the bubble may also be included in the patch image, resulting in unnecessary computations for neural network training or utilization. Therefore, the embodiment of this document preprocesses pathological images through the process illustrated in Figure 4 to use only the data essential for neural network judgment, thereby reducing resource consumption for neural network utilization.

[0067] FIG. 5 is an example diagram of an operation of generating multiple patches by dividing an area corresponding to a tissue in a pathological image into preset sizes according to one embodiment.

[0068] Referring to FIG. 5, the device (100) can assign windows of 512 x 512 pixel size (e.g., red square on the left side of FIG. 5) to regions recognized as tissues among pathological images, and generate patches (e.g., black dotted square on the right side of FIG. 5) including images within each window. At this time, a patch captured from a window located in an internal region of a tissue among the arranged windows completely includes the tissue region, but a patch captured from a window located in a border portion of the tissue among the windows may partially include an external region of the tissue (e.g., null region).

[0069] In this way, for patches that include less than 50% of the internal area of ​​the tissue among multiple patches, the information regarding the tissue area that is the actual target of judgment is insufficient, which may lead to errors in the neural network's judgment when utilized. For this reason, if the percentage of the internal area of ​​the tissue among patches is below a predetermined percentage, the following embodiments of FIG. 6 or FIG. 7 may be applied.

[0070] Figure 6 is an example of modifying a patch when the tissue area included in the patch is 30% or more and 50% or less.

[0071] Referring to FIG. 6, in the case of a patch that includes 30% or more and 50% or less of the internal area of ​​the organization among the patches generated by allocating a window, the area where the organization area included in the patch is located can be made left-right symmetrical or up-down symmetrical to enhance the organization area included in the patch.

[0072] For example, the device (100) can recognize the area where the tissue area is located by dividing the square of the patch into nine equal parts, and determine left-right symmetry or up-down symmetry in the direction in which the tissue area increases, thereby increasing the tissue area within the patch.

[0073] For example, in the case of the upper right of Fig. 6, when the square of the patch is divided into 9 equal parts, since most of the tissue is located on the lower side of the patch, the tissue area within the patch can be increased by determining the upper and lower symmetry.

[0074] For example, in the case of the lower right of Fig. 6, when the square of the patch is divided into 9 equal parts, since most of the tissue is located on the left side of the patch, the left-right symmetry can be determined to enhance the tissue area within the patch.

[0075] Figure 7 is an example of modifying a patch when the tissue area included in the patch is less than 30%.

[0076] Referring to FIG. 7, for a patch that includes less than 30% of the internal area of ​​an organization among the patches generated by allocating a window, the organization area within the patch can be increased by copying the organization area included within the patch and pasting it into an empty area.

[0077] For example, in the case of Fig. 7, when the square of the patch is divided into nine equal parts, the lower area where the tissue is located can be copied to another divided area to enhance the tissue area within the patch.

[0078] In step S1030, the device (100) inputs a plurality of patches preprocessed in step S1020 into a first neural network model trained to distinguish whether or not tumor tissue is included in a pathological image, thereby determining the probability that each of the plurality of patches includes tumor tissue.

[0079] The first neural network model of this document prioritizes only whether or not there is tumor tissue in the pathology image, and proceeds to the next step only when there is a high probability that the pathology image contains tumor tissue, and does not proceed to the next step when there is a low probability that the pathology image contains tumor tissue, thereby reducing the resource consumption of the neural network model.

[0080] FIG. 8 is an example diagram of a first neural network model that determines the probability that a specific instance is included in a patch according to one embodiment.

[0081] Referring to Figure 8, the first neural network model may be configured as a neural network with a Multiple Instance Learning (MIL) structure. A MIL neural network is a neural network that excels at determining whether a specific class is included in a given set of data.

[0082] To this end, the first neural network model can be trained based on training data labeled with a single BAG class that only specifies whether an instance corresponding to tumor tissue is included in a pathology image. For example, the first neural network model can only learn whether an instance corresponding to tumor tissue is included in a pathology image through training data labeled with only two BAG classes: "Class 1" for pathology images containing tumor tissue and "Class 0" for pathology images not containing tumor tissue. After training is complete, the first neural network model can output the probability that the input data contains an instance corresponding to tumor tissue.

[0083] At this time, the first neural network model of this document does not receive the entire pathological image as input, but rather the segmented patches as input, so it can reduce resource consumption by omitting judgment on blank images, and only distinguishes patches with a high probability of containing tumor tissue, thereby enabling more focused and precise observation in the subsequent use of the second neural network model.

[0084] In step S1040, the device (100) can select a patch to be observed from among a plurality of patches based on the probability determined by the first neural network model for each patch. For example, among the patches generated in step S1020, only patches with a high probability of containing tumor tissue can be used for the next second neural network model, and if the probability of containing tumor tissue determined by the first neural network model for all patches generated in step S1020 is less than or equal to a preset probability, the device (100) can omit the use of the second neural network model and conclude that there is no tumor tissue in the corresponding pathology image.

[0085] Meanwhile, if there is a patch among the patches generated in step S1020 that has a probability of containing tumor tissue determined by the first neural network model to be greater than a preset probability, the device (100) can select a patch with a probability greater than a preset probability and input it into the second neural network model in step S1050.

[0086] This is similar to the method where, when a specialist observes a pathology image and there is a part that is suspected to be tumor tissue, he or she observes that part together with another specialist to discuss it to reach a more accurate conclusion, but on the other hand, if there is no part that is suspected to be tumor tissue from the initial observation, he or she quickly concludes that there is no probability of cancer without further discussion.

[0087] At this time, the second neural network model is designed and trained in a different manner from the first neural network model, and can determine whether the patch contains tumor tissue through a different judgment method from the first neural network model.

[0088] The embodiment of this document presents the form of a second neural network model that can operate in conjunction with the first neural network model through the following Figures 9 to 13.

[0089] FIG. 9 is an example diagram showing a second neural network model configured and operating in the form of a recurrent neural network (RNN) according to one embodiment.

[0090] Referring to Fig. 9, the second neural network model can be configured as a neural network with an RNN structure. RNN neural networks generally have the advantage of making judgments on data by considering changes according to the temporal order when data with a temporal meaning are given. In addition, RNN (Recurrent Neural Networks) neural networks can determine whether tumor tissue is included by identifying not only the temporal order but also the spatial relationship and changes between the input patch orders. For example, in the case of training the second neural network model, a pathology image containing tumor tissue is segmented, and among the multiple patches generated, the order of patches containing a large amount of tumor tissue is input to the RNN neural network so that the presence of tumor is learned, and the presence of tumor tissue can be learned through the spatial relationship and changes between the patch orders. At this time, when training the RNN neural network, the input value input to the input layer of the RNN neural network can utilize the feature value extracted from the MIL neural network for a specific patch, and the output value input to the output layer of the RNN neural network can be set to be a class of whether or not cancer tissue is included (included: 1, not included: 0).

[0091] Accordingly, the device (100) arranges the patches in order of increasing probability for patches whose probability determined in step S1040 is greater than a predetermined threshold value, and sequentially inputs the feature values ​​extracted from the first neural network model for the corresponding patches into the second neural network model, thereby outputting the probability that the corresponding pathological image includes tumor tissue.

[0092] FIG. 10 and FIG. 11 are exemplary diagrams showing a second neural network model configured and operated in the form of an autoencoder with one encoder and one decoder according to one embodiment.

[0093] Referring to Fig. 10, the second neural network model can be configured as a neural network with an autoencoder structure with one encoder and one decoder. When any input data of Fig. 10 is input, the input data is compressed as much as possible (= expression vector), and the decoder is a neural network that restores the original input data form from the features of the compressed data as much as possible. The autoencoder of the second neural network model according to the embodiment of Fig. 10 can be trained to encode and decode and restore the input data based on training data of a pathological image containing only normal tissue. In this case, the autoencoder of the second neural network model according to the embodiment of Fig. 10 is trained to perform restoration well when a pathological image containing normal tissue is input, but to have poor restoration ability when a pathological image containing abnormal tissue (e.g., tumor tissue) is input. In the autoencoder, the difference between the original input data and the restored output data is called the reconstruction error.

[0094] Referring to FIG. 11, since the autoencoder of the second neural network model according to the embodiment of FIG. 10 will have a low restoration ability for a pathological image including abnormal tissue, when the device (100) receives the selected patch in step S1040 and performs encoding and decoding, it can determine a patch with a restoration error greater than a preset value (e.g., 0.5 in FIG. 11) as the location of tumor tissue among the pathological images.

[0095] FIGS. 12 to 14 are exemplary diagrams showing a second neural network model configured and operating in the form of an autoencoder including two encoders and one decoder according to one embodiment.

[0096] Referring to Fig. 12, the second neural network model can be configured as a neural network with an autoencoder structure, including two encoders and one decoder. The first encoder and the second encoder of Fig. 12 are neural networks that compress input data when input data is input, and the decoder is a neural network that restores the compressed data back to its original input data form from the characteristics of the data.

[0097] In the autoencoder of the second neural network model according to the embodiment of FIG. 12, the first encoder and the second encoder may be trained through different learning data or may have different layer structures.

[0098] Figures 13 and 14 illustrate an embodiment in which the first encoder and the second encoder are trained differently from each other.

[0099] First, in the embodiment of FIG. 13, in the neural network structure of the autoencoder according to the embodiment of FIG. 12, the first encoder is trained with parameters to encode input data based on training data of a pathological image containing only normal tissue, and the second encoder is trained with parameters to encode input data based on training data of a pathological image containing only abnormal tissue. Accordingly, the first expression vector generated by the first encoder and the second expression vector generated by the second encoder for the same input data may have different feature values.

[0100] That is, in the embodiment of FIG. 13, the linkage of the first encoder and decoder is trained to perform restoration well for patches containing only normal tissue, but not to properly perform restoration for patches containing abnormal tissue. The linkage of the second encoder and decoder is trained to perform restoration well for patches containing abnormal tissue, but not to properly perform restoration for patches containing only normal tissue. Accordingly, the first encoder and the second encoder exhibit different performances depending on whether an image containing only normal tissue or an image containing abnormal tissue is input, and the performance results can be confirmed through the restoration error between the input data and the first output data and the restoration error between the input data and the second output data.

[0101] Accordingly, in the embodiment of FIG. 13, when a patch is input, if the first restoration error through the linkage of the first encoder and the decoder is greater than the second restoration error through the linkage of the second encoder and the decoder, the device (100) can determine that the patch is abnormal (=including tumor tissue) and determine the location corresponding to the patch as the location of the tumor. In addition, when a patch is input, the device (100) can determine that the patch is normal (=excluding tumor tissue) if the first restoration error through the linkage of the first encoder and the decoder is less than the second restoration error through the linkage of the second encoder and the decoder.

[0102] Next, in the embodiment of FIG. 14, in the neural network structure of the autoencoder according to the embodiment of FIG. 12, both the first encoder and the second encoder learn parameters to encode input data based on training data of pathological images containing only normal tissues. At this time, the first encoder and the second encoder may be trained through training data having samples of different normal tissues, or may be designed to have different layer structures. Accordingly, the first expression vector generated by the first encoder and the second expression vector generated by the second encoder for the same input data may have different feature values.

[0103] In this case, the embodiment of FIG. 14 is trained so that both the first encoder and the second encoder perform restoration well when a pathological image including normal tissue is input, but the restoration ability is poor when a pathological image including abnormal tissue (e.g., tumor tissue) is input. Accordingly, since the first encoder and the second encoder of the second neural network model according to the embodiment of FIG. 14 have compression and restoration abilities for pathological images including normal tissue, the standard deviation for each restoration error is not large. In contrast, since the compression and restoration abilities for pathological images including abnormal tissue (e.g., tumor tissue) are not trained, the standard deviation for the difference in each restoration error appears large.

[0104] Accordingly, referring to FIG. 14, the device (100) can determine that a tumor tissue is included in the pathological image when the standard deviation of the difference between the restoration error of the first expression vector and the restoration error of the second expression vector generated by each of the two encoders is greater than or equal to a preset value (e.g., 0.1 in FIG. 13).

[0105] According to the above-described embodiment, the present invention can more effectively handle the diversity and complexity of cancer tissues by linking two or more neural networks learned in different ways and using them for cancer diagnosis, and can alleviate the overfitting problem of the neural network model and improve generalization ability.

[0106] In addition, since the neural network model of the present invention can be trained in different ways for the same training data set, the computation and resources required for training and executing the model can be reduced, thereby improving practicality, and since neural network models with different special features are linked, a diagnosis can be made by considering various characteristics of cancer tissue.

[0107] Therefore, the present invention can contribute significantly to the advancement of medical technology by achieving practical application of deep learning and machine learning technologies in the field of histopathological examination and, at the same time, significantly improving the accuracy and efficiency of cancer tissue diagnosis.

[0108] The various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but should be understood to encompass various modifications, equivalents, or alternatives of the embodiments. In connection with the description of the drawings, similar reference numerals may be used to refer to similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0109] In this document, the phrases "A or B", "at least one of A and B", "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can each include all possible combinations of the items listed together in that phrase. Terms such as "1", "2", or "first" or "second" may be used merely to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a (e.g., a first) component is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0110] The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0111] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., a memory) readable by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0112] Additionally, the processor of the embodiments of the present document can call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to the at least one command called. These one or more commands may include code generated by a compiler or code executable by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0113] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0114] The methods according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated on a machine-readable storage medium, such as a manufacturer's server, an application store's server, or a server's memory.

[0115] According to various embodiments, each component (e.g., a module or a program) of the described components may include one or more entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. A method for performing a cancer metastasis tissue determination device operated by a processor, An action of acquiring a pathological image containing tissue that is the target of cancer detection; An operation of dividing the above pathological image into preset sizes to create multiple patches; An operation of inputting the plurality of patches into a first neural network model trained to distinguish whether or not tumor tissue is included in a pathology image and determining the probability that each of the plurality of patches includes tumor tissue; An operation of selecting a patch to be observed among the plurality of patches based on the above probability; and An operation of inputting the selected patches into a second neural network model trained to determine whether tumor tissue is included based on multiple patches, thereby determining whether tumor tissue is included or the location of tumor tissue in the pathology image. method.

2. In paragraph 1, The above first neural network model is A neural network with a MIL (Multiple Instance Learning) structure is constructed, and a single BAG class that only determines whether an instance corresponding to tumor tissue is included in a pathological image is learned based on labeled learning data, and outputs the probability that the input data includes the instance. method.

3. In paragraph 2, The action of selecting the patch to be the above judgment target is An operation of classifying a patch determined to have a probability greater than a predetermined threshold; and Including an operation of arranging patches in the order of high probability for the above classified patches. method.

4. In paragraph 3, The above second neural network model is It is composed of a neural network with a RNN (Recurrent Neural Networks) structure, and is trained to determine whether tumor tissue is included by identifying changes in the order of input patches and the spatial relationship of the input patches, and outputs the probability that the classified patches include tumor tissue when the classified patches are input in the order in which they are arranged. method.

5. In paragraph 2, The above second neural network model is A neural network having an autoencoder structure including an encoder and a decoder, which learns to encode and decode input data based on learning data of a pathological image containing only normal tissues to restore the input data, and determines a patch with a reconstruction error greater than a preset value when encoding and decoding the selected patch as an input as the location of a tumor tissue in the pathological image. method.

6. In paragraph 2, The above second neural network model is A neural network having an autoencoder structure including two encoders and one decoder trained based on different training data, which is trained to encode and decode input data based on training data of a pathological image containing only normal tissue and to restore the input data, and when encoding and decoding the selected patch is performed by inputting the selected patch and performing encoding and decoding, if the standard deviation of the difference value of the reconstruction error by each of the two encoders is greater than a preset value, it is determined that the pathological image contains tumor tissue. method.

7. In paragraph 1, The action of generating the above multiple patches is An action of determining the border of a tissue included in the above pathological image; An action to remove data from an area outside the boundaries of the above organization; and An operation including generating a patch by dividing the inner area of ​​the border of the above organization into a preset size, method.

8. In paragraph 7, The operation of generating the above plurality of patches is performed after the operation of generating the above patches, If the tissue area included in the patch is 30% or more and 50% or less, an operation of making the tissue area included in the patch symmetrical left and right or up and down within the patch is included. method.

9. In paragraph 7, The operation of generating the above plurality of patches is performed after the operation of generating the above patches, If the tissue area included in the above patch is less than 30%, the action of copying the tissue area included in the above patch and pasting it into the blank area is included. method.

10. Memory containing instructions; and Includes a processor that performs a predetermined operation based on the above command, The operation of the above processor is as follows: An action of acquiring a pathological image containing tissue that is the target of cancer detection; An operation of dividing the above pathological image into preset sizes to create multiple patches; An operation of inputting the plurality of patches into a first neural network model trained to distinguish whether or not tumor tissue is included in a pathology image and determining the probability that each of the plurality of patches includes tumor tissue; An operation of selecting a patch to be observed among the plurality of patches based on the above probability; and An operation of inputting the selected patches into a second neural network model trained to determine whether tumor tissue is included based on multiple patches, thereby determining whether tumor tissue is included or the location of tumor tissue in the pathology image. Cancer metastasis tissue identification device.

Images