Method for training an artificial neural network to provide a judgment result for a pathological specimen, and computing system for carrying out the same

Abstract

A method and a computing system for performing such a method are disclosed, in which only parts stained with a specific stain are extracted from a pathology slide stained with a mixed stain containing various types of stains, and an artificial neural network is trained to make judgments on pathology images stained with various staining techniques. According to one aspect of the present invention, there is provided an invention in which a neural network training system generates a training data set including M individual training data (where M is a natural number equal to or greater than 2) and trains the training data set.

Description

[Technical field]

[0001] The present invention relates to a method for training an artificial neural network that provides a judgment result for a pathological specimen, and a computing system for performing this training. More specifically, the present invention relates to a method for making a judgment on a pathological image stained by various staining techniques by extracting only a portion stained with a specific stain from a pathological slide stained with a mixed stain containing various types of stains, and training an artificial neural network to this portion, and a computing system for performing this training. [Background technology]

[0002] As the average life expectancy of modern people increases, the incidence of serious diseases such as cancer also tends to increase. Therefore, pathological diagnostic methods are becoming more sophisticated, such as checking the expression of specific biomarkers and analyzing the state of the tumor microenvironment (TME) to predict the prognosis and determine treatment methods for patients diagnosed with cancer.

[0003] Hematoxylin-Eosin staining (hereinafter referred to as H&E staining) is a method in which the cell nuclei and cytoplasmic stroma are stained purple and pink, respectively, with hematoxylin and eosin. It is the most basic method used in pathology diagnosis, in which a pathologist grasps the morphological characteristics of the cells and tissues of a specimen through an optical microscope.

[0004] Immunohistochemistry (IHC) staining is a method often used to measure the expression levels of biomarkers, such as specific proteins in tissues. The expression levels of biomarkers present in tissues based on antigen-antibody binding can be confirmed by staining them with brown diaminobenzidine (DAB) staining agent, and hematoxylin staining is used as a counterstain.

[0005] Cocktail staining or multiplex IHC staining methods, in which multiple biomarkers are stained simultaneously to measure their expression levels and analyze the positional relationships between them, are also in the practical stage. In this case, various colors such as red and blue in addition to brown are used, and hematoxylin staining is used as a counterstain as usual.

[0006] In most cases, it is preferable to measure the expression of such biomarkers only in cancer lesions. For example, in the case of Ki-67, which indicates the degree of proliferation of cancer cells using Ki-67 protein that is temporarily expressed during cell division, it is appropriate to measure only cancer cells. Since Ki-67 may also be expressed in normal cells during cell division, normal cell regions must be excluded when measuring the expression level of such biomarkers. Even when analyzing the positional relationship between biomarkers, positional relationship analysis centered on cancer cells is significant, and therefore, the distinction between cancer lesions and normal cell regions is very important in pathological diagnosis.

[0007] Meanwhile, with the development of image analysis deep learning technology, attempts are underway to apply and commercialize deep learning technology in various medical fields.In the field of pathological diagnosis, instead of the conventional method of pathologists examining and reading stained tissue slides at high magnification through an optical microscope to make a diagnosis, a method is gradually being put into practical use in which tissue slides are converted into high-resolution digital images through a digital slide scanner and then the pathologist reads the images through a computer monitor.In addition, products are appearing that allow pathologists to refer to the results of image analysis using deep learning technology to make faster and more accurate diagnoses.

[0008] Methods for diagnosing cancer and analyzing its severity by applying image analysis and deep learning technology to whole-slide images (WSIs), which are images generated by scanning an entire slide, have already reached the commercial level, and there are products approved as medical devices both domestically and internationally for some carcinomas such as prostate cancer. All of these products perform functions such as analyzing WSIs generated from H&E stained slides to determine the presence or absence of cancerous lesions and visualizing the location of detected lesions.

[0009] In an advanced automated pathology diagnostic system that can automatically measure the expression levels of biomarkers, it is necessary to perform functions such as detecting and measuring biomarker expression by analyzing WSI generated from various types of stained slides other than H&E staining. In this case, it is important to detect and identify the location of cancer lesions from the images, so it is necessary to develop an automated method that can perform such functions.

[0010] When developing a method to detect lesions by analyzing H&E stained WSIs using image analysis and deep learning technology, a pathologist with expertise in cancer lesions will review the H&E stained WSIs used in the development of deep learning technology and annotate the lesion areas. This data construction process is also necessary when developing a method to detect lesions by analyzing WSIs with other types of staining. In other words, a pathologist will review WSIs with other types of staining than the target H&E staining and annotate the lesion areas, allowing the development of a method to detect lesions from WSIs with the relevant staining.

[0011] Since WSIs generated from various IHC-stained or cocktail-stained slides may have different hues depending on the staining method, when a deep learning technique for detecting cancer lesions generated from a WSI with a specific staining method is applied to a WSI with another type of staining, the detection performance may be significantly reduced. Meanwhile, new subjects to be stained by IHC staining methods may emerge as research on diseases progresses, so it is not realistic to use all existing IHC-stained and cocktail-stained images for technology development. In other words, the current deep learning technique requires a pathologist to annotate the lesion area for each WSI with different staining to construct data, making it difficult to develop a lesion detection method required for an advanced automatic pathology diagnosis system. Summary of the Invention [Problem to be solved by the invention]

[0012] The technical problem to be solved by the present invention is to provide a versatile method and system capable of detecting lesions caused by diseases such as cancer by analyzing pathology images generated from pathology slides prepared by staining with various staining methods.

[0013] Another object of the present invention is to provide a versatile lesion detection method and system for detecting cancer lesions by analyzing pathology images generated from pathology slides produced by staining a specific object with DAB and other colors and staining cell nuclei with hematoxylin, regardless of the type of object. [Means for solving the problem]

[0014] The present invention provides a method for training an artificial neural network, the method comprising: a step of a neural network training system generating a dataset including M pieces of individual training data (where M is a natural number equal to or greater than 2); and a step of the neural network training system inputting each of the M pieces of individual training data included in the dataset into an artificial neural network to train the artificial neural network, wherein the step of generating a dataset including the M pieces of individual training data includes a step of generating mth training data included in the dataset for all m, where 1≦m≦M, and the step of generating the mth training data includes a step of acquiring an mth original pathology image (wherein the mth original pathology image is a pathology image stained with a mixed stain obtained by mixing a predetermined stain and one or more stains excluding the predetermined stain), a step of performing color deconvolution to extract a portion stained with the stain from the mth original pathology image, thereby generating an mth training pathology image corresponding to the mth original pathology image, and a step of generating the mth training pathology image based on the mth original pathology image.

[0015] In one embodiment, the stain may be a Hematoxylin stain.

[0016] In one embodiment, a portion of the first to Mth original pathological images and another portion of the first to Mth original pathological images may be pathological images stained with different mixed stains.

[0017] In one embodiment, the step of extracting a portion stained with a dye from the mth original pathological image and generating an mth training pathological image corresponding to the mth original pathological image may include a step of converting the signal intensity of each channel in a color space representing the mth original pathological image into optical density, and a step of converting the optical density into staining intensity using a predetermined correlation equation.

[0018] According to another aspect of the present invention, there is provided a method for providing a judgment result for a predetermined pathological specimen via an artificial neural network trained by the above-mentioned artificial neural network training method, the method including the steps of: acquiring, by a computing system, a pathological image stained with a mixed stain comprising a mixture of a stain and one or more stains excluding the stain; extracting, by the computing system, a portion stained with the stain from the pathological image to generate an extracted image corresponding to the pathological image; and outputting, by the computing system, a judgment result for the pathological specimen judged by the artificial neural network based on the extracted image.

[0019] In one embodiment, at least some of the pathological image and the first to Mth original pathological images may be pathological images stained with a mixed staining agent different from each other.

[0020] In one embodiment, the mth original pathology image is annotated with a lesion area caused by a predetermined disease, and the step of generating the mth training pathology image based on the mth original pathology image may include a step of annotating the lesion area annotated in the mth original pathology image onto the mth training pathology image to generate the mth training data.

[0021] According to another aspect of the invention there is provided a computer program recorded on a medium for being provided in a data processing apparatus to perform the method as described above.

[0022] According to another aspect of the present invention, there is provided a computer readable recording medium having recorded thereon a computer program for carrying out the method described above.

[0023] According to another aspect of the present invention, the system is an artificial neural network training system, comprising a processor and a memory for storing a computer program, which, when executed by the processor, causes the computing system to execute a method for training an artificial neural network, the method comprising the steps of: generating a dataset including M individual training data (where M is a natural number equal to or greater than 2) by the neural network training system; and inputting each of the M individual training data included in the dataset to an input layer of the artificial neural network by the neural network training system. and training the artificial neural network by inputting the M individual training data, wherein the step of generating a training dataset including M individual training data includes a step of generating m-th training data included in the dataset for all m, where 1≦m≦M, and the step of generating the m-th training data includes a step of acquiring an m-th original pathology image (wherein the m-th original pathology image is a pathology image stained with a mixed stain comprising a predetermined stain and one or more stains excluding the predetermined stain, and a lesion area caused by a predetermined disease is annotated in the m-th original pathology image), a step of generating an m-th training pathology image corresponding to the m-th original pathology image by performing color deconvolution to extract a portion stained with the stain from the m-th original pathology image, and a step of annotating the lesion area annotated in the m-th original pathology image to the m-th training pathology image to generate the m-th training data.

[0024] In one embodiment, the stain may be a Hematoxylin stain.

[0025] In one embodiment, a portion of the first to Mth original pathological images and another portion of the first to Mth original pathological images may be pathological images stained with different mixed stains.

[0026] In one embodiment, the step of extracting a portion stained with a dye from the mth original pathological image and generating an mth training pathological image corresponding to the mth original pathological image may include a step of converting the signal intensity of each channel in a color space representing the mth original pathological image into optical density, and a step of converting the optical density into staining intensity using a predetermined correlation equation.

[0027] According to another aspect of the present invention, there is provided a system for providing a judgment result for a pathological specimen, the system including a processor and a memory for storing a computer program, which, when executed by the processor, causes the system for providing a judgment result to execute a method for providing a judgment result for a pathological specimen via an artificial neural network trained by the artificial neural network training method described in paragraph 1, the method for providing a judgment result including the steps of: the system for providing a judgment result acquiring a pathological image stained with a mixed stain comprising a mixture of a stain and one or more stains excluding the stain; the system for providing a judgment result extracting a portion stained with the stain from the pathological image to generate an extracted image corresponding to the pathological image; and the system for providing a judgment result outputting a judgment result for the pathological specimen judged by the artificial neural network based on the extracted image.

[0028] In one embodiment, the pathological image and at least some of the first to Mth original pathological images may be pathological images stained with a mixed staining agent different from each other. Effect of the Invention

[0029] According to the technical concept of the present invention, it is possible to effectively train an artificial neural network capable of determining a lesion or disease using images stained by different methods. That is, even if a plurality of images used as training data are not pathological images stained by the same method, they can be used as training data for training an artificial neural network by the above-mentioned neural network training method.

[0030] Meanwhile, in the case of a conventional technique for determining a pathological image using an artificial neural network, learning and determination are performed on pathological images stained with a specific method, but according to the technical idea of ​​the present invention, learning and determination are possible through images stained with different staining methods. For example, a specific object is stained with DAB or another color, and cell nuclei are stained with hematoxylin to prepare a slide, and a pathological image generated from the slide is analyzed to detect cancer lesions regardless of the type of object. [Brief description of the drawings]

[0031] In order to more fully understand the drawings referred to in the detailed description of the invention, a brief description of each of the drawings is provided.

[0032] [Figure 1] 1 is a schematic diagram illustrating an environment in which a method for learning an artificial neural network and a method for providing a judgment result for a pathological specimen according to the technical concept of the present invention are performed. [Diagram 2] 1 is a flowchart illustrating a neural network training method according to an embodiment of the present invention. [Figure 3a] FIG. 13 is a diagram showing an example of an original pathology image. [Figure 3b] FIG. 3B is a diagram showing annotated lesion areas in the pathological image of FIG. 3A. [Figure 3c] 3b is an example showing an image in which a portion stained with the staining agent to be extracted is extracted from the original pathology image shown in FIG. 3a. [Figure 3d] This shows training data generated from original pathology images tagged with annotation information such as that shown in Figure 3b. [Figure 4] 4 is an example illustrating a process of generating a learning image corresponding to an original pathology image according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating an example of a method for providing a judgment result for a pathology specimen according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram showing a schematic configuration of an artificial neural network training system according to an embodiment of the present invention. [Figure 7] 1 is a diagram showing a schematic configuration of a system for providing a judgment result according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Since the present invention can be modified in various ways and can have various embodiments, a specific embodiment will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to the specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. In describing the present invention, if a detailed description of related publicly known technology is deemed to obscure the gist of the present invention, the detailed description will be omitted.

[0034] Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms first, second, etc. do not indicate a particular order, but are used only to distinguish one component from another component.

[0035] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless otherwise clearly meant in the context.

[0036] In this specification, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof set forth in the specification, but are to be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0037] In addition, in this specification, when a component "transmits" data to another component, it means that the component may transmit data directly to the other component, or may transmit data to the other component via at least one other component. Conversely, when a component "transmits" data to another component, it means that data is transmitted from the component to the other component without passing through any other component.

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, in which the same reference numerals in different drawings denote the same elements.

[0039] FIG. 1 is a schematic diagram illustrating an environment in which a method for learning an artificial neural network and a method for providing a diagnosis result for a pathological specimen according to the technical concept of the present invention are performed.

[0040] 1, a method for training an artificial neural network according to an embodiment of the present invention is performed by a neural network training system 100, and a method for providing a judgment result for a pathology specimen according to an embodiment of the present invention is performed by a system for providing a judgment result for a pathology specimen 200 (hereinafter, referred to as a "judgment result providing system"). The neural network training system 100 can train an artificial neural network 300 for detecting a lesion contained in a pathology specimen, outputting diagnostic information for the pathology specimen, or providing prognostic information and / or reaction information for a treatment method, and the judgment result providing system 200 can make various judgments (e.g., detection of a lesion, presence or absence of disease, prognosis, judgment on a treatment method, etc.) on a target specimen using the trained artificial neural network 300.

[0041] The neural network learning system 100 and / or the judgment result providing system 200 may be a computing system, which is a data processing device having computing power for realizing the technical idea of ​​the present invention, and may include not only a server, which is generally a data processing device to which a client can be connected via a network, but also a computer device such as a personal computer or a mobile terminal.

[0042] The neural network learning system 100 and / or the judgment result providing system 200 may be realized by any physical device, but a person having ordinary skill in the technical field of the present invention will be able to easily deduce that the neural network learning system 100 and / or the judgment result providing system 200 according to the technical idea of ​​the present invention can be realized by organically combining multiple physical devices as necessary.

[0043] The neural network training system 100 can train the neural network 300 based on training data generated from a plurality of pathological specimens. The pathological specimens may be biopsies taken from various organs of the human body and biological tissues removed by surgery.

[0044] The neural network training system 100 generates individual training data using digital pathological images of a pathological specimen, and inputs this to an input layer of the neural network 300 to train the neural network 300.

[0045] The pathology image may be a digital slide image of a pathology specimen, or a part of a digital slide image. The slide of a pathology specimen may be a part obtained by slicing the pathology specimen. The digital slide image of a pathology specimen may be generated by slicing the pathology specimen to prepare a glass slide, staining the slide with a predetermined mixed staining agent, and digitizing the slide. In other words, the pathology image may be a pathology slide image obtained by slicing the pathology specimen to prepare a slide and staining the slide with a predetermined mixed staining agent (staining reagent), or an image obtained by dividing the stained pathology slide image into a predetermined size.

[0046] The pathology image may be a pathology image obtained by staining a slide of a pathology specimen with a mixed staining agent in which a plurality of individual staining agents (staining reagents) are mixed. The pathology image may be a whole slide image or a part of the whole slide image (e.g., a patch image obtained by dividing the whole slide image into a certain size).

[0047] The mixed stain may be a stain obtained by mixing a predetermined stain to be extracted and one or more stains excluding the predetermined stain to be extracted.

[0048] In one embodiment, the stain to be extracted may be a hematoxylin stain (hereinafter referred to as "H-stain"), and the following description will be given using an example in which the stain is an H-stain. However, the technical idea of ​​the present invention is not limited to this, and it goes without saying that various types of stains may be used.

[0049] When the stain to be extracted is an H-stain, the mixed stain may be a hematoxylin-eosin stain (H&E stain), a mixed stain for an IHC staining method in which hematoxylin staining is the counterstain, or a mixed stain for a DAB staining method in which hematoxylin staining is the counterstain, or may be any of various cocktail stains including hematoxylin stain.

[0050] The neural network 300 may be an artificial neural network trained to output a probability value for the presence or absence of a disease or the presence or absence of a lesion for a given disease. The neural network 300 can output a numerical value, i.e., a probability value, indicating a judgment result for a target specimen (e.g., the possibility of the onset of a disease) based on data input via an input layer.

[0051] In one embodiment, the artificial neural network 300 may be a patch-level clash fication neural network, which may be a neural network that receives an image in units of a patch and outputs a value for classifying the corresponding patch.

[0052] In another embodiment, the artificial neural network 300 may be a pixel-level knotification neural network, which may be a neural network that outputs a value for classifying an image for each pixel.

[0053] As used herein, an artificial neural network is an artificially constructed neural network based on the operating principles of human neurons, includes a multi-layer perceptron model, and may refer to a collection of information representing a set of design items that define the artificial neural network.

[0054] In one embodiment, the artificial neural network 300 may be or include a convolutional neural network.

[0055] Meanwhile, the trained neural network 300 may be stored in the determination result providing system 200, and the determination result providing system 200 can make a determination on a predetermined diagnostic target sample using the trained artificial neural network.

[0056] As shown in Fig. 1, the neural network learning system 100 and / or the judgment result providing system 200 may be realized in the form of a subsystem of a given parent system 10. The parent system 10 may be a server. The server 10 means a data processing device having a computing capability for realizing the technical idea of ​​the present invention, and an average expert in the technical field of the present invention can easily deduce that not only a data processing device to which a client can be connected via a network, but also any device capable of performing a specific service, such as a personal computer, a mobile terminal, etc., can be defined as a server.

[0057] Alternatively, depending on the embodiment, the neural network learning system 100 and the judgment result providing system 200 may be realized in a form separate from each other.

[0058] FIG. 2 is a flowchart illustrating a neural network training method according to an embodiment of the present invention.

[0059] 2, the neural network training system 100 can generate a training data set including M individual training data, where M is a natural number equal to or greater than 2. To this end, the neural network training system 100 can generate the mth training data included in the data set for all m, where 1≦m≦M (S100).

[0060] To generate the mth training data to be included in the training data set, the neural network training system 100 can acquire the mth original pathology image (where N is a natural number equal to or greater than 2) (S110).

[0061] According to an embodiment, the neural network learning system 100 can receive the mth original pathology image corresponding to a given pathology specimen from an external terminal, and can also acquire the mth original pathology image corresponding to the pathology specimen from a memory device that pre-stores the mth original pathology image corresponding to the pathology specimen.

[0062] The mth pathological image may be a pathological image stained with a mixed stain obtained by mixing a predetermined stain to be extracted (for example, hematoxylin stain) and one or more stains excluding the stain to be extracted.

[0063] The mth pathological image may be labeled with predetermined information. The labeled information may vary depending on the purpose of the artificial neural network 300. For example, the labeled information of the mth pathological image may be the presence or absence of a disease.

[0064] Alternatively, the mth original pathology image may be annotated with a lesion area caused by a predetermined disease. In other words, the mth original pathology image may be tagged with annotation information of the lesion area included in the mth original pathology image.

[0065] Fig. 3a is a diagram showing an example of an original pathology image, and Fig. 3b is a diagram showing an annotated lesion area of ​​the pathology image of Fig. 3a. In the example of Fig. 3b, the area shown by a solid line indicates the annotated lesion area.

[0066] Also, referring to FIG. 2, the neural network learning system 100 can extract a portion stained with a stain (e.g., hematoxylin stain) from the mth original pathology image to generate the mth learning pathology image (S120).

[0067] In one embodiment, the neural network learning system 100 can generate an m-th learning pathology image corresponding to the m-th original pathology image by performing color deconvolution. Figure 4 is an example showing a process of generating an m-th learning pathology image corresponding to the m-th original pathology image according to one embodiment of the present invention.

[0068] 4, the neural network learning system 100 can convert the signal intensity of each channel in the color space representing the mth original pathology image into optical density (S121). Then, the neural network learning system 100 can convert the optical density into staining intensity by a predetermined correlation formula (S122).

[0069] On the other hand, FIG. 3c is an example showing an image in which a portion stained with a stain to be extracted (eg, hematoxylin stain) is extracted from the original pathology image shown in FIG. 3a.

[0070] 2, the neural network learning system 100 can generate m-th learning data included in a learning data set based on the m-th learning pathology image (S130). Specifically, the neural network learning system 100 can generate the m-th learning data by tagging the m-th learning pathology image with information labeled on the m-th original image.

[0071] For example, the mth original image may be annotated with a lesion area as shown in Fig. 3b. In such a case, the mth learning pathology image generated as shown in Fig. 3c may be tagged with annotation information of the mth original image, an example of which is shown in Fig. 3d. That is, Fig. 3d shows learning data generated from an original pathology image tagged with annotation information as shown in Fig. 3b.

[0072] On the other hand, if there is diagnostic information, prognostic information, and / or response information to a particular treatment method for the pathological specimen corresponding to the mth training data, the neural network training system 100 can set this as the label of the mth training data.

[0073] When the training data set including M individual training data is generated by the above method, the neural network training system 100 can input the generated training data set to the input layer of the neural network 300 to train the neural network 300 (S140 in FIG. 2). Deep learning techniques, in which training data tagged with correct information is input to the artificial neural network 300 and the artificial neural network 300 is trained by methods such as gradient descent and backpropagation, are widely known and will not be described in detail.

[0074] Meanwhile, in some embodiments, the M pathological images used to configure the learning dataset may be stained with two or more different mixed stains. In other words, a part of the first to M original pathological images and another part of the first to M original pathological images may be pathological images stained with different mixed stains. Of course, the two or more mixed stains used to stain the M pathological images may include a common stain to be extracted. For example, when M is 150, the first to 50 pathological images may be pathological images stained by the H&E staining method, the 51st to 100th pathological images may be pathological images stained by the IHC staining method (counterstaining is H-staining), and the 101st to 150th pathological images may be pathological images stained by the DAB staining method (counterstaining is H-staining). Of course, in some embodiments, the M pathological images may be pathological images stained by four or more different mixed stains.

[0075] As described above, the neural network training method according to the technical concept of the present invention extracts only the parts stained with the staining agent to be extracted from the original pathological image, and uses the extracted parts as training data for the artificial neural network. Therefore, the neural network training method according to the technical concept of the present invention has an effect of effectively performing training using pathological images produced by various staining methods using the staining agent to be extracted. In other words, the neural network training method according to the technical concept of the present invention has an effect of effectively training an artificial neural network capable of determining a lesion or disease using images stained by different methods. That is, even if the multiple images used as training data are not pathological images stained by the same method, the neural network training method has an effect of being able to use them as training data for training one artificial neural network (e.g., 300).

[0076] Fig. 5 is a flow chart showing an example of a method for providing a determination result for a pathological specimen according to an embodiment of the present invention. The method for providing a determination result for a pathological specimen according to Fig. 5 is performed by a determination result providing system 200, which may store an artificial neural network 300 trained by the neural network learning system 100.

[0077] 5, the judgment result providing system 200 can acquire a judgment target pathology image of a predetermined judgment target pathology specimen (S210). At this time, the judgment target pathology image may be a pathology slide image stained with a mixed stain obtained by mixing a slide of the judgment target pathology specimen with an extraction target stain (e.g., hematoxylin stain) and one or more stains excluding the extraction target stain (e.g., hematoxylin stain).

[0078] The judgment result providing system 200 can extract a portion stained with the extraction target dye from the judgment target pathology image to generate an extracted image corresponding to the judgment target pathology image (S220). The process of generating input data corresponding to the judgment target specimen pathology image is very similar to the above-mentioned process, so a separate description will be omitted.

[0079] The determination result providing system 200 can input input data to the artificial neural network 300 and output a determination result for the pathological specimen based on the result output by the artificial neural network (S230).

[0080] On the other hand, the staining method used to stain at least a part of the original pathological image used to generate the learning data may be different from the staining method used to stain the pathological image to be judged. That is, the image to be judged and at least a part of the images (the first to Mth original pathological images) used to train the artificial neural network 300 may be pathological images stained with different mixed stains. For example, all or at least a part of the images used to train the artificial neural network 300 are pathological images stained with the H&E staining method, but the image to be judged may be an image stained with the IHC method.

[0081] Alternatively, the staining method used to stain the target image may be different from the staining method used to stain any of the images used to train the artificial neural network 300. For example, the staining method used to stain the target image may be a DAB method, but all of the images used to train the artificial neural network 300 may be images stained by an H&E staining method.

[0082] In the case of a conventional technique for determining a pathological image using an artificial neural network, learning and determination are performed on pathological images stained with a specific method, but according to the technical idea of ​​the present invention, it is possible to learn and determine through images stained with different staining methods. For example, it is possible to detect cancer lesions by analyzing pathological images generated from slides produced by staining a specific object with DAB or another color and staining cell nuclei with hematoxylin, regardless of the type of object.

[0083] FIG. 6 is a diagram showing a schematic configuration of an artificial neural network learning system 100 according to an embodiment of the present invention, and FIG. 7 is a diagram showing a schematic configuration of a judgment result providing system 200 according to an embodiment of the present invention.

[0084] The artificial neural network learning system 100 and the judgment result providing system 200 refer to a logical configuration having hardware resources and / or software necessary to realize the technical idea of ​​the present invention, and do not necessarily refer to one physical component or one device. In other words, the artificial neural network learning system 100 and the judgment result providing system 200 refer to a logical combination of hardware and / or software provided to realize the technical idea of ​​the present invention, and may be realized as a set of logical configurations for realizing the technical idea of ​​the present invention by being provided in devices separated from each other and performing their respective functions, as necessary. Also, the artificial neural network learning system 100 and the judgment result providing system 200 may refer to a set of configurations realized for each function or role to realize the technical idea of ​​the present invention. The artificial neural network learning system 100 and the judgment result providing system 200 may be located in different physical devices or in the same physical device. In addition, depending on the implementation example, the software and / or hardware constituting each of the components of the artificial neural network learning system 100 and the judgment result providing system 200 may be located in different physical devices, and each module may be realized by organically combining the components located in the different physical devices with each other.

[0085] In addition, the term "module" in this specification may refer to a functional and structural combination of hardware for carrying out the technical idea of ​​the present invention and software for driving the hardware. For example, a module refers to a logical unit of a specific code and a hardware resource for carrying out the specific code, and does not necessarily refer to physically connected code or one type of hardware, as can be easily inferred by an average expert in the technical field of the present invention.

[0086] 6, the artificial neural network training system 100 may include a storage module 110, an acquisition module 120, a generation module 130, and a training module 140. Depending on the embodiment of the present invention, some of the above-mentioned components may not necessarily be essential components for the realization of the present invention, and it goes without saying that the artificial neural network training system 100 may include more components than the above. For example, the artificial neural network training system 100 may further include a communication module (not shown) for communicating with an external device, and a control module (not shown) for controlling the components and resources of the artificial neural network training system 100.

[0087] The storage module 110 is capable of storing the trained artificial neural network 40 .

[0088] The acquisition module 120 can acquire M original pathology images stained with a predetermined stain to be extracted and a mixed stain obtained by mixing one or more stains excluding the stain to be extracted.

[0089] The generation module 130 can generate individual training data based on each original pathology image, and can configure a training data set including a plurality of individual training data.

[0090] The training module 140 can train the artificial neural network 300 based on a training data set.

[0091] 7, the judgment result providing system 200 may include a storage module 210, an acquisition module 220, a generation module 230, and a judgment module 240. Depending on the embodiment of the present invention, some of the above-mentioned components may not necessarily correspond to components essential to realizing the present invention, and it goes without saying that the judgment result providing system 200 may include more components than the above components depending on the embodiment. For example, the judgment result providing system 200 may further include a communication module (not shown) for communicating with the three-axis vibration sensor 20 and a control module (not shown) for controlling the components and resources of the judgment result providing system 200.

[0092] The storage module 210 is capable of storing the trained artificial neural network 40 .

[0093] The acquisition module 220 can acquire a target pathology image in which a predetermined pathological specimen is stained with a mixed stain obtained by mixing one or more stains excluding the stain to be extracted.

[0094] The generation module 230 can generate input data based on the target pathology image.

[0095] The judgment module 240 inputs input data to the artificial neural network 300 and can make a judgment on the specimen to be judged based on the predicted value output from the artificial neural network 300.

[0096] Meanwhile, depending on the implementation, the artificial neural network training system 100 and the judgment result providing system 200 may include a processor and a memory for storing a program executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include a high-speed random access memory, and may include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory by the processor and other components may be controlled by a memory controller.

[0097] Meanwhile, the method according to the embodiment of the present invention may be realized in the form of computer readable program instructions and stored in a computer readable recording medium, and the control program and the target program according to the embodiment of the present invention may also be stored in a computer readable recording medium. The computer readable recording medium includes any kind of recording device in which data readable by a computer system is stored.

[0098] The program instructions contained on the recording media may be those specially designed and constructed for the purposes of the present invention, or they may be of the type well known and available to those having skill in the software arts.

[0099] Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, etc. Also, computer readable recording media may be distributed among computer systems connected via a network, so that computer readable code is stored and executed in a distributed fashion.

[0100] Examples of program instructions include not only machine code, such as produced by a compiler, but also higher level language codes that may be executed by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0101] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0102] The above description of the present invention is for illustrative purposes only, and a person having ordinary skill in the art to which the present invention pertains can easily understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described in a single form can be implemented in a distributed form, and similarly, each component described as being distributed can be implemented in a combined form.

[0103] The scope of the present invention is indicated by the claims set forth below, rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalent concepts should be interpreted as being included within the scope of the present invention. [Industrial Applicability]

[0104] The present invention can be used in a method for training an artificial neural network to provide a judgment result for a pathological specimen, and in a computing system for carrying out this method.

Claims

1. 1. A method of training an artificial neural network, comprising: A neural network training system generates a training data set including M individual training data (where M is a natural number equal to or greater than 2); said neural network training system training said artificial neural network based on said training data set; Including, The step of generating a training data set including the M individual training data includes a step of generating m-th training data included in the training data set for all m, where 1≦m≦M; The step of generating m training data includes: acquiring an m-th original pathology image (wherein the m-th original pathology image is a pathology image stained with a mixed stain obtained by mixing a predetermined extraction target stain and one or more stains excluding the extraction target stain); extracting a portion stained with the extraction target stain from the mth original pathology image to generate an mth learning pathology image corresponding to the mth original pathology image; generating the mth training data based on the mth training pathology images.

2. 2. The method of claim 1, wherein the stain to be extracted is a hematoxylin stain.

3. 2. The artificial neural network training method according to claim 1 , wherein a portion of the first to M original pathology images and another portion of the first to M original pathology images are pathology images stained with a mixed staining agent different from each other.

4. The step of extracting a portion stained with the extraction target stain from the mth original pathology image to generate an mth learning pathology image corresponding to the mth original pathology image includes: converting the signal intensity of each channel in a color space representing the m original pathology images to an optical density; and converting said optical density into staining intensity by a predetermined correlation equation.

5. The mth original pathology image is annotated with a lesion area caused by a predetermined disease, The step of generating the mth learning data based on the mth learning pathology image includes:

2. The artificial neural network training method according to claim 1, further comprising the step of: annotating the mth training pathology image with a lesion area annotated in the mth original pathology image to generate the mth training data.

6. A method for providing a judgment result for a predetermined pathological specimen to be judged through an artificial neural network trained by the method for training an artificial neural network according to claim 1, comprising: A computing system acquires a pathology image to be judged, which is stained with a mixed stain obtained by mixing the stain to be extracted and one or more stains excluding the stain to be extracted; a step of the computing system extracting a portion stained with the extraction target stain from the pathology image to be determined, and generating an extracted image corresponding to the pathology image to be determined; a step of outputting a judgment result for the pathological specimen judged by the artificial neural network based on the extracted image, by the computing system; A method comprising:

7. The method according to claim 6 , wherein the target pathology image and at least some of the first to Mth original pathology images are pathology images stained with mixed stains different from each other.

8. A computer program recorded on a medium for being provided in a data processing apparatus for carrying out the method according to any one of claims 1 to 7.

9. A computer-readable recording medium having a computer program recorded thereon for carrying out the method according to any one of claims 1 to 7.

10. 1. An artificial neural network learning system, comprising: A processor; A memory for storing a computer program; Including, The computer program, when executed by the processor, causes the artificial neural network training system to perform a method for training an artificial neural network, The method for training the artificial neural network includes: The neural network training system generates a training data set including M individual training data (where M is a natural number equal to or greater than 2); the neural network training system training the artificial neural network based on the training data set; The step of generating a training data set including the M individual training data includes a step of generating m-th training data included in the training data set for all m, where 1≦m≦M; The step of generating m training data includes: acquiring an m-th original pathology image (wherein the m-th original pathology image is a pathology image stained with a mixed stain obtained by mixing a predetermined extraction target stain and one or more stains excluding the extraction target stain); extracting a portion stained with the extraction target stain from the mth original pathology image to generate an mth learning pathology image corresponding to the mth original pathology image; generating the mth training data based on the mth training pathology images.

11. 11. The artificial neural network training system of claim 10, wherein the stain to be extracted is a hematoxylin stain.

12. 11. The artificial neural network learning system of claim 10, wherein a portion of the first to M original pathology images and another portion of the first to M original pathology images are pathology images stained with a mixed staining agent different from each other.

13. The mth original pathology image is annotated with a lesion area caused by a predetermined disease, The step of generating the mth learning data based on the mth original pathology image includes:

11. The artificial neural network training system of claim 10, further comprising a step of annotating the mth training pathology image with a lesion area annotated in the mth original pathology image to generate the mth training data.

14. The step of extracting a portion stained with the extraction target stain from the mth original pathology image to generate an mth learning pathology image corresponding to the mth original pathology image includes: converting the signal intensity of each channel in a color space representing the m original pathology images to an optical density; and converting said optical density into staining intensity by a predetermined correlation equation.

15. A system for providing judgment results for pathology samples, comprising: A processor; A memory for storing a computer program; Including, The computer program, when executed by the processor, causes the determination result providing system to execute a method for providing a determination result for a pathological specimen via an artificial neural network trained by the artificial neural network training method according to claim 1, The method for providing a judgment result includes: The determination result providing system acquires a pathology image to be determined stained with a mixed stain obtained by mixing the extraction target stain and one or more stains excluding the extraction target stain; a step of the judgment result providing system extracting a portion stained with the extraction target stain from the judgment target pathology image to generate an extracted image corresponding to the judgment target pathology image; A system for providing a judgment result for a pathological specimen, comprising a step of outputting a judgment result for the pathological specimen to be judged, the judgment result being judged by the artificial neural network based on the extracted image.

16. The system for providing judgment results according to claim 15, wherein the judgment target image and at least a portion of the first to Mth original pathology images are pathology images stained with mixed stains different from each other.

Images