Method for training an artificial neural network to provide a judgment result for a pathological specimen, and computing system for carrying out the same

An artificial neural network trained on multi-stained pathology slides addresses the limitations of conventional tumor microenvironment analysis, providing accurate cancer diagnosis and prognosis by superimposing multi-channel images, thus enhancing analysis accuracy.

JP7580158B2Active Publication Date: 2024-11-11DEEP BIO
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023540030
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2021-01-07
Filing Date
2022-01-07
Publication Date
2024-11-11
Estimated Expiration
2042-01-07

AI Technical Summary

Technical Problem

Existing methods for analyzing the tumor microenvironment in cancer diagnosis and prognosis are limited by the need for complex manual analysis of multiple slides and lack comprehensive understanding of positional relationships, with high-cost multiplex IHC methods not widely adopted.

Method used

A method using an artificial neural network trained on pathology slides stained with H&E and various IHC reagents, enabling comprehensive analysis by superimposing multi-channel images of consecutive sections to understand the tumor microenvironment accurately.

Benefits of technology

Enables highly accurate understanding of cancer diagnosis, prognosis, and treatment responsiveness by analyzing the tumor microenvironment without high-cost multiplex IHC, reducing errors and increasing analysis accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580158000001
    Figure 0007580158000001
  • Figure 0007580158000002
    Figure 0007580158000002
  • Figure 0007580158000003
    Figure 0007580158000003
Patent Text Reader

Abstract

A method and computing system for training an artificial neural network to make accurate disease determinations using pathology slides in which serial sections of a single specimen are stained with a variety of different staining reagents is disclosed. According to one aspect of the present invention, there is provided a method for training an artificial neural network, the method comprising: a neural network training system generating a training dataset including M pieces of individual training data (where M is a natural number equal to or greater than 2); and the neural network training system training the artificial neural network based on the training dataset, wherein the generating the training dataset including the M pieces of individual training data includes generating mth training data included in the training dataset for all m, 1<=m<=M, and the generating the mth training data includes acquiring first through Nth pathology slide images (where N is a natural number equal to or greater than 2) (where the first through Nth pathology slide images are pathology slide images obtained by staining serial sections of a single pathology specimen with different staining reagents), and generating the mth training data based on the first through Nth pathology slide images.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a method for training an artificial neural network to provide a diagnosis result for a pathological specimen, and a computing system for performing the same, and more particularly to a method for training an artificial neural network using pathological slides in which serial sections of a single specimen are stained with various different staining reagents, thereby enabling accurate diagnosis of a disease, and a computing system for performing the same. [Background technology]

[0002] The adjacent cells that interact with tumor cells around the tumor and affect tumor growth are called the tumor microenvironment. Research into the tumor microenvironment is extremely important for understanding the diagnosis, prognosis, and response to specific treatment methods regarding the current state of cancer, as well as for developing new treatment methods.

[0003] In the past, immunohistochemistry (IHC) staining reagents targeting specific immune cells or proteins expected to be present in the tumor microenvironment were used to analyze the tumor microenvironment. That is, a pathological specimen was stained with an IHC staining reagent for a specific target, and the staining results were visually read by a pathologist through an optical microscope to grasp the positional relationship of the target and measure the amount. In this case, since various immune cells or proteins must be viewed and judged in a complex manner, multiple slides consisting of consecutive sections of a single specimen are prepared, each of which is stained with H&E and various target IHC staining reagents, and the results are read by a pathologist and combined. Since the positional relationship of various elements that make up the tumor microenvironment is important, there are limitations to this type of analysis, and in order to overcome these limitations, a multiplex IHC staining method that stains various targets at once has emerged, but it is not cheap and has not been widely used.

[0004] Meanwhile, technologies have been developed and commercialized that use deep learning, particularly convolutional neural networks (CNNs), to analyze pathology slide images to diagnose the current state of cancer, diagnose prognosis according to severity, and measure specific IHC staining results. The currently developed technologies target single stained pathology slide images, and the analysis results are provided in the form of specific classifications or numbers, making it difficult to easily combine them to understand the positional relationship to the tumor microenvironment. For example, according to a previous patent (JP6650453B2), slide images consisting of consecutive sections are used to determine the prognosis of cancer, but the results of analyzing each slide image separately are combined to determine the prognosis, and no analysis is performed that comprehensively analyzes the positional relationship of all slide images.

[0005] Therefore, there is a need for a technical concept that enables comprehensive analysis, including the positional relationship, of multi-stained slides consisting of consecutive sections from a single specimen so that the tumor microenvironment can be analyzed without using a multiplex IHC method. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6650453 Summary of the Invention [Problem to be solved by the invention]

[0007] The technical problem that the present invention aims to achieve is to provide a method and system for training an artificial neural network using pathology slides in which each successive section of a single specimen is stained with H&E or various targeted IHC staining reagents, etc., so that the trained artificial neural network can comprehensively analyze the tumor microenvironment, thereby enabling highly accurate diagnosis, prognosis, and response to specific therapeutic methods regarding the current state of cancer to be understood. [Means for solving the problem]

[0008] According to one aspect of the present invention, there is provided a method for training an artificial neural network, comprising: a step of a neural network training system generating a training dataset including M pieces of individual training data (where M is a natural number equal to or greater than 2); and a step of the neural network training system training the artificial neural network based on the training dataset, wherein the step of generating the training dataset including the M pieces of individual training data includes a step of generating mth training data included in the training dataset for all m, where 1<=m<=M, and the step of generating the mth training data includes a step of acquiring a first pathology slide image through an Nth pathology slide image (where N is a natural number equal to or greater than 2), where the first pathology slide image through the Nth pathology slide image are pathology slide images obtained by staining serial sections of a single pathology specimen with different staining reagents, and a step of generating the mth training data based on the first pathology slide image through the Nth pathology slide images.

[0009] In one embodiment, the step of generating the mth learning data based on the first pathology slide image through the Nth pathology slide image includes a step of converting the first pathology slide image through the Nth pathology slide image into a single multi-channel image by channel stacking, and the mth learning data may include the multi-channel image.

[0010] In one embodiment, the learning data includes N channels, and the step of converting the first pathology slide image through the Nth pathology slide image into one multi-channel image by channel superposition may include a step of constituting the nth channel of the multi-channel image with each pixel value of the Nth pathology slide image, for all natural numbers n, 1<=n<=N.

[0011] In one embodiment, the step of generating the mth learning data based on the first pathology slide image through the Nth pathology slide image includes a step of identifying biological tissue areas present in the first pathology slide image through the Nth pathology slide image, a step of aligning the first pathology slide image through the Nth pathology slide image so that the positions and morphologies of the biological tissue areas present in each of the first pathology slide image through the Nth pathology slide image match, and a step of converting the aligned first pathology slide image through the aligned Nth pathology slide image into a single multi-channel image by channel superposition, and the mth learning data may include the multi-channel image.

[0012] In one embodiment, the learning data includes N channels, and the step of converting the aligned first pathology slide image through the aligned Nth pathology slide image into one multi-channel image by channel superposition may include a step of constituting the nth channel of the multi-channel image with each pixel value of the aligned Nth pathology slide image for all natural numbers n, 1<=n<=N.

[0013] In one embodiment, the step of aligning the first through N pathology slide images so that positions and morphologies of biological tissue regions present in each of the first through N pathology slide images match includes a step of calculating a transformation relationship corresponding to the i-th pathology slide image for all natural numbers i, 1<=i<=N (wherein the transformation relationship corresponding to the i-th pathology slide image is a transformation relationship between the i-th pathology slide image and the corresponding aligned i-th pathology slide image), and the step of generating the m learning data based on the first through N pathology slide images further includes a step of transforming a lesion annotation area assigned to the j-th pathology slide image using the transformation relationship corresponding to the j-th pathology slide image, for all natural numbers j, 1<=j<=N, and a step of transforming the lesion annotation area of ​​the transformed first pathology slide image through the lesion annotation area of ​​the transformed N-th pathology slide image into one multi-channel lesion annotation area by channel superposition, and the m learning data may further include the multi-channel lesion annotation area.

[0014] According to another aspect of the present invention, there is provided a method for providing a judgment result for a predetermined pathology specimen to be judged via an artificial neural network trained by the above-mentioned artificial neural network training method, the method including a step of a computing system acquiring a first pathology slide image to an Nth pathology slide image to be judged (wherein N is a natural number equal to or greater than 2) (wherein the first pathology slide image to the Nth pathology slide image to be judged are pathology slide images in which consecutive sections of the pathology specimen to be judged are stained with different staining reagents), and a step of the computing system outputting a judgment result for the pathology specimen to be judged by the artificial neural network based on the first pathology slide image to the Nth pathology slide image to be judged.

[0015] According to another aspect of the invention there is provided a computer program recorded on a medium for installing in a data processing device and for carrying out the method as described above.

[0016] According to another aspect of the present invention, there is provided a computer readable recording medium having recorded thereon a computer program for carrying out the method described above.

[0017] According to another aspect of the present invention, there is provided an artificial neural network training system, comprising a processor and a memory for storing a computer program which, when executed by the processor, causes the computing system to perform a method for training an artificial neural network, the artificial neural network training system comprising: generating a training data set including M individual training data, where M is a natural number equal to or greater than 2, by the neural network training system; and training the artificial neural network based on the training data set by the neural network training system, the M individual training data being generated by the neural network training system. the step of generating a learning dataset including the above includes a step of generating mth learning data included in the learning dataset for all m, where 1<=m<=M, and the step of generating the mth learning data includes a step of acquiring a first pathology slide image through an Nth pathology slide image (where N is a natural number equal to or greater than 2) (where the first pathology slide image through the Nth pathology slide image are pathology slide images obtained by staining successive sections of a single pathology specimen with different staining reagents), and a step of generating the mth learning data based on the first pathology slide image through the Nth pathology slide images.

[0018] According to another aspect of the present invention, there is provided a system for providing a judgment result for a pathology specimen, the system including a processor and a memory for storing a computer program, which, when executed by the processor, causes the computing system to execute a method for providing a judgment result for a pathology specimen via an artificial neural network trained by the above-mentioned artificial neural network training method, the method for providing the judgment result including the steps of acquiring a first to an Nth pathology slide image to be judged (wherein N is a natural number equal to or greater than 2) (wherein the first to the Nth pathology slide images to be judged are pathology slide images in which consecutive sections of a predetermined pathology specimen to be judged are stained with different staining reagents), and outputting a judgment result for the pathology specimen to be judged by the artificial neural network based on the first to the Nth pathology slide images to be judged. Effect of the Invention

[0019] According to the technical idea of ​​the present invention, a method and system can be provided in which an artificial neural network is trained using pathology slides in which each successive section of a single specimen is stained with H&E or various targeted IHC staining reagents, etc., so that the trained artificial neural network can comprehensively analyze the tumor microenvironment, thereby enabling diagnosis, prognosis, and response to specific treatment methods regarding the current state of cancer to be grasped with high accuracy.

[0020] In addition, according to the technical idea of ​​the present invention, instead of a costly and uncommon method such as multiplex IHC, multiple pathology slide images generated using a widely used method can be superimposed and used, thereby obtaining effects similar to those of multiplex IHC. In addition, the accuracy of tumor microenvironment analysis can be improved by eliminating the possibility of errors that may occur in the process of separating the location information of each target from the results of multiplex IHC through color filtering. [Brief description of the drawings]

[0021] In order to more fully understand the drawings referred to in the detailed description of the present invention, a brief description of each of the drawings is provided. [Figure 1] 1 is a schematic diagram illustrating an environment in which a method for training an artificial neural network and a method for providing a judgment result for a pathological specimen according to the technical concept of the present invention are performed. [Diagram 2] 1 is a flowchart illustrating a neural network training method according to an embodiment of the present invention. [Diagram 3] FIG. 2 shows one multi-channel image generated by channel superposition from multiple pathology slide images represented in the RGB color model. [Figure 4] FIG. 2 is a diagram illustrating an example of a process for generating individual learning data according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating an example of a method for providing a determination result for a pathology specimen according to an embodiment of the present invention. [Figure 6] FIG. 1 is a diagram showing a schematic configuration of an artificial neural network training system according to an embodiment of the present invention. [Figure 7] 1 is a diagram showing a schematic configuration of a judgment result providing system according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Since the present invention can be modified in various ways and can have various embodiments, a specific embodiment is shown in the drawings and described in detail. However, this is not intended to limit the present invention to the specific embodiment, and it should be understood that the present invention includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present invention. In the description of the present invention, if a detailed description of related publicly known technology is deemed to make the gist of the present invention unclear, the detailed description will be omitted.

[0023] Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. Terms such as first, second, etc. do not indicate a particular order, but are used only to distinguish one component from another component.

[0024] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless otherwise clearly meant in the context.

[0025] As used herein, terms such as "comprise" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described herein, and are to be understood as not precluding the presence or additional possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0026] In addition, in this specification, when a component "transmits" data to another component, it means that the component can transmit data directly to the other component, or can transmit data to the other component via at least one other component. Conversely, when a component "transmits" data to another component, it means that the component transmits data to the other component without via the other component.

[0027] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, in which the same reference numerals indicate the same elements.

[0028] FIG. 1 is a schematic diagram showing an environment in which a method for training an artificial neural network and a method for providing a determination result for a pathological specimen according to the technical concept of the present invention are carried out.

[0029] 1, a method for training an artificial neural network according to an embodiment of the present invention can be performed by a neural network training system 100, and a method for providing a judgment result for a pathological specimen according to an embodiment of the present invention can be performed by a system for providing a judgment result for a pathological specimen (200; hereinafter, referred to as a "judgment result providing system"). The neural network training system 100 can train an artificial neural network 300 for providing diagnostic information, prognostic information and / or reaction information for a treatment method for a pathological specimen, and the judgment result providing system 200 can make various judgments (e.g., whether or not a disease occurs, prognosis, judgment on a treatment method, etc.) on a target specimen using the trained artificial neural network 300.

[0030] The neural network learning system 100 and / or the judgment result providing system 200 may be a computing system, which is a data processing device having computing power for realizing the technical idea of ​​the present invention, and may include not only a server, which is generally a data processing device to which a client can be connected via a network, but also a computer device such as a personal computer or a mobile terminal.

[0031] The neural network learning system 100 and / or the judgment result providing system 200 can be realized by a single physical device, but an average expert in the technical field of the present invention can easily deduce that the neural network learning system 100 and / or the judgment result providing system 200 according to the technical idea of ​​the present invention can be realized by organically combining multiple physical devices as necessary.

[0032] The neural network training system 100 can train the neural network 300 based on training data generated from a plurality of pathological specimens.

[0033] The pathology specimens may be biopsies taken from various organs of the human body and tissues removed by surgery. The neural network training system 100 may train the neural network 300 by generating personalized training data using digital pathology slide images of successive sections of the pathology specimen and inputting the personalized training data to the input layer of the neural network 300.

[0034] In one embodiment, the neural network 300 may be an artificial neural network trained to output a probability value for the presence or absence of a given disease. The neural network 300 can output a numerical value, i.e., a probability value, indicating a judgment result for a target sample (e.g., the possibility of the presence or absence of a disease) based on data input via an input layer.

[0035] An artificial neural network in this specification is an artificially constructed neural network based on the operating principles of human neurons, includes a multi-layer perceptron model, and can refer to a collection of information representing a series of design items that define the artificial neural network.

[0036] In one embodiment, the artificial neural network may be or include a convolutional neural network.

[0037] Meanwhile, the trained neural network 300 can be stored in the determination result providing system 200, and the determination result providing system 200 can make a determination on a predetermined diagnostic target specimen using the trained artificial neural network.

[0038] As shown in Fig. 1, the neural network learning system 100 and / or the judgment result providing system 200 can be realized in the form of a subsystem of a given parent system 10. The parent system 10 may be a server. The server 10 means a data processing device having a computing capability for realizing the technical idea of ​​the present invention, and an average expert in the technical field of the present invention can easily deduce that not only a data processing device to which a client can be connected via a network, but also any device capable of executing a specific service, such as a personal computer or a mobile terminal, can be defined as a server.

[0039] Alternatively, depending on the embodiment, the neural network learning system 100 and the judgment result providing system 200 may be realized in a form separate from each other.

[0040] FIG. 2 is a flowchart illustrating a neural network training method according to an embodiment of the present invention.

[0041] 2, the neural network training system 100 can generate a training data set including M individual training data, where M is a natural number equal to or greater than 2. To this end, the neural network training system 100 can generate m-th training data to be included in the training data set for all m, where 1<=m<=M (S100).

[0042] To generate the mth training data to be included in the training dataset, the neural network training system 100 can acquire the first pathology slide image through the Nth pathology slide image (where N is a natural number greater than or equal to 2) (S110).

[0043] In this case, the first pathology slide image to the Nth pathology slide image may be pathology slide images in which consecutive sections of a single pathology specimen are stained with different staining reagents.

[0044] Each section of the pathology specimen may be a slice of the pathology specimen to produce a digital slide image, or the pathology specimen may be continuously sliced ​​to produce a plurality of glass slides, which may then be stained with different staining reagents and digitized to produce the first through Nth pathology slide images. In this case, the staining reagents may be H&E (Hematoxylin and Eosin) staining reagents or IHC (immunohistochemistry) staining reagents for a specific target.

[0045] For example, a single pathology specimen can be sliced ​​into successive slices, and each sliced ​​section can be stained sequentially with an H&E staining reagent, a first IHC staining reagent, a second IHC staining reagent, etc. to produce a glass slide, which can then be digitally imaged to generate multiple pathology slide images corresponding to the pathology specimen.

[0046] According to an embodiment, the neural network learning system 100 can receive the first pathology slide image through the Nth pathology slide image corresponding to a specified pathology specimen from an external terminal, and can also acquire the first pathology slide image through the Nth pathology slide image from a memory device in which the first pathology slide image through the Nth pathology slide image corresponding to the pathology specimen are pre-stored.

[0047] Meanwhile, the neural network learning system 100 can generate m-th learning data based on the first pathology slide image through the N-th pathology slide image (S120).

[0048] In one embodiment, the neural network training system 100 can generate the mth training data by a channel superposition method, that is, the neural network training system 100 can convert the first pathology slide image to the Nth pathology slide image into one multi-channel image by channel superposition, and the mth training data may include the multi-channel image.

[0049] Fig. 3 shows a multi-channel image generated from multiple pathology slide images represented by the RGB color model by channel superposition. Fig. 3 shows the conversion of slide images of four consecutive sections extracted from a single pathology specimen into a multi-channel image.

[0050] Referring to FIG. 3, multi-channel image 20 may be made up of 12 channels, which is the product of the number of slide images, four, and the number of channels, three, that make up each slide image.

[0051] The first channel 21-1 may be comprised of an R channel value for each pixel of the first slide image, the second channel 21-2 may be comprised of a G channel value for each pixel of the first slide image, the third channel 21-3 may be comprised of a B channel value for each pixel of the first slide image, the fourth channel 22-1 may be comprised of an R channel value for each pixel of the second slide image, the fifth channel 22-2 may be comprised of a G channel value for each pixel of the second slide image, the sixth channel 22-3 may be comprised of a B channel value for each pixel of the second slide image, and the third channel 22-4 may be comprised of a G channel value for each pixel of the second slide image. The seventh channel 23-1 may be composed of the R channel values ​​of each pixel of the third slide image, the eighth channel 23-2 may be composed of the G channel values ​​of each pixel of the third slide image, the ninth channel 23-3 may be composed of the B channel values ​​of each pixel of the first slide image, the tenth channel 24-1 may be composed of the R channel values ​​of each pixel of the fourth slide image, the eleventh channel 24-2 may be composed of the G channel values ​​of each pixel of the fourth slide image, and the twelfth channel 24-3 may be composed of the B channel values ​​of each pixel of the fourth slide image.

[0052] Meanwhile, the position and direction of each pathology slide image extracted from a single pathology specimen may be slightly shifted during the process of producing the corresponding slide image. In this case, a process of aligning each pathology slide image should be performed before performing channel superposition of each pathology slide image, and a flowchart of this case is shown in FIG.

[0053] Referring to FIG. 4, the neural network learning system 100 can identify tissue regions present in the first pathology slide image through the Nth pathology slide image (S121).

[0054] There may be various methods by which the neural network training system 100 identifies a tissue region from an image. In one embodiment, when information on a tissue region is pre-annotated in a slide image, the neural network training system 100 may identify the tissue region using the pre-annotated information. Alternatively, the neural network training system 100 may identify the tissue region using a pre-trained neural network for determining a tissue region. Alternatively, the tissue region may be identified by various known methods.

[0055] Meanwhile, the neural network learning system 100 can align the first through Nth pathology slide images such that the positions and shapes of the tissue regions present in each of the first through Nth pathology slide images match (S122). In one embodiment, the neural network learning system 100 can repeatedly perform the process of aligning the (j-1)th pathology slide image and the jth pathology slide image such that the positions and shapes of the tissue regions in the (j-1)th pathology slide image and the tissue regions in the jth pathology slide image match for all natural numbers j, 2<=j<=N.

[0056] Image matching is a technique used in the field of computer vision, and refers to a processing technique that transforms different images to display them in a single coordinate system. Methods for matching two images include a method of transforming the images so that the contours of tissue regions contained in the images are as similar as possible, or a method of transforming the images so that feature points within the tissue regions match as much as possible. Specifically, a matching algorithm based on the similarity between two images measured by SIFT (Scale-Invariant Feature Transform), SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross Correlation), etc. can be used.

[0057] Meanwhile, in the process of matching the first pathology slide image through the matched Nth pathology slide image, a transformation relationship between each matched pathology slide can be calculated. That is, the neural network learning system 100 can calculate a transformation relationship corresponding to the ith pathology slide image for all natural numbers i, where 1<=i<=N. Here, the transformation relationship corresponding to the ith pathology slide image is a transformation relationship between the ith pathology slide image and the corresponding matched ith pathology slide image.

[0058] Next, referring to FIG. 4, the neural network learning system 100 can convert the aligned first pathology slide image to the aligned Nth pathology slide image into one multi-channel image by channel superposition (S123), which is similar to what has been described above with reference to FIG. 3, and therefore will not be described in detail.

[0059] In one embodiment, each pathology slide image may be pre-annotated with a lesion area, and in such a case, the training data may further include the pre-annotated lesion annotation area.

[0060] In contrast, to explain in more detail, the neural network learning system 100 can deform the lesion annotation area assigned to the jth pathology slide image using a transformation relationship corresponding to the jth pathology slide image for all natural numbers j, 1<=j<=N, and convert the lesion annotation area of ​​the deformed first pathology slide image to the lesion annotation area of ​​the deformed Nth pathology slide image into one multi-channel lesion annotation area by channel superposition, and the mth learning data may further include a multi-channel lesion annotation area.

[0061] On the other hand, if there is diagnostic information, prognostic information, and / or response information to a particular treatment method for the pathological specimen corresponding to the mth training data, the neural network training system 100 can set this as the label of the mth training data.

[0062] When a training data set including M individual training data is generated by the above-described method, the neural network training system 100 can train the neural network 300 by inputting the generated training data set into the input layer of the neural network 300 (S130 in FIG. 2).

[0063] The neural network learning method according to the technical idea of ​​the present invention can obtain an effect similar to that of multiplex IHC by superimposing and utilizing multiple pathology slide images generated using a widely used method instead of a high-cost, uncommon method such as multiplex IHC, and can improve the accuracy of tumor microenvironment analysis by eliminating the possibility of errors that may occur in the process of separating the location information of each target from the results of multiplex IHC through color filtering.

[0064] 5 is a flowchart showing an example of a method for providing a determination result for a pathological specimen according to an embodiment of the present invention. The method for providing a determination result for a pathological specimen according to FIG. 5 can be performed by a determination result providing system 200, which can store an artificial neural network 300 trained by the neural network training system 100.

[0065] 5, the judgment result providing system 200 can obtain the first to Nth judgment target pathology slide images of a predetermined judgment target pathology specimen (S210). At this time, the first to Nth judgment target pathology slide images are pathology slide images in which consecutive sections of the judgment target pathology specimen are stained with different staining reagents.

[0066] The judgment result providing system 200 can generate input data based on the first to Nth pathology slide images of the specimen to be judged (S220). The process of generating input data corresponding to the first to Nth pathology slide images of the specimen to be judged is very similar to the process described with reference to Figures 3 and 4, so a separate description will be omitted.

[0067] The determination result providing system 200 can input input data to the artificial neural network 300 and output a determination result for the pathological specimen to be determined based on the result output by the artificial neural network (S230).

[0068] FIG. 6 is a diagram showing a schematic configuration of an artificial neural network learning system 100 according to an embodiment of the present invention, and FIG. 7 is a diagram showing a schematic configuration of a judgment result providing system 200 according to an embodiment of the present invention.

[0069] The artificial neural network learning system 100 and the judgment result providing system 200 may refer to a logical configuration having hardware resources and / or software necessary to realize the technical idea of ​​the present invention, and do not necessarily refer to one physical component or one device. In other words, the artificial neural network learning system 100 and the judgment result providing system 200 may refer to a logical combination of hardware and / or software provided to realize the technical idea of ​​the present invention, and may be realized as a set of logical configurations that are installed in devices separated from each other and perform their respective functions to realize the technical idea of ​​the present invention, as necessary. The artificial neural network learning system 100 and the judgment result providing system 200 may also refer to a set of configurations that are realized separately for each function or role to realize the technical idea of ​​the present invention. The components of the artificial neural network learning system 100 and the judgment result providing system 200 may be located in different physical devices or in the same physical device. In addition, depending on the embodiment, the combination of software and / or hardware constituting each component of the artificial neural network learning system 100 and the judgment result providing system 200 may also be located in different physical devices, and the components located in the different physical devices may be organically combined with each other to realize each module.

[0070] In addition, in this specification, a module may refer to a functional and structural combination of hardware for carrying out the technical idea of ​​the present invention and software for driving the hardware. For example, a module may refer to a logical unit of a certain code and a hardware resource for executing the certain code, and does not necessarily refer to a physically connected code or a type of hardware, which can be easily inferred by an average expert in the technical field of the present invention.

[0071] 6, the artificial neural network training system 100 may include a storage module 110, an acquisition module 120, a generation module 130, and a training module 140. Depending on the embodiment of the present invention, some of the above components may not necessarily be essential components for realizing the present invention, and it goes without saying that the artificial neural network training system 100 may include more components than these depending on the embodiment. For example, the artificial neural network training system 100 may further include a communication module (not shown) for communicating with an external device, and a control module (not shown) for controlling the components and resources of the artificial neural network training system 100.

[0072] The storage module 110 is capable of storing the artificial neural network 40 to be trained.

[0073] The acquisition module 120 can acquire a first pathology slide image through an Nth pathology slide image in which consecutive sections for each single pathology specimen are stained with different staining reagents.

[0074] The generation module 130 can generate individual training data based on the first pathology slide image through the Nth pathology slide image, and can configure a training data set including a plurality of individual training data.

[0075] The training module 140 can train the artificial neural network 300 based on a training data set.

[0076] 7, the judgment result providing system 200 may include a storage module 210, an acquisition module 220, a generation module 230, and a judgment module 240. Some of the above-mentioned components may not necessarily be essential components for realizing the present invention, and the judgment result providing system 200 may include more components depending on the embodiment. For example, the judgment result providing system 200 may further include a communication module (not shown) for communicating with the three-axis vibration sensor 20 and a control module (not shown) for controlling the components and resources of the judgment result providing system 200.

[0077] The storage module 210 can store the trained artificial neural network 40 .

[0078] The acquisition module 220 can acquire a first to an Nth target pathology slide image, in which consecutive sections of a given target pathology specimen are stained with different staining reagents.

[0079] The generation module 230 can generate input data based on the first to Nth target pathology slide images.

[0080] The judgment module 240 inputs the input data to the artificial neural network, and can make a judgment on the specimen to be judged based on the predicted value output from the artificial neural network 40.

[0081] Meanwhile, according to an example of an embodiment, the artificial neural network learning system 100 and the judgment result providing system 200 may include a processor and a memory for storing a program executed by the processor. The processor may include a single-core CPU or a multi-core CPU. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state memory devices. Access to the memory by the processor and other components may be controlled by a memory controller.

[0082] Meanwhile, the method according to the embodiment of the present invention may be realized in the form of computer readable program instructions and stored in a computer readable recording medium, and the control program and the target program according to the embodiment of the present invention may also be stored in a computer readable recording medium. The computer readable recording medium includes any kind of recording device in which data that can be read by a computer system is stored.

[0083] The program instructions contained on the recording media may be those specially designed and constructed for the purposes of the present invention, or they may be of the type commonly known and available to those having skill in the software arts.

[0084] Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, etc. Also, computer-readable recording media can be distributed among computer systems connected via a network to store and execute computer-readable code in a distributed manner.

[0085] Examples of program instructions include not only machine code, such as produced by a compiler, but also high level language codes that may be executed by a device for processing information electronically, such as a computer, using an interpreter or the like.

[0086] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0087] The above description of the present invention is for illustrative purposes only, and a person having ordinary skill in the art to which the present invention pertains can easily understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. For example, each component described in a single form can be implemented in a distributed manner, and similarly, components described as being distributed can be implemented in a combined form.

[0088] The scope of the present invention is indicated by the claims set forth below rather than by the above detailed description, and all modifications and variations that fall within the meaning and scope of the claims and their equivalent concepts should be construed as being included within the scope of the present invention. [Industrial Applicability]

[0089] The present invention can be used in a method for training an artificial neural network to provide a judgment result for a pathological specimen, and in a computing system for carrying out this method.

Claims

1. 1. A method of training an artificial neural network, comprising: A neural network training system generates a training data set including M individual training data (where M is a natural number equal to or greater than 2); said neural network training system training said artificial neural network based on said training data set; Including, The step of generating a learning data set including the M individual learning data includes a step of generating m-th learning data included in the learning data set for all m, where 1<=m<=M; The step of generating m training data includes: A step of acquiring a first pathology slide image through an Nth pathology slide image (where N is a natural number equal to or greater than 2) which are pathology slide images of serial sections of a single pathology specimen stained with different staining reagents; generating the mth learning data based on the first pathology slide image through the Nth pathology slide image; Including, The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: converting the first pathology slide image through the Nth pathology slide image into one multi-channel image by channel stacking; The mth training data includes the multi-channel image. How to train artificial neural networks.

2. The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: Identifying a biological tissue region present in the first pathology slide image through the Nth pathology slide image; aligning the first through Nth pathology slide images such that positions and morphologies of biological tissue regions present in each of the first through Nth pathology slide images coincide with each other; converting the first aligned pathology slide image through the Nth aligned pathology slide image into one multi-channel image by channel superposition; Including, 2. The method of claim 1, wherein said m training data comprises said multi-channel images.

3. The step of aligning the first to Nth pathology slide images so that positions and shapes of biological tissue regions present in each of the first to Nth pathology slide images coincide with each other includes: calculating, for all natural numbers i, 1<=i<=N, a transformation relationship corresponding to an i-th pathology slide image, between the i-th pathology slide image and its corresponding aligned i-th pathology slide image; The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: For all natural numbers j, 1<=j<=N, transforming a lesion annotation region assigned to a j-th pathology slide image using a transformation relationship corresponding to the j-th pathology slide image; converting the lesion annotation area of ​​the first transformed pathology slide image through the lesion annotation area of ​​the Nth transformed pathology slide image into one multi-channel lesion annotation area by channel superposition; Further comprising: The method of claim 2 , wherein the m training data further comprises the multi-channel lesion annotation regions.

4. A method for providing a judgment result for a predetermined pathological specimen to be judged through an artificial neural network trained by the method for training an artificial neural network according to claim 1, comprising: A computing system acquires a first pathology slide image to an Nth pathology slide image (where N is a natural number equal to or greater than 2) which are pathology slide images in which consecutive sections of the pathology specimen to be judged are stained with different staining reagents; converting the first to Nth pathology slide images into one multi-channel image by channel superposition; The computing system outputs a judgment result for the pathological specimen judged by the artificial neural network based on the first pathological slide image to the Nth pathological slide image to be judged, which are converted into one multi-channel image; A method comprising:

5. A computer program recorded on a recording medium for being installed in a data processing device and for carrying out the method according to any one of claims 1 to 4.

6. A computer-readable recording medium having a computer program recorded thereon for carrying out the method according to any one of claims 1 to 4.

7. 1. An artificial neural network learning system, comprising: A processor and a memory for storing a computer program, The computer program, when executed by the processor, causes the artificial neural network training system to perform a method for training an artificial neural network, The artificial neural network learning system comprises: A neural network training system generates a training data set including M individual training data (where M is a natural number equal to or greater than 2); said neural network training system training said artificial neural network based on said training data set; Including, The step of generating a learning data set including the M individual learning data includes a step of generating m-th learning data included in the learning data set for all m, where 1<=m<=M; The step of generating m training data includes: A step of acquiring a first pathology slide image through an Nth pathology slide image (where N is a natural number equal to or greater than 2) which are pathology slide images of consecutive sections of a single pathology specimen stained with different staining reagents; generating the mth learning data based on the first pathology slide image through the Nth pathology slide image; The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: converting the first pathology slide image through the Nth pathology slide image into one multi-channel image by channel stacking; The mth learning data is the multi-channel image An artificial neural network learning system comprising:

8. The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: converting the first pathology slide image through the Nth pathology slide image into one multi-channel image by channel superposition; 8. The artificial neural network training system of claim 7, wherein said m training data comprises said multi-channel images.

9. The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: Identifying a biological tissue region present in the first pathology slide image through the Nth pathology slide image; aligning the first through Nth pathology slide images such that positions and morphologies of biological tissue regions present in each of the first through Nth pathology slide images coincide with each other; converting the first aligned pathology slide image through the Nth aligned pathology slide image into one multi-channel image by channel superposition; Including, 8. The artificial neural network training system of claim 7, wherein said m training data comprises said multi-channel images.

10. The step of aligning the first to Nth pathology slide images so that positions and shapes of biological tissue regions present in each of the first to Nth pathology slide images coincide with each other includes: calculating, for all natural numbers i, 1<=i<=N, a transformation relationship corresponding to an i-th pathology slide image, between the i-th pathology slide image and its corresponding aligned i-th pathology slide image; The step of generating the mth learning data based on the first pathology slide image to the Nth pathology slide image includes: For all natural numbers j, 1<=j<=N, transforming a lesion annotation region assigned to a j-th pathology slide image using a transformation relationship corresponding to the j-th pathology slide image; converting the lesion annotation area of ​​the first transformed pathology slide image through the lesion annotation area of ​​the Nth transformed pathology slide image into one multi-channel lesion annotation area by channel superposition; Further comprising:

10. The artificial neural network training system of claim 9, wherein the m training data further comprises the multi-channel lesion annotation regions.

11. A system for providing judgment results for pathology samples, comprising: A processor and a memory for storing a computer program, The computer program, when executed by the processor, causes the pathology specimen determination result providing system to execute a method for providing a pathology specimen determination result via an artificial neural network trained by the artificial neural network training method according to claim 1, The method for providing a judgment result includes: A step of acquiring a first pathology slide image to an Nth pathology slide image (where N is a natural number equal to or greater than 2) which are pathology slide images in which consecutive sections of a predetermined pathology specimen to be judged are stained with different staining reagents; converting the first to Nth pathology slide images into one multi-channel image by channel superposition; a step of outputting a judgment result for the pathological specimen judged by the artificial neural network based on the first pathological slide image to the Nth pathological slide image to be judged, the pathological specimen being judged based on the first pathological slide image to be judged that have been converted into one multi-channel image; A system for providing judgment results for pathology samples, comprising:

Citation Information

Patent Citations

  • Method and system for detecting pathological abnormalities in digital pathology images and method for annotating tissue slides

    JP2019525151A

  • Computational pathology system and method for early cancer prediction

    JP6650453B2