Image analysis method, image analysis device, image analysis program, learned deep learning algorithm and manufacturing method of learned deep learning algorithm

Abstract

The invention relates to an image analysis method, an image analysis device, an image analysis program, a learned deep learning algorithm and a manufacturing method thereof. Specifically, the present invention provides an image analysis method for generating data showing a tumorigenesis state with respect to an image of a tissue or a cell. This image analysis method analyzes an image of a tissue or a cell using a deep learning algorithm of a neural network structure, and generates analysis data from an image to be analyzed including the tissue or the cell to be analyzed, and inputs the analysis data to the deep learning algorithm. Data for displaying the tumorigenesis state of the tissue or the cell in the analysis target image is generated by a deep learning algorithm.

Description

[0001] This divisional application is a divisional application based on the Chinese patent application with application number 201811371880.8, application date November 16, 2018, and invention name “Image analysis method, device and program, and learned deep learning algorithm and its manufacturing method”. [Technical field]

[0002] The present invention relates to an image analysis method, an image analysis device, a program, a method for manufacturing a learned deep learning algorithm, and a learned deep learning algorithm. More specifically, it relates to an image analysis method, an image analysis device, a program, a method for manufacturing a learned deep learning algorithm, and a learned deep learning algorithm for generating data showing a tumor state at an arbitrary position of an image of a tissue or cell. [Background technology]

[0003] Patent document 1 discloses an image diagnosis support device for classifying tissue images into four groups of normal, benign tumor, precancerous state and cancerous state in pathological tissue images. The image classification means extracts the attention area from the image data, calculates the feature quantity showing the characteristics of the attention area, and classifies the groups based on the calculated feature quantity. The feature quantity is the density of blocks per unit area in the cell nucleus, the density of the block area, the area of ​​the block, the thickness of the block, and the length of the block. The image judgment means learns the relationship between such feature quantity and judgment result, and makes judgment based on the learned learning parameters. The learning is performed by machine learning using a learning algorithm such as a support vector machine.

[0004] [Prior art literature]

[0005] [Patent Literature]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2010-203949

[0007] [Summary of the Invention]

[0008] [Problems to be solved by the invention]

[0009] When determining whether a diagnosis is a malignant tumor or not, a histopathological diagnosis using a pathological tissue specimen is performed. In addition, histopathological diagnosis is often performed as an intraoperative rapid diagnosis to determine the resection site of tissue containing a malignant tumor during surgery. During the operation, the patient's affected part is cut open and the histopathological diagnosis is used to determine whether the tumor is malignant, whether there is a tumor remaining at the end of the resected tissue, whether there is lymph node metastasis, etc. The result of the intraoperative rapid diagnosis determines the direction of the subsequent surgery for the patient waiting.

[0010] Pathological histological diagnosis is made by physicians, especially pathologists, by observing tissue specimens under a microscope or the like. However, in order to make an accurate and definitive diagnosis by observation of tissue specimens, it is necessary for skilled pathologists to repeat the observation of tissue specimens of various cases over a long period of time, and the training of pathologists also requires a lot of time.

[0011] The shortage of pathologists is serious. As a result of the shortage of pathologists, the definitive diagnosis of malignant tumors of patients is delayed, the start of treatment is delayed, or there is a danger that treatment will be started without waiting for a definitive diagnosis. In addition, since both the usual histological diagnosis and rapid intraoperative diagnosis are concentrated in a small number of pathologists, the workload of one pathologist has become huge, and the labor status of the pathologists themselves has also become a problem. However, there is no solution to this problem yet.

[0012] Therefore, the device can support pathological histological diagnosis, and in particular, the closer the diagnosis is to the judgment by the human eye, the greater the contribution to eliminating the shortage of pathologists and improving the labor status of pathologists.

[0013] In terms of the device supporting pathological histological diagnosis, the invention described in the above-mentioned Patent Document 1 performs pathological judgment of specimen tissue based on image analysis by machine learning. In this method, it is necessary to generate feature quantities by human hands. In the method of generating feature quantities by human hands, there is a problem that the human strength has a great influence on the performance of image analysis.

[0014] For example, in histological diagnosis or cytological diagnosis using a microscope, one of the objects of observation is the state of the cell nucleus, and benign tumors and malignant tumors are distinguished from each other based on the size or shape of the cell nucleus and the sequence state of multiple cell nuclei. Therefore, in pathological histological diagnosis, it is very important to be able to extract the cell nucleus with high accuracy, which is the basis of histological diagnosis and cytological diagnosis. The extracted nuclear area can be automatically detected whether it is a non-tumor cell or a tumor cell, which helps the diagnosis.

[0015] The present invention aims to provide an image analysis method, an image analysis device, a program, a method for manufacturing a learned deep learning algorithm, and a learned deep learning algorithm for generating data showing the tumor state from an image of tissue or cells.

[0016]

Methods for solving the problem

[0017] One embodiment of the present invention is an image analysis method. In this embodiment, the image analysis method is an image analysis method for analyzing an image of a tissue or cell using a deep learning algorithm (60) having a neural network structure, generating analysis data (80) (S21 to S23) from an analysis target image (78) containing the tissue or cell to be analyzed, inputting the analysis data (80) (S24) to the deep learning algorithm (60), and generating data (82, 83) (S25 to S28) showing the tumor state of the tissue or cell in the analysis target image (78) by the deep learning algorithm (60). Thus, it becomes possible to generate data showing the tumor state of the tissue or cell in the analysis target image.

[0018] The image to be analyzed is an image of a specimen for histological diagnosis. The image to be analyzed (78) preferably contains a hue consisting of one primary color or a hue (R, G, B) consisting of a combination of two or more primary colors.

[0019] The image to be analyzed is an image of a specimen for cytological diagnosis. The image to be analyzed (78) preferably contains a hue consisting of one primary color, or contains a hue (R, G, B) consisting of a combination of two or more primary colors.

[0020] The data (82, 83) indicating the tumorigenic state are preferably data for distinguishing between a region indicating the nucleus of a tumor cell and other regions.

[0021] The data (82, 83) indicating the tumorigenic state are preferably data indicating the boundary between the nucleus region of the tumor cell and the other regions.

[0022] The deep learning algorithm (60) preferably determines whether it is a tumor cell nucleus or a non-tumor cell nucleus.

[0023] Preferably, a plurality of analysis data (80) are generated for each region having a designated number of pixels for one analysis target image (78). This makes it possible to improve the discrimination accuracy of the neural network (60).

[0024] The analysis data (80) is generated for each area of ​​a designated number of pixels with a designated pixel as the center and including surrounding pixels, and the deep learning algorithm (60) preferably generates a label indicating the tumor state for the designated pixel in the input analysis data (80). This makes it possible to improve the discrimination accuracy of the neural network (60).

[0025] The number of nodes of the input layer (60a) of the neural network (60) preferably corresponds to the product of the number of designated pixels of the analysis data (80) and the number of combined primary colors, thereby making it possible to improve the discrimination accuracy of the neural network (60).

[0026] The specimen is a stained specimen, and the analysis target image (78) is preferably an image of the stained specimen photographed under a bright field of a microscope.

[0027] The training data (74C, 74NC) used in the learning of the deep learning algorithm (60) are bright field images (70C, 70NC) of a specimen of a tumor-containing tissue sample collected from an individual or a specimen of a tumor cell-containing sample collected from an individual, which are produced by staining for bright field observation and photographed under the bright field of a microscope, and fluorescent images (71C, 71NC) of cell nuclei, which are produced by staining a corresponding specimen or the same specimen by fluorescent nuclear staining and photographed under fluorescent observation using a fluorescent microscope. The fluorescent images (71C, 71NC) are preferably generated based on the position of the fluorescent images (71C, 71NC) within the specimen corresponding to the position of the acquired bright field image (70C, 70NC) within the specimen.

[0028] As staining for bright field observation, hematoxylin is preferably used for nuclear staining.

[0029] When the specimen is a tissue sample, the staining for bright field observation is preferably hematoxylin-eosin staining, and when the specimen is a cell-containing sample, the staining for bright field observation is preferably Papanicolaou staining.

[0030] The training data (74C, 74NC) preferably includes label values ​​showing the tumorigenic state of cells extracted from the bright field images (70C, 70NC) and the fluorescent images (71C, 71NC), thereby enabling the neural network (50) to learn the label values ​​showing the tumorigenic state of cells.

[0031] The training data (74C, 74NC) preferably includes a label value for each pixel of the bright field image (70C, 70NC), thereby enabling the neural network (50) to learn a label value representing the tumorigenic state of the cell.

[0032] The training data (74C, 74NC) is preferably generated for each region of a specified number of pixels in the bright field image (70C, 70NC), thereby enabling the neural network (50) to learn a label value representing the tumorigenic state of a cell with high accuracy.

[0033] The deep learning algorithm (60) preferably classifies the analysis data (80) into classes indicating the tumorigenic state of cells included in the analysis target image (78). This makes it possible to classify the tumorigenic state at any position of the analysis target image containing cells.

[0034] The output layer (60b) of the neural network (60) is preferably a node using a Soft Max function as an activation function. Thus, the neural network (60) can classify any position of the analysis target image containing cells into a limited number of classes.

[0035] The deep learning algorithm (60) preferably generates data (82) showing the tumor state contained in the analysis target image (78) for each unit pixel every time the analysis data (80) is input. Thus, it becomes possible to classify the tumor state for each unit pixel (1 pixel) of the analysis target image containing cells.

[0036] The deep learning algorithm (60) is preferably generated according to the type of tissue sample or the type of cell-containing sample. This makes it possible to use the deep learning algorithm (60) according to the type of the cell-containing analysis target image, thereby improving the discrimination accuracy of the neural network (60).

[0037] Furthermore, it is preferred that the analysis data (80) is processed using a deep learning algorithm (60) corresponding to the type of sample selected from a plurality of deep learning algorithms (60) according to the type of tissue sample or the type of cell-containing sample. This makes it possible to use a deep learning algorithm (60) according to the type of the cell-containing analysis target image, thereby improving the discrimination accuracy of the neural network (60).

[0038] One embodiment of the present invention is an image analysis device. In this embodiment, the image analysis device (200A) is an image analysis device that analyzes an image of a tissue or cell using a deep learning algorithm (60) having a neural network structure, and includes a processing unit (20A) that generates analysis data (80) from an analysis target image (78) containing the tissue or cell to be analyzed, inputs the analysis data (80) to the deep learning algorithm (60), and generates data (82, 83) showing the tumor state of the tissue or cell in the analysis target image (78) by the deep learning algorithm (60). Thus, it becomes possible to generate data showing the tumor state of the tissue or cell in the analysis target image.

[0039] One embodiment of the present invention is a computer program. In this embodiment, the computer program is a computer program for analyzing an image of a tissue or cell using a deep learning algorithm (60) having a neural network structure, and is a program for causing a computer to execute the following processing: a process of generating analysis data (80) from an analysis target image (78) containing the tissue or cell to be analyzed, a process of inputting the analysis data (80) to the deep learning algorithm (60), and a process of generating data (82, 83) showing the tumor state of the tissue or cell in the analysis target image (78) by the deep learning algorithm (60). Thus, it becomes possible to generate data showing the tumor state of the tissue or cell in the analysis target image.

[0040] One embodiment of the present invention is a manufacturing method after learning a deep learning algorithm. In this embodiment, the manufacturing method after learning a deep learning algorithm (60) includes: a first acquisition step (S10) of acquiring first training data (72r, 72g, 72b) corresponding to a first training image (70C) photographed for tumor tissue or tumor cells, a second acquisition step (S13a) of acquiring second training data (73C) corresponding to a second training image (71C) showing a nuclear region of a tumor cell in the first training image (70C), and a third acquisition step (S13b) of acquiring third training data (72r, 72g, 72b) corresponding to a third training image (70NC) photographed for non-tumor tissue or non-tumor cells. 2g, 72b), a fourth acquisition step (S13b) of acquiring fourth training data (73NC) corresponding to a fourth training image (71NC) showing a nuclear region of non-tumor cells in the third training image (70NC), a learning step (S14 to S20) of causing the neural network (50) to learn the relationship between the first training data (72r, 72g, 72b) and the second training data (73C), and a learning step (S14 to S20) of causing the neural network (50) to learn the relationship between the third training data (72r, 72g, 72b) and the fourth training data (73NC). Thus, it becomes possible to produce a deep learning algorithm for generating data showing the tumor state of tissues or cells in the image to be analyzed.

[0041] Preferably, the first training data (72r, 72g, 72b) and the third training data (72r, 72g, 72b) are used as the input layer (50a) of the neural network (50), and the second training data (73C) and the fourth training data (73NC) are used as the output layer (50b) of the neural network (50) corresponding to the first training data (72r, 72g, 72b) and the third training data (72r, 72g, 72b) respectively.

[0042] The method preferably includes a step (S10) of generating first training data (72r, 72g, 72b) from the first training image (70C) before the first acquisition step, a step (S13a) of generating second training data (73C) from the second training image (71C) before the second acquisition step, a step (S10) of generating third training data (72r, 72g, 72b) from the third training image (70NC) before the third acquisition step, and a step (S13b) of generating fourth training data (73NC) from the fourth training image (71NC) before the fourth acquisition step. Thus, it becomes possible to produce a deep learning algorithm for generating data showing the tumor state of tissue or cells in the image to be analyzed.

[0043] The first training image (70C) is a bright field image (70C) of tumor cells obtained by photographing a stained image of a specimen prepared by staining a tissue sample collected from an individual or a cell-containing specimen collected from an individual under bright field observation under a microscope. The second training image (71C) is a fluorescent image (71C) obtained by photographing a stained image of a specimen prepared by staining a tissue sample or a cell-containing specimen for bright field observation under a microscope, wherein the position of the fluorescent image (71C) in the specimen corresponds to the position of the first training image (70C) in the specimen. The third training image (71C) is a fluorescent image (71C) obtained by photographing a stained image of a specimen prepared by staining a tissue sample or a cell-containing specimen for fluorescent nuclear staining under fluorescent observation under a microscope. The image (70NC) is a bright field image (70NC) containing non-tumor cells, which is obtained by photographing a stained image of a specimen produced by staining a tissue sample collected from an individual or a cell-containing specimen collected from an individual under bright field observation under a microscope. The fourth training image (71NC) is a fluorescent image (71NC) obtained by photographing a stained image of a specimen produced by staining a tissue sample or a cell-containing specimen for bright field observation under a microscope. It is preferably a fluorescent image (71NC) in which the position within the specimen of the fluorescent image (71NC) corresponds to the position within the specimen of the third training image (70NC) obtained.

[0044] One embodiment of the present invention is a deep learning algorithm (60) that has been learned. In this embodiment, the deep learning algorithm (60) that has been learned is a deep learning algorithm (60) that uses the first training data (72r, 72g, 72b) and the third training data (72r, 72g, 72b) as the input layer (50a) of the neural network (50), and uses the second training data (73C) and the fourth training data (73NC) as the output layer (50b) of the neural network (50) corresponding to the first training data (72r, 72g, 72b) and the third training data (72r, 72g, 72b) respectively. 0), wherein the first training data (72r, 72g, 72b) is generated from a first training image (70C) of tumor tissue or tumor cells, the second training data (73C) is displayed in the nuclear region of the tumor cells in the first training image (70C), the third training data (72r, 72g, 72b) is generated from a third training image (70NC) of non-tumor tissue or non-tumor cells, and the fourth training data (73NC) is displayed in the nuclear region of the non-tumor cells in the third training image (70NC).

[0045] [Effects of the invention]

[0046] According to the present invention, data showing the tumorigenic state can be generated for any position in an image of tissue or cells.

[0047] [Brief description of the attached drawings]

[0048]

Figure 1

[0049]

Figure 2

[0050]

Figure 3

[0051]

Figure 4

[0052]

Figure 5

[0053]

Figure 6

[0054]

Figure 7

[0055]

Figure 8

[0056]

Fig. 9

[0057]

Fig.10

[0058]

Fig.11

[0059]

Fig.12

[0060]

Fig.13

[0061]

Fig.14

[0062]

Fig.15

[0063]

Fig.16

[0064]

Fig.17

[0065]

Fig.18

[0066]

Fig.19

[0067]

Fig. 20

[0068]

Fig.21

[0069]

Fig. 22

[0070]

Fig.23

[0071] Hereinafter, the outline and embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and drawings, the same reference numerals are used to represent the same or similar components, and thus the description of the same or similar components will be omitted.

[0072] The present invention relates to an image analysis method for analyzing images of tissues or cells, which uses a deep learning algorithm with a neural network structure, preferably a folded neural network structure.

[0073] In the present invention, the image of a tissue or a cell is an image obtained from a specimen of a tissue sample or a specimen of a cell-containing sample. The specimen of a tissue sample or a specimen of a cell-containing sample is collected from an individual. The individual is not particularly limited, preferably a mammal, and more preferably a human. When collecting samples from the above-mentioned individual, it does not matter whether the individual is alive or dead. The above-mentioned tissue is not limited as long as it exists in the individual. As tissues collected from the above-mentioned individual, surgical resection tissues, biopsy tissues, etc. can be cited. The above-mentioned cell-containing sample is not limited as long as it is collected from an individual. For example, expectoration, pleural effusion, ascites, urine, cerebrospinal fluid, bone marrow, blood, cyst fluid, etc. can be cited. The above-mentioned tissue and the above-mentioned cell-containing sample may contain tumor cells. The above-mentioned tumor may also be any one of benign and malignant. The above-mentioned tumor may also be any one of epithelial and non-epithelial. As the above-mentioned tumor, a malignant epithelial tumor is preferably used. The malignant tumor is not particularly limited, and examples of the malignant tumor include respiratory malignant tumors arising from the trachea, bronchus or lung, etc.; digestive tract malignant tumors arising from the pharynx, esophagus, stomach, duodenum, jejunum, ileum, cecum, appendix, ascending colon, transverse colon, sigmoid colon, rectum or anus, etc.; liver cancer; pancreatic cancer; urinary tract malignant tumors arising from the bladder, ureter or kidney; female genital malignant tumors arising from the ovary, fallopian tube and uterus, etc.; breast cancer; prostate cancer; skin cancer; endocrine malignant tumors of the hypothalamus, pituitary gland, thyroid gland, parathyroid gland, adrenal gland, etc.; central nervous system malignant tumors; solid tumors such as malignant tumors arising from bone and soft tissues. More preferably, there are respiratory epithelial malignancies such as lung cancer (squamous cell carcinoma, small cell carcinoma, large cell carcinoma, adenocarcinoma), gastrointestinal epithelial malignancies such as gastric cancer, duodenal cancer, colorectal cancer (sigmoid colon cancer, rectal cancer, etc.), liver cancer, pancreatic cancer, bladder cancer, thyroid cancer, ovarian cancer, breast cancer, and prostate cancer. Most preferably, gastric cancer.

[0074] The above-mentioned specimen refers to a state in which the above-mentioned tissue sample or cell-containing sample is processed in a manner that can be observed by a microscope, etc., such as a specimen for a microscope. The above-mentioned specimen can be prepared according to a known method. For example, in the case of a tissue specimen, after collecting the tissue from the above-mentioned individual, the tissue is fixed with a specified fixative (formalin fixation, etc.), the fixed tissue is paraffin-embedded, and the paraffin-embedded tissue is thinly sliced. The thin slices are placed on a glass slide. The glass slide carrying the slice is stained for observation with an optical microscope, that is, for bright field observation, and a specified sealing treatment is performed to complete the specimen. A typical example of a tissue specimen is a specimen for histological diagnosis (pathological specimen), and the staining is hematoxylin-eosin (HE) staining.

[0075] For example, in the case of a cell-containing sample, the cells in the sample are attached to a slide by centrifugation, smearing, etc., fixed with a specified fixative (ethanol, etc.), stained for bright field observation, and a specified sealing treatment is performed to complete the sample. A typical example of a cell-containing sample is a cytological diagnosis specimen (cytological diagnosis specimen), and the staining is Papanicolaou staining. The cytological diagnosis specimen also includes a stamp specimen of a tissue collected for the tissue specimen.

[0076] HE staining, Papanicolaou staining and nuclear staining are all hematoxylin. Hematoxylin is widely used as a nuclear staining agent in tissue cell staining (for example, immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to the whole specimen using such hematoxylin in nuclear staining.

[0077] In the above-mentioned image analysis, a deep learning algorithm that has been trained using four types of training images is used. The above-mentioned image analysis generates analysis data from the analysis object image containing the tissue or cells of the analysis object obtained from the above-mentioned specimen. The above-mentioned analysis data is input into the above-mentioned deep learning algorithm to generate data showing the tumor state of the cells present in the above-mentioned analysis object image. The tumor state means that the tissue at the position of the focus pixel of the discrimination object is a tumor. Preferably, the tumor state means whether the tissue at the position of the focus pixel of the discrimination object is a tumor or not. In addition, the data showing the tumor state is data showing whether the nuclear region of the cell present in the analysis object image is the nuclear region of the tumor cell or the nuclear region of the non-tumor cell. The data showing the tumor state can be a label value that can distinguish the nuclear region of the tumor cell, the nuclear region of the non-tumor cell, and the nuclear region of neither the tumor cell nor the tumor cell.

[0078] Among the four types of training images, the first training image is an image of tumor tissue or tumor cells contained in a specimen containing a tumor tissue sample collected from an individual or a specimen containing tumor cells collected from an individual. This image is obtained from a specimen stained in a manner that allows the tissue structure or cell structure to be identified by microscopic observation. The above-mentioned staining is not limited as long as it can identify the tissue structure or cell structure, and is preferably a staining for bright field observation. The above-mentioned staining for bright field observation is not limited as long as at least the cell nucleus and the parts outside the cell nucleus can be stained in a distinguishable hue. When the above-mentioned specimen is a mammalian tissue specimen, for example, HE staining can be cited. In addition, for example, when the above-mentioned specimen is a specimen containing mammalian cells, Papanicolaou staining can be cited.

[0079] The first training data and the first preliminary training data are generated from the first training image. The first training data is information about a single color image in which the hue contained in the first training image is separated for each primary color. The first preliminary training data is binary data indicating that the cells contained in the first training image are nuclear regions of tumor cells. The tissue or cells contained in the first training image contain tumor cells, for example, as determined by a physician's specimen diagnosis. As the first preliminary training data generated from the first training image, a label value indicating "region that is a tumor cell" is generated.

[0080] Among the four types of training images, the second training image is an image that shows where the nuclear region of the cell is in the first training image, that is, it shows which region of the first training image is the correct answer of "it is the nuclear region". The above-mentioned cell image is preferably an image showing the nuclear region of the tumor cell. This image is an image captured by applying fluorescent nuclear staining to selectively stain the cell nucleus to the same specimen as the specimen from which the first training image was obtained, or a specimen corresponding to the specimen from which the first training image was obtained (for example, a continuous section specimen). The above-mentioned fluorescent nuclear staining is not limited, and 4',6-diamidino-2-phenylindole (DAPI) staining can be cited. The second preliminary training data is generated from the second training image. The second preliminary training data is binarized data generated by binarizing the second training image, and shows whether it is the region of the cell nucleus or the region other than it.

[0081] The second preliminary training data and the first preliminary training data are associated with each other for each pixel, and a label value indicating "a nuclear region of a tumor cell" is further assigned to a pixel indicating a cell nuclear region in the second preliminary training data. Data in which a label value indicating "a nuclear region of a tumor cell" is assigned to a cell nuclear region in the second preliminary training data is used as the second training data.

[0082] Among the four types of training images, the third training image is an image of non-tumor tissue or non-tumor cells contained in a specimen containing a tissue sample collected from an individual or a specimen containing a cell sample collected from an individual. This image is obtained from a specimen stained in a manner that allows the tissue structure or cell structure to be identified by microscopic observation. The above-mentioned staining is not limited as long as it can identify the tissue structure or cell structure, and is preferably a staining for bright field observation. The above-mentioned staining for bright field observation is not limited as long as at least the cell nucleus and the parts outside the cell nucleus can be stained in a color-distinguished manner. When the above-mentioned specimen is a mammalian tissue specimen, for example, HE staining can be cited. In addition, for example, when the above-mentioned specimen is a specimen containing mammalian cells, Papanicolaou staining can be cited. The above-mentioned staining is preferably the same as the staining applied to the specimen when the first training image is obtained.

[0083] The third training data and the third preliminary training data are generated from the third training image. The third training data is information about a single color image in which the hue contained in the third training image is separated for each primary color. The third preliminary training data is binary data indicating that the cells contained in the third training image are regions that are non-tumor cells. The tissue or cells contained in the third training image include non-tumor cells, for example, as determined by a physician's specimen diagnosis. As the third preliminary training data generated from the third training image, a label value indicating "region that is non-tumor cell" is generated.

[0084] Among the four types of training images, the fourth training image is an image showing where the nuclear region of the cell is in the third training image, that is, an image showing which region of the third training image is the correct answer of "it is the nuclear region". The above-mentioned cell image is preferably an image showing the nuclear region of a non-tumor cell. This image is an image captured by applying fluorescent nuclear staining to selectively stain the cell nucleus to the same specimen as the specimen from which the first training image was obtained, or to a specimen corresponding to the specimen from which the third training image was obtained (for example, a continuous section specimen). As the above-mentioned fluorescent nuclear staining, there is no limitation, and 4',6-diamidino-2-phenylindole (DAPI) staining can be cited. The above-mentioned staining is preferably the same as the staining applied to the specimen when the second training image was obtained. The fourth preliminary training data is generated from the fourth training image. The fourth preliminary training data is binarized data generated by binarizing the fourth training image, and shows whether it is the region of the cell nucleus or the region other than it.

[0085] The fourth preliminary training data and the third preliminary training data are associated with each pixel, and a label value indicating "a non-tumor cell nuclear region" is further assigned to a pixel showing a cell nuclear region in the fourth preliminary training data. Data to which a label value indicating "a non-tumor cell nuclear region" is assigned to the cell nuclear region in the second preliminary training data is used as the fourth training data.

[0086] The training data includes tumor part training data and non-tumor part training data.

[0087] The nuclear region of the above-mentioned cell refers to a region containing one or more nuclei.

[0088] In the outline and embodiments of the present invention, a case where the tumorigenic state of cells included in an image of a HE-stained tissue sample is determined by a deep learning algorithm will be described as an example.

[0089] [Overview of deep learning methods and image analysis methods]

[0090] Hereinafter, the deep learning method and the image analysis method will be described in outline first, and then, each of the plurality of embodiments of the present invention will be described in detail.

[0091] Overview of deep learning methods

[0092] exist Figure 1 , an example of inputting training data to a neural network using a whole slice image 70W containing tumor tissue and non-tumor tissue is shown. The area surrounded by a solid line indicated by the symbol C shown in the whole slice image 70W shows the area of ​​tumor tissue. Symbol R1 indicates the area used as the first training image 70C. The area surrounded by a solid line indicated by the symbol NC shows the area of ​​non-tumor tissue. Symbol R2 indicates the area used as the third training image 70NC (shown in Figure 2 ) area used. Whether the first training image 70C contains tumor tissue can be determined before or after the first training image 70C is obtained. The same is true for whether the third training image 70NC contains non-tumor tissue, which can be determined before or after the third training image 70NC is obtained.

[0093] exist Figure 1 In FIG. 1 , the first training image 70C includes a plurality of hues in order to capture the HE-stained specimen as, for example, a color image under bright field observation of a microscope.

[0094] The first training image (bright field image) 70C can be acquired in advance using an image acquisition device such as a known optical microscope, fluorescence microscope, or virtual slide scanner. In the example, in the present embodiment, the color image acquired from the image acquisition device is preferably a 24-bit color in the color space of RGB. In the 24-bit color of RGB, it is preferred that the concentration (color concentration) of each red, green, and blue is represented by 8 bits (256 stages). The first training image (bright field image) 70C only needs to be an image containing one or more primary colors. The first training data and the first preliminary training data 70CL are generated from the first training image 70C.

[0095] In the present invention, hue is illustratively defined by a combination of three primary colors of light or a combination of three primary colors of color. The first training data is generated by separating the hue presented in the first training image 70C generated from the first training image 70C into each primary color and representing the data with a symbol corresponding to the density of each primary color. Figure 1 The single color images 72R, 72G, and 72B of each primary color, red (R), green (G), and blue (B), which are the three primary colors of light, are obtained.

[0096] If the color concentration of each color is symbolized for each pixel on the single-color images 72R, 72G, and 72B, the entire image becomes a symbolized diagram 72r, 72g, and 72b corresponding to the color concentration of each pixel for each R, G, and B image. The color concentration can also be symbolized using numerical values ​​representing 256 levels of each color. In addition, the color concentration can also be pre-processed for the numerical values ​​representing the 256 levels of each color. For example, the color concentration in each pixel can be symbolized using a digital number represented by 8 levels from value 0 to value 7. Figure 1 The color density symbolization diagrams 72r, 72g, and 72b in the single-color images of R, G, and B shown by way of example represent the color density in each pixel with symbols of 8 levels ranging from value 0 to value 7 (3 levels when expressed in gradations). Figure 1 The color density symbolization diagrams 72r, 72g, and 72b shown in FIG. 7 serve as first training data. Symbols indicating color density are also referred to as color density values ​​in this specification.

[0097] exist Figure 1 , the first preliminary training data 70CL is binary data indicating that the tissue contained in the first training image 70C is a tumor cell. The binary data may be image data or a label value corresponding to each pixel in the first training image 70C. For example, since the first training image 70C is a tumor tissue, the same numerical value is assigned to all pixels in the first training image 70C as a label value indicating that they are tumor cells.

[0098] exist Figure 1 In the example, the second training image 71C is an image obtained by photographing a fluorescent nuclear stained specimen in grayscale or color with two or more levels under fluorescence observation of a fluorescence microscope. The second training image 71C can be acquired in advance using a bright field image acquisition device such as a known fluorescence microscope or a virtual slide scanner. The second preliminary training data 71CN is generated from the second training image 71C.

[0099] The second preliminary training data 71CN is binary data of a region showing a cell nucleus of the tissue to be learned, which is generated from the second training image 71C obtained by imaging the tissue to be learned.

[0100] The second preliminary training data 71CN is generated by digitizing the second training image 71C of grayscale or color with more than two levels as a black and white fluorescent image by binarization processing. By binarizing the second training image 71C, the area of ​​the cell nucleus and the area outside it are distinguished, and the area of ​​the cell nucleus is determined. The judgment of whether it is the area of ​​the cell nucleus or the area outside it is performed, for example, by comparing the color concentration of each pixel in the image with a specified condition (for example, a threshold value of the color concentration). The above-mentioned binarized data can be image data or a label corresponding to each pixel in the second training image 71C. For example, a label of "1" is attached to the pixel in the area showing the cell nucleus, and a label of "0" is attached to the pixel outside the area of ​​the cell nucleus.

[0101] Next, the second training data 73C is generated. The second training data 73C is data generated from the first preliminary training data 70CL and the second preliminary training data 71CN, and is used as the true value image 73C to make the neural network 50 learn as the correct answer. Since the second training data 73C is generated from the first training image 70C and the second training image 71C containing the tumor tissue, the correct answer showing the tumor state of the tissue or cell is the nuclear region of the tumor cell. The second training data 73C contains label values ​​that distinguish between the nuclear region of the tumor cell corresponding to each pixel of the second training image 71C and other regions. For example, the label value representing the nuclear region of the tumor cell is "1", and the label representing other regions is "0". Figure 1 The second training data 73C shown in FIG. 7 is displayed as an image for convenience of explanation, but when input to the above-mentioned neural network, a label value that distinguishes between the nuclear region representing the tumor cell and other regions is given to each pixel.

[0102] In deep learning, Figure 1 The color density symbolized graphs 72r, 72g, 72b (also referred to as first training data) and the second training data 73C shown in the figure are used as the tumor part training data 74C. Specifically, the neural network 50 is trained with the color density symbolized graphs 72r, 72g, 72b as the input layer 50a and the second training data 73C as the output layer 50b. That is, the neural network 50 is trained with the relationship between the color density symbolized graphs 72r, 72g, 72b of each color R, G, B and the second training data 73C.

[0103] exist Figure 2 In the middle, the full slice like 70W and Figure 1 Similarly, the third training image 70NC includes a plurality of hues because the HE-stained specimen is captured as a color image under bright field observation of a microscope.

[0104] The third training image (bright field image) 70NC is generated by the same method as the first training image 70C. The third training data and the third preliminary training data 70NCL are generated from the third training image 70NC.

[0105] The third training data is generated by the same method as the first training data. Figure 3 The color density symbolization diagrams 72r, 72g, and 72b shown in FIG. 7 serve as third training data.

[0106] exist Figure 2 In the third preliminary training data 70NCL, the third preliminary training data 70NCL is binary data indicating that the tissue contained in the third training image 70NC is a non-tumor cell. The above-mentioned binary data may be image data or a label value corresponding to each pixel in the third training image 70NC. For example, since the third training image 70NC is a non-tumor tissue, the same numerical value is assigned to all pixels in the third training image 70NC as a label value indicating that it is a non-tumor cell. The numerical value indicating that it is a non-tumor cell is distinguished from the numerical value indicating that it is a tumor cell.

[0107] exist Figure 2 In the embodiment, the fourth training image 71NC is an image obtained by photographing a fluorescent nuclear stained specimen in grayscale or color with two or more levels under fluorescence observation of a fluorescence microscope. The fourth training image 71NC can be acquired in advance using a bright field image acquisition device such as a known fluorescence microscope or a virtual slide scanner. The fourth preliminary training data 71NCN is generated from the fourth training image 71NC.

[0108] The fourth preliminary training data 71NCN is generated by the same method as the second preliminary training data 71CN.

[0109] Next, the fourth training data 73NC is generated. The fourth training data 73NC is data generated from the third preliminary training data 70NCL and the fourth preliminary training data 71NCN, and is used as a true image 73NC to make the neural network 50 learn as a correct answer. Since the fourth training data 73NC is generated from the third training image 70NC and the fourth training image 71NC containing non-tumor tissue, the correct answer showing the tumor state of the tissue or cell is the nuclear region of the non-tumor cell. The fourth training data 73NC contains label values ​​that distinguish between the nuclear region of the non-tumor cell corresponding to each pixel of the fourth training image and other regions. For example, the label value representing the nuclear region of the non-tumor cell is "2", and the label value representing the other region is "0". Figure 2 The fourth training data 73NC shown in FIG. 7 is displayed as an image for the sake of convenience of explanation. However, when input into the above-mentioned neural network, each pixel is given a label value that distinguishes the nuclear region representing the tumor cell from other regions.

[0110] In deep learning methods, Figure 2 The color density symbolized graphs 72r, 72g, 72b (also referred to as the third training data) and the fourth training data 73NC shown in the figure are used as the non-tumor part training data 74NC. Specifically, the neural network 50 is trained with the color density symbolized graphs 72r, 72g, 72b as the input layer 50a and the fourth training data 73NC as the output layer 50b. That is, the neural network 50 is trained with the relationship between the color density symbolized graphs 72r, 72g, 72b of each color R, G, B and the fourth training data 73NC.

[0111] Reference Figure 3 (a) to (c) illustrate a method for generating the tumor part training data 74C. The tumor part training data 74C is a combination of the color density symbolization diagrams 72r, 72g, 72b of each color of R, G, and B and the second training data 73C. Figure 3 In (a), the image size (the size of each training data) is simplified for convenience of explanation, and the color density symbolization images 72r, 72g, 72b and the second training data 73C have a total of 81 pixels, namely, 9 pixels in the vertical direction and 9 pixels in the horizontal direction.

[0112] exist Figure 3 (b) shows an example of pixels constituting the tumor part training data 74C. Figure 3 The three values ​​74a shown in the upper section of (b) are the concentration values ​​of the R, G, and B colors in each pixel. Exemplarily, the three values ​​are stored in the order of red (R), green (G), and blue (B). The color concentration value of each pixel in the color concentration symbolization diagrams 72r, 72g, and 72b is represented by 8 levels from value 0 to value 7. This is an example of image preprocessing, in which the brightness of the images 72R, 72G, and 72B of each color represented by 256 levels during video recording is converted into 8 levels of color concentration values. For example, the color concentration value has the lowest brightness (a color level group with a low brightness value when represented by 256 levels of RGB colors) as the color concentration value 0, and a value that gradually increases as the brightness increases is assigned, and the highest brightness (a color level group with a high brightness value when represented by 256 levels of RGB colors) is used as the color concentration value 7. Figure 3 The value 74b shown in the lower part of (b) is the label value of the second training data 73C. For example, the label value 1 represents the nucleus of the tumor cell, and the label value 0 represents the area outside the nucleus. Figure 1 In the second training data 73C shown in , the position of the pixel whose label value changes from 1 to 0 or from 0 to 1 corresponds to the boundary between the nuclear region of the tumor cell and the region other than the nuclear region.

[0113] Figure 3The tumor part training data 75C shown in (c) is cut out Figure 3 The data of the area of ​​the specified number of pixels (hereinafter referred to as "window size") of the tumor part training data 74C shown in (a) is shown. The window size of the tumor part training data 75C is also simplified to 3×3 pixels for convenience of explanation, but the actual preferred window size is illustratively about 113×113 pixels, and the size of the window that can accommodate about 3×3 nuclei of normal gastric epithelial cells is preferred from the perspective of learning efficiency. For example, Figure 3 As shown in (c), a window W1 of 3×3 pixels is set, and the window W1 is moved with respect to the tumor part training data 74C. The center of the window W1 is located at any pixel in the tumor part training data 74C. For example, the tumor part training data 74C in the window W1 represented by the black frame is cut out as the tumor part training data 75C of the window size. The cut out tumor part training data 75C of the window size is Figure 1 The neural network 50 shown in FIG.

[0114] like Figure 1 As shown, the number of nodes in the input layer 50a of the neural network 50 corresponds to the product of the number of pixels of the tumor part training data 75C of the input window size and the number of primary colors contained in the image (for example, the three primary colors of light are R, G, and B). The color density value data 76 of each pixel of the tumor part training data 75C of the window size is used as the input layer 50a of the neural network, and the label value 77 of the pixel located at the center among the label values ​​74b of the second training data 73C corresponding to each pixel of the tumor part training data 75C is used as the output layer 50b of the neural network to make the neural network 50 learn. The color density value data 76 of each pixel is the collection data of the color density values ​​74a of each color of R, G, and B of each pixel of the tumor part training data 75C. As an example, when the window size of the tumor part training data 75C is 3×3 pixels, since each pixel is given one color concentration value 74a for R, G, and B respectively, the number of color concentration values ​​of the color concentration value data 76 becomes "27" (3×3×3=27), and the number of nodes in the input layer 50a of the neural network 50 also becomes "27".

[0115] In this way, the tumor part training data 75C of the window size input to the neural network 50 is not generated by the user but can be automatically generated by the computer. This promotes effective deep learning of the neural network 50.

[0116] like Figure 3As shown in (c), in the initial state, the center of window W1 is located at the upper left corner of the tumor part training data 74C. Afterwards, the tumor part training data 75C of the window size is cut out from window W1, and the position of window W1 is moved each time the neural network 50 is learned. Specifically, window W1 is moved in units of 1 pixel in a manner such that, for example, all pixels of the tumor part training data 74C are scanned by the center of window W1. Thus, the tumor part training data 75C of the window size cut out from all pixels of the tumor part training data 74C is used in the learning of the neural network 50. Thus, the degree of learning of the neural network 50 can be improved, and as a result of deep learning, effective results are obtained. Figure 5 A deep learning algorithm of the neural network 60 structure shown in FIG.

[0117] exist Figure 4 In the figure, non-tumor part training data 74NC is displayed. The method of generating non-tumor part training data 74NC is the same as that described in the tumor part training data 74C. However, in the non-tumor part training data 74NC, for example, the label value 2 represents the nuclear region of non-tumor cells, and the label value 0 represents the region other than the nuclear region. That is, in Figure 2 In the fourth training data 73NC shown in FIG. 1 , the position of the pixel whose label value changes from 2 to 0 or from 0 to 2 corresponds to the boundary between the nuclear region of the non-tumor cell and the region other than the nuclear region.

[0118] The tumor part training data 74C and the non-tumor part training data 74NC are combined into three-value data, in which the label representing the nuclear region of tumor cells is 1, the label representing the nuclear region of non-tumor cells is 2, and the label representing the nuclear region that is neither a tumor cell nor a part of the nuclear region is 0, which is used as true value data to enable the neural network 50 to learn.

[0119] · Overview of image analysis methods

[0120] like Figure 5 As shown, in the image analysis method, analysis data 80 is generated from an analysis object image (bright field image) 78 obtained by photographing a specimen of tissue or cells containing the analysis object. The specimen is preferably stained in the same manner as the first training image. The analysis object image 78 can also be obtained as, for example, a color image using, for example, a known microscope or a virtual slice scanner. The analysis object image (bright field image) 78 only needs to be an image containing one or more primary colors. If the color analysis object image 78 is symbolized with color concentration values ​​of R, G, and B for each pixel, the entire image can be represented as a symbolized diagram of the color concentration values ​​in each pixel for R, G, and B respectively (color concentration symbolized diagrams 79r, 79g, and 79b for analysis). Figure 5The color density symbolization diagrams 79r, 79g, and 79b that are shown by way of example to represent the symbols of the color density in the single-color images of R, G, and B colors display the color density values ​​represented by symbols in 8 stages from value 0 to value 7 instead of the images 79R, 79G, and 79B of the three primary colors.

[0121] The analysis data 80 is data of a region (i.e., window size) of a specified number of pixels cut out of the color density symbolization images 79r, 79g, and 79b, and includes data of color density values ​​of tissues or cells included in the analysis target image 78. The analysis data 80 of the window size is also simplified to 3×3 pixels for convenience of explanation, similarly to the tumor part training data 75C and the non-tumor part training data 75NC, but the actual preferred window size is illustratively about 113×113 pixels, and the size of the nuclei of about 3×3 normal gastric epithelial cells is preferably placed therein from the aspect of discrimination accuracy, and is, for example, about 113×113 pixels with a 40-fold field of view. For example, a window W2 of 3×3 pixels is set, and the window W2 is moved with respect to the color density symbolization images 79r, 79g, and 79b. The center of the window W2 is located at any pixel in the color density symbolization diagram 79r, 79g, 79b. If the color density symbolization diagram 79r, 79g, 79b is cut out from the window W2 indicated by the black frame of 3×3 pixels, for example, the analysis data 80 of the window size is obtained. In this way, the analysis data 80 is generated from the color density symbolization diagrams 79r, 79g, 79b for each area with the designated pixel as the center and the surrounding pixels. The designated pixel refers to the pixel of the color density symbolization diagram 79r, 79g, 79b located at the center of the window W2, and the surrounding pixels refer to the pixels of the color density symbolization diagram 79r, 79g, 79b included in the range of the window size with the designated pixel as the center. In the analysis data 80, the color density value for each pixel is stored in the order of red (R), green (G), and blue (B), similarly to the tumor part training data 74C and the non-tumor part training data 74NC.

[0122] In the image analysis method, there are Figure 1 and Figure 2 The deep learning algorithm 60 of the neural network learned by using the tumor part training data 75C and the non-tumor part training data 75NC of the window size shown in FIG. The analysis data 80 is processed to generate data 83 showing the tumor state in the tissue or cell to be analyzed.

[0123] Refer to Figure 5, the analysis data 80 cut out from the color density symbolization diagrams 79r, 79g, and 79b of each color of R, G, and B is input to the neural network 60 constituting the deep learning algorithm. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors contained in the image. If the color density value data 81 of each pixel of the analysis data 80 is input to the neural network 60, the estimated value 82 (three values) of the pixel located at the center of the analysis data 80 is output from the output layer 60b. For example, when the estimated value is 1, it indicates that it is the nuclear region of the tumor cell, when the estimated value is 2, it indicates that it is not the nuclear region of the tumor cell, but the nuclear region of the non-tumor cell, and when the estimated value is 0, it indicates that it is a region other than the cell nucleus. That is, the estimated value 82 output from the output layer 60b of the neural network 60 is data generated for each pixel of the analysis target image, and is data showing the tumor state of the cell in the analysis target image. The estimated value 82 is distinguished by three values, such as value 1, value 2, and value 0, for the nuclear region of the tumor cell, the nuclear region of the non-tumor cell, and the region outside the cell nucleus. The estimated value 82 is also called a label value, and is also called a class in the following description of the neural network. The neural network 60 generates a label showing the tumor state of the cell for the pixel located at the center of the analysis data 80 input. In other words, the neural network 60 classifies the analysis data 80 into a class showing the tumor state of the tissue or cell contained in the analysis object image. Furthermore, the color density value data 81 of each pixel is the collective data of the color density values ​​of the R, G, and B colors of each pixel of the analysis data 80.

[0124] Then, while the window W2 is moved in units of one pixel so as to scan all pixels of the color density symbolization images 79r, 79g, and 79b of each color of R, G, and B at the center of the window W2, the analysis data 80 is cut out according to the window size. The cut analysis data 80 is input to the neural network 60. Thus, a label value 83 is obtained as data showing the tumor state of the tissue or cell in the analysis target image. Figure 5In the example shown in , by also performing tumor cell nucleus region detection processing on the label value 83, a tumor cell nucleus region emphasized image 84 showing the region of the tumor cell nucleus is obtained. Specifically, the tumor cell nucleus region detection processing is, for example, a process of detecting pixels whose estimated value 82 is 1, and is actually a process of discriminating the region of the tumor cell nucleus. The tumor cell nucleus region emphasized image 84 is an image in which the region of the cell nucleus obtained by the image analysis processing is overlapped and displayed on the analysis object image 78 of the analysis object. In addition, after discriminating the region of the tumor cell nucleus, a process of displaying the tumor cell nucleus and other regions (i.e., non-tumor cell nuclei or other than the cell nucleus) on the display device in a identifiable manner may be performed. For example, a process of drawing a line between the region of the tumor cell nucleus filled with color and other regions is performed, and these are displayed in a identifiable manner on the display device.

[0125] <First embodiment>

[0126] In the first embodiment, the configuration of a system for implementing the deep learning method and the image analysis method described in the above overview will be described in detail.

[0127] [Overview of the structure]

[0128] Reference Figure 6 , the image analysis system involved in the first embodiment includes a deep learning device 100A and an image analysis device 200A. The supplier-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A causes the neural network 50 to learn using training data, and provides the deep learning algorithm 60 trained by the training data to the user. The deep learning algorithm composed of the learned neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via the recording medium 98 or the network 99. The image analysis device 200A uses the deep learning algorithm composed of the learned neural network 60 to analyze the image of the analysis object.

[0129] The deep learning device 100A is composed of, for example, a general-purpose computer, and performs deep learning processing based on the flowchart described below. The image analysis device 200A is composed of, for example, a general-purpose computer, and performs image analysis processing based on the flowchart described below. The recording medium 98 is a computer-readable non-temporary tangible recording medium such as a DVD-ROM or USB memory.

[0130] The deep learning device 100A is connected to the imaging device 300. The imaging device 300 includes an imaging element 301 and a fluorescence microscope 302, and captures a bright field image and a fluorescence image of a learning specimen 308 placed on a stage 309. The learning specimen 308 is stained as described above. The deep learning device 100A obtains the first training image 70 and the second training image 71 captured by the imaging device 300.

[0131] The image analysis device 200A is connected to the imaging device 400. The imaging device 400 includes an imaging element 401 and a fluorescence microscope 402, and captures a bright field image of a specimen 408 to be analyzed set on a stage 409. The specimen 408 to be analyzed is stained in advance as described above. The image analysis device 200A acquires the image 78 to be analyzed captured by the imaging device 400.

[0132] A known fluorescence microscope or virtual slide scanner having a function of imaging a specimen can be used as the imaging devices 300 and 400. The imaging device 400 may be an optical microscope as long as it has a function of imaging a specimen.

[0133] [Hardware Configuration]

[0134] Reference Figure 7 , the supplier-side device 100 ( 100A, 100B) includes a processing unit 10 ( 10A, 10B), an input unit 16 and an output unit 17.

[0135] The processing unit 10 includes: a CPU (Central Processing Unit) 11 for performing data processing described later, a memory 12 used in a data processing operation area, a recording unit 13 for recording programs and processing data described later, a bus 14 for transmitting data between the units, an interface unit 15 for inputting and outputting data with an external device, and a GPU (Graphics Processing Unit) 19. An input unit 16 and an output unit 17 are connected to the processing unit 10. By way of example, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the calculation processing (for example, parallel calculation processing) performed by the CPU 11. In the following description, the processing performed by the CPU 11 also includes the processing performed by the CPU 11 using the GPU 19 as an accelerator.

[0136] In addition, the processing unit 10 performs the following Fig.10In order to process the steps described in the above, the program and the neural network 50 before learning according to the present invention are recorded in advance in the recording unit 13 in an executable format, for example. The executable format is a format generated by converting the programming language by a compiler, for example. The processing unit 10 performs processing using the program recorded in the recording unit 13 and the neural network 50 before learning.

[0137] In the following description, unless otherwise specified, the processing performed by the processing unit 10 refers to the processing performed by the CPU 11 based on the program and the neural network 50 stored in the recording unit 13 or the memory 12. The CPU 11 uses the memory 12 as a working area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately records long-term storage data such as calculation results in the recording unit 13.

[0138] Reference Figure 8 , the user-side device 200 (200A, 200B, 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26 and an output unit 27.

[0139] The processing unit 20 includes: a CPU (Central Processing Unit) 21 for performing data processing described later, a memory 22 used in a data processing operation area, a recording unit 23 for recording programs and processing data described later, a bus 24 for transmitting data between the units, an interface unit 25 for inputting and outputting data with an external device, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. By way of example, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the calculation processing (for example, parallel calculation processing) performed by the CPU 21. That is, in the following description, the processing performed by the CPU 21 also includes the processing performed by the CPU 21 using the GPU 29 as an accelerator.

[0140] In addition, the processing unit 20 performs the following Fig.13 The processing of each step described in the above is performed by pre-recording the program and the deep learning algorithm 60 of the learned neural network structure according to the present invention in the recording unit 23, for example, in an executable form. The executable form is, for example, a form generated by converting the programming language by a compiler. The processing unit 20 performs processing using the program and the deep learning algorithm 60 recorded in the recording unit 23.

[0141] In the following description, unless otherwise specified, the processing performed by the processing unit 20 refers to the processing actually performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the recording unit 23 or the memory 22. The CPU 21 uses the memory 22 as a working area to temporarily store necessary data (intermediate data during processing, etc.), and appropriately records long-term storage data such as calculation results in the recording unit 23.

[0142] [Function blocks and processing order]

[0143] Deep learning processing

[0144] Reference Fig. 9 , the processing unit 10A of the deep learning device 100A according to the first embodiment includes a training data generating unit 101, a training data input unit 102, and an algorithm updating unit 103. These functional blocks are implemented by installing a program that causes a computer to execute deep learning processing in the recording unit 13 or the memory 12 of the processing unit 10A, and the CPU 11 executes the program. The window size database 104 and the algorithm database 105 are recorded in the recording unit 13 or the memory 12 of the processing unit 10A.

[0145] The first training image 70C, the second training image 71C, the third training image 70NC, and the fourth training image 71NC of the specimen for learning are pre-photographed by the imaging device 300 and pre-stored in the recording unit 13 or the memory 12 of the processing unit 10A. The neural network 50 is pre-stored in the algorithm database 105 in correspondence with, for example, the type of tissue sample (e.g., tissue name) from which the specimen to be analyzed is derived or the type of cell-containing sample.

[0146] The processing unit 10A of the deep learning device 100A performs Fig.10 Use the processing shown in Fig. 9 The processing of steps S10 to S14, S19 and S20 is performed by the training data generation unit 101. The processing of step S15 is performed by the training data input unit 102. The processing of steps S16 to S18 is performed by the algorithm update unit 103.

[0147] In steps S10 to S20 described below, deep learning processing for a pair of first training images 70C and second training images 71C and a pair of third training images 70NC and fourth training images 71NC is described. The pair of first training images 70C and second training images 71C is a pair of training images for learning whether the pixel of interest is a region of a cell nucleus and is a cell nucleus of a tumor cell, or a region that is not a cell nucleus. The pair of third training images 70NC and fourth training images 71NC is a pair of training images for learning whether the pixel of interest is a region of a cell nucleus and is a cell nucleus of a non-tumor cell, or a region that is not a cell nucleus.

[0148] In step S10, the processing unit 10A generates color density symbolization diagrams 72r, 72g, and 72b for each color of R, G, and B from the input first training image 70C. The color density symbolization diagrams 72r, 72g, and 72b are produced by representing the color density values ​​of each color of R, G, and B of each pixel of the first training image 70C in stages. In the present embodiment, the color density symbolization diagrams 72r, 72g, and 72b are produced for each R, G, and B color scale image with the color density value being 8 stages from a value of 0 to a value of 7. The color density value is allocated, for example, with the lowest brightness being the color density value 0, and values ​​that gradually increase as the brightness increases, and the highest brightness being the color density value 7.

[0149] In step S11, the processing unit 10A generates first preliminary training data 70CL showing a region of tumor cells in the tissue to be learned in the first training image 70C. The first preliminary training data 70CL is used to generate second three-value training data 73C described later. The first preliminary training data 70CL includes a label value indicating a region containing tumor cells.

[0150] For example, the processing unit 10A displays an image (whole slice image 70W) of a wide area including the area of ​​the first training image 70C on the output unit 17. The pathologist who makes the judgment visually confirms the image of the whole slice image 70W displayed on the output unit 17. The pathologist specifies the area in the whole slice image 70W judged to contain tumor cells via, for example, the input unit 16, and writes a solid line such as red in the whole slice image 70W. The area surrounded by the red solid line in the whole slice image is the area judged by the pathologist to contain tumor cells. Similarly, the pathologist writes a solid line of, for example, blue different from red, for the area in the whole slice image 70W judged to not contain tumor cells in the first training image 70C. The area surrounded by the blue solid line in the whole slice image 70W is the area judged by the pathologist to not contain tumor cells. Instead of displaying the whole slice image 70W on the output unit 17 , the processing unit 10A may cause the pathologist to make a judgment and obtain the whole slice image 70W after the pathologist's judgment is completed through the I / F unit 15 , for example, through the network 99 .

[0151] The processing unit 10A assigns, for example, a value of "1" as a label value indicating that the region is a tumor region to each pixel of the bright field image obtained from the region surrounded by the red solid line in the full slice image 70W. Thereafter, the processing unit 10A generates the first preliminary training data 70CL showing the tumor state of the tissue of the learning object by cutting out a region corresponding to the region of the first training image 70C from the full slice image for judgment by the pathologist. The processing unit 10A assigns, for example, a value of "2" as a label value indicating that the region is a non-tumor region to each pixel of the bright field image obtained from the region surrounded by the blue solid line. Thereafter, the processing unit 10A generates the third preliminary training data 70NCL showing the tumor state of the tissue of the learning object by cutting out a region corresponding to the region of the third training image 70NC from the full slice image for judgment by the pathologist.

[0152] Furthermore, in the whole slice image indicated by the red and blue solid lines according to the pathologist's judgment result, the area surrounded by red and the area surrounded by blue do not overlap and are displayed as mutually distinguished areas. There are also cases where the area containing tumor cells and the area not containing tumor cells are adjacent, and the area surrounded by red and the area surrounded by blue are adjacent.

[0153] In step S12, the processing unit 10A binarizes the color level of each pixel of the input second training image 71C to generate the second preliminary training data 71CN showing the cell nucleus region of the tissue to be learned. The second preliminary training data 71CN showing the cell nucleus region is used to generate the second ternary training data 73C described later. The binarization process is performed, for example, by comparing the color level of each pixel in the image with a specified threshold value. Instead of comparing with the threshold value, the binarization may be performed using the maximum likelihood estimation method.

[0154] In step S13a, the processing unit 10A generates the second training data 73C from the first preliminary training data 70CL and the second preliminary training data 71CN. The second training data 73C (ternary image) is used to generate the tumor part training data 74C for the neural network 50 to learn as a correct answer. That is, the processing unit 10A assigns a label value indicating that it is a nuclear region of a tumor cell and a label value indicating that it is a region other than the nuclear region to each pixel at the same position of the corresponding first preliminary training data 70CL and the second preliminary training data 71CN, based on the label value indicating that the pixel is a tumor cell and the label value indicating that the cell nucleus region and the region other than the nuclear region. The processing unit 10A generates the second training data 73C by performing such a judgment process on each pixel for all the pixels included in the first preliminary training data 70CL and the second preliminary training data 71CN.

[0155] As an example of the determination of each pixel, the processing unit 10A assigns, for example, a label value of "1" or "0" as a label value of the tumor state of the cell assigned to each image in the second training data 73C. For example, a value of "1" is assigned to a pixel corresponding to a cell nucleus of a tumor cell, and a value of "0" is assigned to a pixel in a region that is not a cell nucleus.

[0156] For example, if the value of the pixel of interest of the second preliminary training data 71CN indicating the cell nucleus region is a label value indicating the region of the cell nucleus, the processing unit 10A also refers to the label value of the pixel of interest of the first preliminary training data 70CL indicating the tumorization state, and assigns the label value of the referenced pixel of interest as the label value of the tumorization state of the pixel of interest of the second training data 73C. If the value of the referenced pixel of interest is, for example, a value "1" indicating a region of tumor cells, the processing unit 10A assigns a value "1" to the pixel of interest of the second training data 73C. In contrast, if the value of the pixel of interest of the second preliminary training data 71CN indicating the cell nucleus region is a value indicating a region other than the cell nucleus, the processing unit 10A assigns a value "0" to the pixel of interest of the second training data 73C without referring to the label value of the pixel of interest of the first preliminary training data 70CL.

[0157] In step S13b, the processing unit 10A generates the fourth training data 73NC from the third preliminary training data 70NCL and the fourth preliminary training data 71NCN. The fourth training data 73NC is used to generate the non-tumor part training data 74NC for the neural network 50 to learn as a correct answer. That is, the processing unit 10A assigns a label value indicating the nuclear region of non-tumor cells and a label value indicating other regions to each pixel at the same position corresponding to the third preliminary training data 70NCL and the fourth preliminary training data 71NCN, based on the label value indicating that the pixel is a non-tumor cell and the label value indicating the cell nucleus region and its exceptional region. The processing unit 10A generates the fourth training data 73NC by performing such a judgment process on each pixel for all the pixels included in the third preliminary training data 70NCL and the fourth preliminary training data 71NCN.

[0158] As an example of the judgment of each pixel, the processing unit 10A assigns, as a label value indicating the tumorigenic state of the cell to each pixel in the fourth training data 73NC, a label value of, for example, "2" or "0". For example, a value of "2" is assigned to a pixel corresponding to a cell nucleus of a non-tumor cell, and a value of "0" is assigned to a pixel in a region that is not a cell nucleus.

[0159] For example, if the value of the pixel of interest of the fourth preliminary training data 71NCN indicating the cell nucleus region is a label value indicating the region of the cell nucleus, the processing unit 10A also refers to the label value of the pixel of interest of the third preliminary training data 70NCL indicating the tumorization state, and assigns the label value of the referenced pixel of interest as the label value indicating the tumorization state of the pixel of interest of the fourth training data 73NC. If the value of the referenced pixel of interest is, for example, a value "2" indicating a region of non-tumor cells, the processing unit 10A assigns a value "2" to the pixel of interest of the second training data 73C. In contrast, if the value of the pixel of interest of the fourth preliminary training data 71NCN indicating the cell nucleus region is a value indicating a region other than the cell nucleus, the processing unit 10A assigns a value "0" to the pixel of interest as the label value indicating the tumorization state of the pixel of interest of the fourth training data 73NC without referring to the label value of the pixel of interest of the third preliminary training data 70NCL.

[0160] In step S14, the processing unit 10A receives input of the type of tissue for learning from the operator on the deep learning device 100A side through the input unit 16. The processing unit 10A sets the window size based on the input tissue type, referring to the window size database 104, and sets the neural network 50 used in the learning by referring to the algorithm database 105. In the present embodiment in which the tissue sample of the stomach is used as the analysis object, the window size is, for example, 113×113 pixels. This pixel size is, for example, the size in an image photographed at 40 times. For example, it supports the size that the entire shape of the cell nucleus region of at least one cell among a plurality of cells ranging from 2 to 9 is included in the window. The window size is a unit of training data input to the neural network 50 at one input, and the product of the number of pixels of the tumor part training data 75C and the number of primary colors of the color contained in the image corresponds to the number of nodes of the input layer 50a. The window size corresponds to the type of the tissue sample or the type of the sample containing cells and is recorded in advance in the window size database 104.

[0161] In step S15, the processing unit 10A generates window-sized tumor portion training data 75C from the first training data and the second training data 73C, which are the color density symbolization images 72r, 72g, and 72b. Specifically, in the above-mentioned "Overview of the Deep Learning Method", as shown in FIG. Figure 3 As described in (a) to (c), the tumor part training data 75C of the window size is generated from the tumor part training data 74C which is a combination of the first training data and the second training data 73C by using the window W1. In addition, the processing unit 10A generates the non-tumor part training data 75NC of the window size from the third training data and the fourth training data 73C which are the color density symbolized images 72r, 72g, 72b. Specifically, in the above-mentioned "Overview of the Deep Learning Method", as shown in FIG. Figure 3 As described in (a) to (c), non-tumor part training data 75NC of the window size is prepared by window W1 from non-tumor part training data 74NC which is a combination of the third training data and the fourth training data 73NC as color density symbolization diagrams 72r, 72g, and 72b.

[0162] exist Fig.10 In step S16 shown in FIG. 1 , the processing unit 10A uses the tumor part training data 75C and the non-tumor part training data 75NC of the window size to train the neural network 50. The learning result of the neural network 50 is accumulated each time the neural network 50 is trained using the tumor part training data 75C and the non-tumor part training data 75NC of the window size.

[0163] In the image analysis method according to the embodiment, since a folded neural network is used and a probabilistic gradient reduction method is used, in step S17, the processing unit 10A determines whether or not the learning results of the predetermined number of trials are accumulated. When the learning results are accumulated for the predetermined number of trials, the processing unit 10A performs the processing of step S18, and when the learning results are not accumulated for the predetermined number of trials, the processing unit 10A performs the processing of step S19.

[0164] When the learning results are accumulated for the designated number of trials, in step S18, the processing unit 10A uses the learning results accumulated in step S16 to update the combined weight w of the neural network 50. In the image analysis method according to the embodiment, since the probabilistic gradient descent method is used, the combined weight w of the neural network 50 is updated at the stage of accumulating the learning results for the designated number of trials. Specifically, the process of updating the combined weight w is a process of performing calculations by the gradient descent method as shown in (Formula 11) and (Formula 12) described later.

[0165] In step S19, the processing unit 10A determines whether to process a specified number of pixels in the input image. When the input image is the tumor part training data 74C and the non-tumor part training data 75NC, the deep learning process ends when a series of processes from step S16 to step S18 are performed on the specified number of pixels in the tumor part training data 74C and the non-tumor part training data 75NC. The learning of the neural network does not necessarily have to be performed on all the pixels in the input image. The processing unit 10A can learn by processing a part of the pixels in the input image, that is, a specified number of pixels. The specified number of pixels can also be all the pixels in the input image.

[0166] When a predetermined number of pixels in the input image are not to be processed, the processing unit 10A performs the following steps in step S20: Figure 3 As shown in (c), the center position of the window is moved in units of 1 pixel in the tumor part training data 74C and the non-tumor part training data 75NC. Thereafter, the processing unit 10A performs a series of processing from step S16 to step S18 in the new window position after the movement. That is, in step S15, the processing unit 10A cuts out the tumor part training data 74C or the non-tumor part training data 75NC with the window size at the new window position after the movement. Then, in step S16, the processing unit 10A uses the tumor part training data 75C and the non-tumor part training data 75NC of the newly cut window size to learn the neural network 50. In step S17, when the learning results of the specified number of trials are accumulated, the processing unit 10A updates the combined weight w of the neural network 50 in step S18. The learning of the neural network 50 of each such window size is performed on a specified number of pixels in the tumor part training data 74C and the non-tumor part training data 75NC.

[0167] By repeating the deep learning process of steps S10 to S20 for one pair of input images described above for a plurality of pairs of different input images, the degree of learning of the neural network 50 is improved. Figure 5 A deep learning algorithm for the neural network structure shown in 60.

[0168] The structure of the neural network

[0169] like Fig.11 As shown in (a), in the first embodiment, a deep learning type neural network is used. A deep learning type neural network, such as Fig.11 Like the neural network 50 shown in FIG. 1 , the neural network 50 includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b. The intermediate layer 50c is composed of a plurality of layers. The number of layers constituting the intermediate layer 50c can be, for example, 5 or more.

[0170] In the neural network 50, a plurality of nodes 89 arranged in layers are connected between layers. Thus, information is transmitted from the layer 50a on the input side to the layer 50b on the output side only in one direction indicated by the arrow D in the figure. In this embodiment, the number of nodes in the input layer 50a corresponds to the number of pixels of the input image, i.e. Figure 3 The product of the number of pixels of the window W1 shown in (c) and the number of primary colors of the color contained in each pixel. Since the pixel data (color density value) of the image can be input to the input layer 50a, the user can input the input image to the input layer 50a without calculating the feature value from the input image separately.

[0171] ·Calculation at each node

[0172] Fig.11 (b) is a schematic diagram showing the calculation at each node. At each node 89, a plurality of inputs are received and one output (z) is calculated. Fig.11 In the example shown in (b), the node 89 receives four inputs. The total input (u) received by the node 89 is expressed by the following (Formula 1).

[0173]

Number 1

[0174] u=w1x1+w2x2+w3x3+w4x4+b (Formula 1)

[0175] Each input has a different weight. In (Formula 1), b is a value called bias. The output (z) of the node becomes the output of the specified function f for the total input (u) represented by (Formula 1), which is expressed by the following (Formula 2). Function f is called the activation function.

[0176]

Number 2

[0177] z=f(u) (Formula 2)

[0178] Fig.11 (c) is a schematic diagram showing the calculation between nodes. In the neural network 50, the nodes that output the result (z) represented by (Formula 2) for the total input (u) represented by (Formula 1) are arranged in layers. The output of the node in the previous layer becomes the input of the node in the next layer. Fig.11 In the example shown in (c), the output of node 89a of the layer on the left side of the figure becomes the input of node 89b of the layer on the right side of the figure. Each node 89b of the layer on the right side receives the output from node 89a of the layer on the left side. Different weights are attached to each connection between each node 89a of the layer on the left side and each node 89b of the layer on the right side. If the outputs of each of the multiple nodes 89a of the layer on the left side are x1 to x4, the inputs of each of the three nodes 89b of the layer on the right side are expressed by the following (Formula 3-1) to (Formula 3-3).

[0179]

Number 3

[0180] u1=w 11 x1+w 12 x2+w 13 x3+w 14 x4+b1(Formula 3-1)

[0181] u2=w 21 x1+w 22 x2+w 23 x3+w 24 x4+b2(Formula 3-2)

[0182] u3=w 31 x1+w 32 x2+w 33 x3+w 34 x4+b3(Formula 3-3)

[0183] By generalizing these (Formula 3-1) to (Formula 3-3), we get (Formula 3-4) where i=1, ...I, j=1, ...J.

[0184]

Number 4

[0185]

[0186] Applying (Equation 3-4) to the activation function gives the output. The output is expressed by the following (Equation 4).

[0187]

Number 5

[0188] z j =f(u j ) (j=1, 2, 3) (Formula 4)

[0189] Activation function

[0190] In the image analysis method according to the embodiment, a rectified linear unit function is used as an activation function. The rectified linear unit function is represented by the following (Formula 5).

[0191]

Number 6

[0192] f(u)=max(u,0) (Formula 5)

[0193] (Equation 5) is a linear function of z = u, where the part where u < 0 is regarded as a function where u = 0. Fig.11 In the example shown in (c), the output of the node j=1 is expressed by (Equation 5) as follows.

[0194]

Number 7

[0195] z1=max((w 11 x1+w 12 x2+w 13 x3+w 14 x4+b1),0)

[0196] Learning of neural networks

[0197] Let the function expressed by the neural network be y(x:w), and the function y(x:w) changes by changing the parameter w of the neural network. Adjusting the function y(x:w) in such a way that the neural network selects a more appropriate parameter w for the input x is called learning of the neural network. When a plurality of inputs and outputs of the function expressed by the neural network are given, if the preferred output for a certain input x is d, the input-output set is given as {(x1, d1), (x2, d2), ..., (x n , d n )}. The set of groups represented by (x, d) is called training data. Specifically, Figure 3 The set of labels of the color density value and the true value image of each pixel in the single color image of R, G, and B shown in (b) is Figure 3 The training data shown in (a).

[0198] The learning of neural network means that even for any input-output combination (x n , d n ), also given input x n The output of the neural network y(x n :w) as close to output d as possible n The error function is a measure of how close the function expressed by the neural network is to the training data.

[0199]

Number 8

[0200] y(x n :w)≈d n

[0201] The error function is also called a loss function. The error function E(w) used in the image analysis method according to the embodiment is expressed by the following (Formula 6). (Formula 6) is called cross entropy.

[0202]

Number 9

[0203]

[0204] The calculation method of the cross entropy of (Formula 6) is explained. In the output layer 50b of the neural network 50 used in the image analysis method involved in the embodiment, that is, in the final layer of the neural network, an activation function is used to classify the input x into a finite number of classes according to the content. The activation function is called the Soft Max function and is represented by the following (Formula 7). Furthermore, in the output layer 50b, the number of nodes arranged is the same as the number of classes k. The total input u of each node k (k=1, ...K) of the output layer L is the output from the previous layer L-1, and is assumed to be u k (L) Thus, the output of the kth node in the output layer is expressed by the following (Formula 7).

[0205]

Number 10

[0206]

[0207] (Equation 7) is a Soft Max function. The output y1, ..., y1 is determined by (Equation 7). K The sum of is often 1.

[0208] Denote each class as C1,…,C K , then the output y of node k in the output layer L is K (i.e. u k (L) ) indicates that the given input x belongs to class C K The probability of . Refer to the following (Formula 8). Input x is classified into the class with the maximum probability expressed by (Formula 8).

[0209]

Number 11

[0210]

[0211] In neural network learning, the function represented by the neural network is regarded as a model of various posterior probabilities. Under such a probability model, the likelihood of the weight w for the training data is evaluated, and the weight w that maximizes the likelihood is selected.

[0212] The target output d of the Soft Max function of (Equation 7) is n , is set to 1 only when the output is the correct class, and is set to 0 when the output is other than that. n =[d n1 , …, d nK ] is expressed in vector form, for example, input x n When the correct answer class is C3, only the target output is d n3 becomes 1, and the other target outputs become 0. When symbolized in this way, the posterior distribution is expressed by the following (Formula 9).

[0213]

Number 12

[0214]

[0215] For the training data {(x n , d n The likelihood L(w) of the weight w of )}(n=1, ..., N) is expressed by the following (Equation 10). By inverting the logarithm of the likelihood L(w), the error function of (Equation 6) is derived.

[0216]

Number 13

[0217]

[0218] Learning means minimizing the error function E(w) calculated based on the training data with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is represented by (Formula 6).

[0219] Minimizing the error function E(w) with respect to parameter w is the same as finding the local minimum of the function E(w). Parameter w is the weight of the connection between nodes. The minimum of weight w is found by repeatedly updating parameter w with an arbitrary initial value as the starting point. An example of such calculation is the gradient descent method.

[0220] In the gradient descent method, a vector represented by the following (Equation 11) is used.

[0221]

Number 14

[0222]

[0223] In the gradient descent method, the current parameter w value will be moved in the negative gradient direction (i.e. )The process of moving is repeated several times. The current weight is used as w (t) , the weight after the move is taken as w (t+1) , the calculation by the gradient descent method is expressed as follows (Formula 12). The value t refers to the number of times the parameter w is moved.

[0224]

Number 15

[0225]

[0226] mark

[0227]

Number 16

[0228] ε

[0229] is a constant that determines the update amount of parameter w, and is called a learning coefficient. By repeating the calculation represented by (Equation 12), the error function E(w (t) ) decreases, and the parameter w reaches a minimum.

[0230] Furthermore, the calculation by (Equation 12) can be performed on all training data (n=1, ..., N) or only on a part of the training data. The gradient descent method performed on only a part of the training data is called a stochastic gradient descent method. In the image analysis method involved in the embodiment, a stochastic gradient descent method is used.

[0231] Image analysis and processing

[0232] Reference Fig.12 , the processing unit 20A of the image analysis device 200A involved in the first embodiment includes an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a tumor cell nucleus region detection unit 204. These functional blocks are realized by installing the program for causing a computer to perform image analysis processing according to the present invention into the recording unit 23 or the memory 22 of the processing unit 20A, and the CPU 21 executes the program. The window size database 104 and the algorithm database 105 are provided from the deep learning device 100A via the recording medium 98 or the network 99, and recorded in the recording unit 23 or the memory 22 of the processing unit 20A.

[0233] The analysis target image 78 of the tissue to be analyzed is taken in advance by the imaging device 400 and recorded in advance in the recording unit 23 or the memory 22 of the processing unit 20A. The deep learning algorithm 60 including the learned combination weight w is stored in the algorithm database 105 in correspondence with the type of tissue sample (e.g., tissue name) from which the specimen of the tissue to be analyzed is derived or the type of sample containing cells, and functions as a program module as a part of a program that causes a computer to execute image analysis processing. That is, the deep learning algorithm 60 is used by a computer equipped with a CPU and a memory to perform calculation or processing of unique information corresponding to the purpose of use that outputs data showing the tumor state in the tissue to be analyzed, so that the computer functions. Specifically, the CPU 21 of the processing unit 20A performs calculations of the neural network 60 based on the learned combination weight w according to the algorithm specified by the deep learning algorithm 60 recorded in the recording unit 23 or the memory 22. The CPU 21 of the processing unit 20A performs calculations on the analysis target image 78 of the analysis target tissue input to the input layer 60a, and outputs a ternary image 83 as data showing the tumor state in the analysis target tissue from the output layer 60b.

[0234] Reference Fig.13 , the processing unit 20A of the image analysis device 200A performs Fig.13 Use the processing shown in Fig.12 , the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the tumor cell nuclear region detection unit 204.

[0235] In step S21, the processing unit 20A generates color density symbolization maps 79r, 79g, and 79b for each of R, G, and B colors from the input analysis target image 78. The method of generating the color density symbolization maps 79r, 79g, and 79b is the same as that in Fig.10 The generation method in step S10 during the deep learning process shown in is the same.

[0236] exist Fig.13In step S22 shown in , the processing unit 20A receives an input of a type of tissue from the user on the image analysis device 200A side through the input unit 26 as an analysis condition. The processing unit 20A sets the window size used in the analysis based on the input type of tissue, referring to the window size database 104 and the algorithm database 105, and obtains the deep learning algorithm 60 used in the analysis. The window size is a unit of analysis data input to the neural network 60 at one input, and the product of the number of pixels of the analysis data 80 of the window size and the number of primary colors contained in the image corresponds to the number of nodes of the input layer 60a. The window size corresponds to the type of tissue and is recorded in advance in the window size database 104. The window size is as follows. Figure 5 The deep learning algorithm 60 is also pre-recorded in correspondence with the type of tissue sample or the type of cell-containing sample. Fig.12 The algorithm database 105 shown in .

[0237] exist Fig.13 In step S23 shown in FIG. 2 , the processing unit 20A generates analysis data 80 of the window size from the color density symbolization maps 79 r , 79 g , and 79 b .

[0238] In step S24, the processing unit 20A inputs the deep learning algorithm 60 Figure 5 The analysis data 80 shown in FIG. The initial position of the window is the same as step S16 during deep learning processing, for example, the pixel at the center of 3×3 pixels in the window corresponds to the position of the upper left corner of the analysis target image. The processing unit 20A inputs the data 81 of 27 color density values ​​of 3×3 pixels×3 primary colors contained in the analysis data 80 of the window size to the input layer 60a, and the deep learning algorithm 60 outputs the discrimination result 82 to the output layer 60b.

[0239] exist Fig.13 In step S25 shown in FIG. 2 , the processing unit 20A records the output to Figure 5 The discrimination result 82 of the output layer 60b shown in FIG. The discrimination result 82 is the estimated value (three values) of the pixel located at the center of the color density symbolized diagram 79r, 79g, 79b of the analysis object. For example, when the estimated value is 1, the nucleus of the tumor cell is displayed, when the estimated value is 2, the nucleus of the non-tumor cell is displayed, and when the estimated value is 0, it is displayed other than the nucleus.

[0240] exist Fig.13 In step S26 shown in FIG. 1 , the processing unit 20A determines whether to process all pixels in the input image. Figure 5 The color density symbolization diagrams 79r, 79g, and 79b shown in FIG. 7 are subjected to the color density symbolization diagrams 79r, 79g, and 79b for all pixels in the color density symbolization diagrams 79r, 79g, and 79b. Fig.13When a series of processes from step S23 to step S25 shown in FIG. 1 are completed, the process of step S28 is performed.

[0241] When not processing all pixels in the input image, the processing unit 20A performs step S27 similarly to step S20 in the deep learning process. Figure 5 In the color density symbolization diagrams 79r, 79g, and 79b shown in , the center position of the window W2 is moved in units of 1 pixel. Thereafter, the processing unit 20A performs a series of processes from step S23 to step S25 at the position of the new window W2 after the movement. In step S25, the processing unit 20A records the discrimination result 82 corresponding to the new window position after the movement. By recording the discrimination result 82 for each such window size for all pixels in the image to be analyzed, a three-value image 83 of the analysis result is obtained. The image size of the three-value image 83 of the analysis result is the same as the image size of the image to be analyzed. In the three-value image 83, the estimated value 2, value 1, and value 0 may be numerical data attached to each pixel, or may be an image displayed in a display color corresponding to each of the values ​​2, value 1, and value 0 instead of the estimated value 2, value 1, and value 0, for example.

[0242] exist Fig.13 In step S28 shown in FIG. 2 , the processing unit 20A outputs the three-value image 83 as the analysis result to the output unit 27 .

[0243] In step S29, following step S28, the processing unit 20A further performs tumor cell nucleus region detection processing on the three-value image 83 of the analysis result. In the three-value image 83, in addition to the nucleus of the tumor cell and the nucleus and nucleus of the non-tumor cell, they are distinguished by three values. Therefore, in the three-value image 83, by detecting the position of the pixel whose estimated value changes from 1 to 0 or from 0 to 1, the nucleus of the tumor cell can be distinguished. In addition, as another embodiment, the boundary between the nucleus of the tumor cell and other areas (i.e., non-tumor cell nuclei or other than the nucleus), that is, the area of ​​the tumor cell nucleus can be detected.

[0244] Optionally, the processing unit 20A creates a tumor cell nucleus region enhanced image 84 by superimposing the obtained tumor cell nucleus region on the analysis target image 78. The processing unit 20A outputs the created tumor cell nucleus region enhanced image 84 to the output unit 27, and ends the image analysis process.

[0245] As described above, the user of the image analysis device 200A can obtain the ternary image 83 as the analysis result by inputting the analysis target image 78 of the analysis target tissue to the image analysis device 200A. The ternary image 83 represents the nuclei of tumor cells, the nuclei of non-tumor cells, and the area other than the nuclei in the analysis target specimen, and the user can identify the area of ​​tumor cell nuclei in the analysis target specimen.

[0246] Furthermore, the user of the image analysis device 200A can obtain a tumor cell nucleus region emphasized image 84 as an analysis result. The tumor cell nucleus region emphasized image 84 is generated, for example, by coloring the tumor cell nucleus region in the analysis target image 78 to be analyzed. In another embodiment, the image is generated by overlapping the boundary between the tumor cell nucleus region and the other region (i.e., non-tumor cell nucleus or other than the cell nucleus). Thus, the user can grasp the tumor cell nucleus region in the analysis target tissue at a glance, and can grasp the tumorization state at a glance.

[0247] The region showing the tumor cell nucleus in the specimen to be analyzed helps a person who is not familiar with the specimen to understand the tumorigenic state of the cell nucleus.

[0248] <Second embodiment>

[0249] Hereinafter, the image analysis system according to the second embodiment will be described with reference to points different from the image analysis system according to the first embodiment.

[0250] [Overview of the structure]

[0251] Reference Fig.14 , the image analysis system involved in the second embodiment includes a user-side device 200, and the user-side device 200 operates as an integrated image analysis device 200B. The image analysis device 200B is composed of, for example, a general-purpose computer, and performs both the deep learning processing and the image analysis processing described in the first embodiment. That is, the image analysis system involved in the second embodiment is an independent system that performs deep learning and image analysis on the user side. The image analysis system involved in the second embodiment is different from the image analysis system involved in the first embodiment in that the integrated image analysis device 200B provided on the user side bears the functions of both the deep learning device 100A involved in the first embodiment and the image analysis device 200A.

[0252] The image analysis device 200B is connected to the imaging device 400. The imaging device 400 acquires the first training image 70C and the second training image 71C of the tissue for learning during the deep learning process, and acquires the analysis target image 78 of the tissue to be analyzed during the image analysis process.

[0253] [Hardware Configuration]

[0254] The hardware configuration of the image analysis device 200B Figure 8 The hardware configuration of the user-side device 200 shown in FIG. 1 is the same.

[0255] [Function blocks and processing order]

[0256] Reference Fig.15 , the processing unit 20B of the image analysis device 200B involved in the second embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203 and a tumor cell nuclear region detection unit 204. These functional blocks are realized by installing a program that enables a computer to perform deep learning processing and image analysis processing on the recording unit 23 or the memory 22 of the processing unit 20B, and the CPU 21 executes this program. The window size database 104 and the algorithm database 105 recorded in the recording unit 23 or the memory 22 of the processing unit 20B are used together during deep learning and image analysis processing. The learned neural network 60 is stored in the algorithm database 105 in advance in correspondence with the type of tissue or the type of cell-containing sample, and the combined weight w is updated by the deep learning process and stored in the algorithm database 105 as a deep learning algorithm 60. Furthermore, the first training image 70C and the second training image 71C, which are first training images for learning, are pre-photographed by the imaging device 400 and recorded in advance in the recording unit 23 or the memory 22 of the processing unit 20B. The analysis target image 78 of the specimen to be analyzed is also pre-photographed by the imaging device 400 and recorded in advance in the recording unit 23 or the memory 22 of the processing unit 20B.

[0257] The processing unit 20B of the image analysis device 200B performs deep learning processing. Fig.10 The processing shown in the figure is performed during the image analysis process. Fig.13 Use the processing shown in Fig.15 , then in the deep learning process, the processing of steps S10 to S14, S19 and S20 is performed by the training data generation unit 101. The processing of step S15 is performed by the training data input unit 102. The processing of steps S16 to S18 is performed by the algorithm update unit 103. In the image analysis process, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the tumor cell nuclear region detection unit 204.

[0258] The order of deep learning processing and the order of image analysis processing performed by the image analysis device 200B according to the second embodiment are the same as the order performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. In addition, the image analysis device 200B according to the second embodiment is different from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following aspects.

[0259] In step S14 during deep learning processing, the processing unit 20B receives input of a type of tissue for learning from a user of the image analysis device 200B via the input unit 26. Based on the input type of tissue, the processing unit 20B sets the window size by referring to the window size database 104, and sets the neural network 50 used for learning by referring to the algorithm database 105.

[0260] As described above, the user of the image analysis device 200B can obtain the ternary image 83 as an analysis result by inputting the analysis target image 78 to the image analysis device 200B. Furthermore, the user of the image analysis device 200B can obtain the tumor cell nucleus region enhanced image 84 as an analysis result.

[0261] According to the image analysis device 200B according to the second embodiment, the user can use the type of tissue selected by the user as the tissue for learning. This means that the learning of the neural network 50 is not outsourced to the supplier, and the user can improve the degree of learning of the neural network 50.

[0262] <Third embodiment>

[0263] Hereinafter, the image analysis system according to the third embodiment will be described with reference to points different from the image analysis system according to the second embodiment.

[0264] [Overview of the structure]

[0265] Reference Fig.16 , the image analysis system involved in the third embodiment includes a supplier-side device 100 and a user-side device 200. The supplier-side device 100 operates as an integrated image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is a device on the cloud server side, which is composed of, for example, a general-purpose computer and performs both the deep learning process and the image analysis process described in the first embodiment. The terminal device 200C is a terminal device on the user side, which is composed of, for example, a general-purpose computer and sends an image of the analysis object to the image analysis device 100B through the network 99 and receives an image of the analysis result from the image analysis device 100B through the network 99.

[0266] The image analysis system according to the third embodiment is similar to the image analysis system according to the second embodiment in that the integrated image analysis device 100B provided on the supplier side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. On the one hand, the image analysis system according to the third embodiment is provided with a terminal device 200C, and is different from the image analysis system according to the second embodiment in that an input interface of an image to be analyzed and an output interface of an image of an analysis result are provided to the terminal device 200C on the user side. That is, the image analysis system according to the third embodiment is a cloud service type system in which the supplier side that performs deep learning processing and image analysis processing provides an input and output interface of an image to be analyzed and an image of an analysis result to the user side.

[0267] The image analysis device 100B is connected to the imaging device 300 , and acquires the first training image 70C and the second training image 71C of the tissue for learning captured by the imaging device 300 .

[0268] The terminal device 200C is connected to the imaging device 400 , and acquires the analysis target image 78 of the analysis target tissue captured by the imaging device 400 .

[0269] [Hardware Configuration]

[0270] The hardware configuration of the image analysis device 100B Figure 7 The hardware configuration of the provider-side device 100 shown in FIG. 200C is the same as that of the terminal device 200C. Figure 8 The hardware configuration of the user-side device 200 shown in FIG. 1 is the same.

[0271] [Function blocks and processing order]

[0272] Reference Fig.17 , the processing unit 10B of the image analysis device 100B involved in the third embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203 and a tumor cell nuclear region detection unit 204. These functional blocks are realized by installing a program that enables a computer to perform deep learning processing and image analysis processing on the recording unit 13 or the memory 12 of the processing unit 10B, and the CPU 11 executes this program. The window size database 104 and the algorithm database 105 recorded in the recording unit 13 or the memory 12 of the processing unit 10B are used in common during deep learning and image analysis processing. The neural network 50 is stored in the algorithm database 105 in advance corresponding to the type of tissue, and the combined weight w is updated by the deep learning process and stored in the algorithm database 105 as a deep learning algorithm 60.

[0273] Furthermore, the first training image 70C, the second training image 71C, the third training image 70NC, and the fourth training image 71NC used for learning are assumed to be pre-photographed by the imaging device 300 and recorded in advance in the recording unit 13 or the memory 12 of the processing unit 10B. The analysis target image 78 of the tissue to be analyzed is also assumed to be pre-photographed by the imaging device 400 and recorded in advance in the recording unit 23 or the memory 22 of the processing unit 20C of the terminal device 200C.

[0274] The processing unit 10B of the image analysis device 100B performs deep learning processing. Fig.10 The processing shown in the figure is performed during the image analysis process. Fig.13 Use the processing shown in Fig.17 , then in the deep learning process, the processing of steps S10 to S14, S19 and S20 is performed by the training data generation unit 101. The processing of step S15 is performed by the training data input unit 102. The processing of steps S16 to S18 is performed by the algorithm update unit 103. In the image analysis process, the processing of steps S21 and S22 is performed by the analysis data generation unit 201. The processing of steps S23, S24, S26 and S27 is performed by the analysis data input unit 202. The processing of steps S25 and S28 is performed by the analysis unit 203. The processing of step S29 is performed by the tumor cell nuclear region detection unit 204.

[0275] The order of deep learning processing and the order of image analysis processing performed by the image analysis device 100B according to the third embodiment are the same as the order performed by the deep learning device 100A and the image analysis device 200A according to the first embodiment, respectively. Furthermore, the image analysis device 100B according to the third embodiment is different from the deep learning device 100A and the image analysis device 200A according to the first embodiment in the following four aspects.

[0276] exist Fig.13 In step S21 of the image analysis process shown in FIG. 1 , the processing unit 10B receives the analysis target image 78 of the tissue to be analyzed from the terminal device 200C on the user side, and generates color density symbolized graphs 79r, 79g, and 79b of the respective colors R, G, and B from the received analysis target image 78. The generation method of the color density symbolized graphs 79r, 79g, and 79b is similar to Fig.10 The generation method in step S10 during the deep learning process shown in is the same.

[0277] exist Fig.13In step S22 of the image analysis process shown in , the processing unit 10B receives input of the type of tissue from the user of the terminal device 200C as an analysis condition via the input unit 26 of the terminal device 200C. The processing unit 10B sets the window size used in the analysis by referring to the window size database 104 and the algorithm database 105 based on the input type of tissue, and acquires the deep learning algorithm 60 used in the analysis.

[0278] In step S28 of the image analysis process, the processing unit 10B transmits the three-value image 83 of the analysis result to the user terminal device 200C. In the user terminal device 200C, the processing unit 20C outputs the received three-value image 83 of the analysis result to the output unit 27.

[0279] In step S29 during the image analysis process, the processing unit 10B, following step S28, further performs a process of detecting the region of the tumor cell nucleus on the three-value image 83 of the analysis result. As an arbitrary step, the processing unit 10B creates a tumor cell nucleus region-enhanced image 84 by superimposing the obtained tumor cell nucleus region on the analysis target image 78 of the analysis target. The processing unit 10B transmits the created tumor cell nucleus region-enhanced image 84 to the terminal device 200C on the user side. In the terminal device 200C on the user side, the processing unit 20C outputs the received tumor cell nucleus region-enhanced image 84 to the output unit 27, and ends the image analysis process.

[0280] As described above, the user of the terminal device 200C can obtain the ternary image 83 as an analysis result by sending the analysis target image 78 of the analysis target tissue to the image analysis device 100B. Furthermore, the user of the terminal device 200C can obtain the tumor cell nucleus region enhanced image 84 as an analysis result.

[0281] According to the image analysis device 100B according to the third embodiment, the user can enjoy the results of the image analysis processing without obtaining the window size database 104 and the algorithm database 105 from the deep learning device 100A. As a result, as a service for analyzing the tissue to be analyzed, a service for identifying the region of the tumor cell nucleus and indicating the tumor state can be provided as a cloud service.

[0282] The number of pathologists is insufficient nationwide. Pathologists are registered in large hospitals in urban areas, but are almost absent in remote medical institutions or relatively small medical institutions such as clinics even in urban areas. The cloud service provided by the image analysis device 100B and the terminal device 200C assists histological diagnosis or cytological diagnosis in such remote or relatively small medical institutions.

[0283] <Other forms>

[0284] As mentioned above, although this invention was demonstrated based on an outline and specific embodiment, this invention is not limited to the above-mentioned outline and each embodiment.

[0285] In the first to third embodiments described above, a case of a gastric tumor is described as an example, but the specimen to be processed is not limited thereto, and a specimen including the above-mentioned tissue specimen or cell specimen can be used.

[0286] In the first to third embodiments, in step S14, the processing units 10A, 20B, 10B refer to the window size database 104 to set the number of pixels of the window size, but the operator or user may directly set the window size. In this case, the window size database 104 is unnecessary.

[0287] In the first to third embodiments described above, in step S14, the processing units 10A, 20B, and 10B set the number of pixels of the window size based on the input tissue type, but the size of the tissue may be input instead of the input of the tissue type. The processing units 10A, 20B, and 10B may set the number of pixels of the window size based on the input tissue size by referring to the window size database 104. In step S22, similarly to step S14, the size of the tissue may be input instead of the input of the tissue type. The processing units 20A, 20B, and 10B may set the number of pixels of the window size based on the input tissue size by referring to the window size database 104 and the algorithm database 105, and obtain the neural network 60.

[0288] Regarding the implementation method of inputting the size of the tissue, the size can be directly input as a numerical value, or the input user interface can be used as a pull-down menu, allowing the user to select a specified numerical range corresponding to the size to be input by the user.

[0289] In addition, in step S14 and step S22, in addition to the type of tissue or the size of tissue, the imaging magnification when the first training image 70C, the second training image 71C, the third training image 70NC, the fourth training image 71NC, and the analysis target image 78 are imaged may be input. In the embodiment of inputting the imaging magnification, the magnification may be directly input as a numerical value, or the user interface for input may be a pull-down menu, for example, so that the user selects a specified numerical range corresponding to the magnification to be input by the user.

[0290] In the first to third embodiments described above, the window size is set to 3×3 pixels for convenience of explanation during deep learning processing and image analysis processing, but the number of pixels of the window size is not limited thereto. The window size may also be set, for example, in accordance with the type of tissue sample or the type of cell-containing sample. In this case, the product of the number of pixels of the window size and the number of primary colors of the color contained in the image corresponds to the number of nodes of the input layer 50a, 60a of the neural network 50, 60.

[0291] In step S14 , the processing units 10A, 20B, and 10B obtain the number of pixels of the window size, and may correct the obtained number of pixels of the window size based on the input imaging magnification.

[0292] In the first to third embodiments described above, in step S18, the processing units 10A, 20B, 10B associate the deep learning algorithm 60 with the type of tissue in a one-to-one manner and record the association in the algorithm database 105. Alternatively, in step S18, the processing units 10A, 20B, 10B associate one deep learning algorithm 60 with a plurality of types of tissues and record the association in the algorithm database 105.

[0293] In the first to third embodiments described above, the hue is defined by a combination of the three primary colors of light or the three primary colors of color, but the number of hues is not limited to three. The number of hues may be four primary colors, namely, red (R), green (G), and blue (B) plus yellow (Y), or two primary colors, namely, a hue of any one of the three primary colors of red (R), green (G), and blue (B). Alternatively, it may be one primary color, namely, only one of the three primary colors of red (R), green (G), and blue (B) (for example, green (G)). The bright field images 70C, 70NC and the analyzed object image 78 obtained using, for example, a known microscope or a virtual slice scanner are not limited to color images of the three primary colors of red (R), green (G), and blue (B), but may be color images of two primary colors, as long as they are images containing one or more primary colors.

[0294] In the above-mentioned first to third embodiments, in step S10, the processing units 10A, 20B, 10B generate the color density symbolization diagrams 72r, 72g, 72b as single color images of three levels of each primary color, but the color levels of the primary colors of the color density symbolization diagrams 72r, 72g, 72b are not limited to three levels. The color levels of the color density symbolization diagrams 72r, 72g, 72b may also be images of two levels, as long as they are images of one level or more. Similarly, in step S21, the processing units 20A, 20B, 10B generate the single color images of each primary color of each color density symbolization diagram 79r, 79g, 79b, but the color levels of the primary colors when making the color density symbolization diagram are not limited to three levels. The primary colors when making the color density symbolization diagram may also be images of two levels, as long as they are images of one level or more. Illustratively, the color scale of the color density symbolization diagrams 72r, 72g, 72b, 79r, 79g, and 79b may be 256 levels (8 levels) with color density values ​​ranging from 0 to 255.

[0295] In the first to third embodiments described above, in step S10, the processing units 10A, 20B, 10B generate color density symbolized graphs 72r, 72g, 72b of each color R, G, and B from the input first training image 70C or the third training image 70NC, but the input first training image 70C may be pre-scaled. That is, the processing units 10A, 20B, 10B may directly obtain the color density symbolized graphs 72r, 72g, 72b of each color R, G, and B from, for example, a virtual slide scanner. Similarly, in step S21, the processing units 20A, 20B, 10B generate color density symbolized graphs 79r, 79g, 79b of each color R, G, and B from the input analysis target image 78, but the input analysis target image 78 may be pre-scaled. That is, the processing units 20A, 20B, and 10B may directly obtain the color density symbolization maps 79r, 79g, and 79b of the respective colors of R, G, and B from, for example, a virtual slide scanner or the like.

[0296] In the first to third embodiments, RGB is used as the color space when generating the color density symbolization diagrams 72 and 79 from the first color training images 70C and 78, but the color space is not limited to RGB. In addition to RGB, YUV, CMY, and CIE L * a * b * Various color spaces.

[0297] In the first to third embodiments described above, in the tumor part training data 74C and the analysis data 80, the concentration value for each pixel is stored in the order of red (R), green (G), and blue (B), but the order of storing and processing the concentration values ​​is not limited to this. For example, the concentration values ​​may be stored in the order of blue (B), green (G), and red (R), as long as the order of the concentration values ​​in the tumor part training data 74C and the non-tumor part training data 74NC is the same as the order of the concentration values ​​in the analysis data 80.

[0298] In the first to third embodiments described above, in step S12, the processing units 10A, 20B, 10B binarize the tone levels of the pixels of the input second training image 71C and the fourth training image 71NC to generate the second preliminary training data 71CN and the fourth preliminary training data 71NCN. However, the second preliminary training data 71CN and the fourth preliminary training data 71NCN that have been binarized in advance may be obtained.

[0299] In the first to third embodiments described above, the processing units 10A and 10B are implemented as an integrated device, but the processing units 10A and 10B do not necessarily need to be an integrated device, and the CPU 11, the memory 12, the recording unit 13, etc. may be arranged elsewhere and connected via a network. The processing units 10A and 10B, the input unit 16, and the output unit 17 do not necessarily need to be arranged in one place, and may be arranged separately and connected to each other via network communication. The same is true for the processing units 20A, 20B, and 20C as for the processing units 10A and 10B.

[0300] In the first to third embodiments, the functional blocks of the training data generation unit 101, the training data input unit 102, the algorithm update unit 103, the analysis data generation unit 201, the analysis data input unit 202, the analysis unit 203, and the tumor cell nucleus region detection unit 204 are executed in a single CPU 11 or a single CPU 21, but these functional blocks do not necessarily need to be executed in a single CPU, and can also be executed in a distributed manner using multiple CPUs. In addition, these functional blocks can be executed in a distributed manner using multiple GPUs, or can be executed in a distributed manner using multiple CPUs and multiple GPUs.

[0301] In the second and third embodiments described above, Fig.10 and Fig.13 The program of the processing of each step described in the above is recorded in advance in the recording unit 13, 23. Alternatively, the program may be installed in the processing unit 10B, 20B from a computer-readable and non-temporary tangible recording medium 98 such as a DVD-ROM or a USB memory. Alternatively, the processing unit 10B, 20B may be connected to the network 99, and the program may be downloaded from an external server (not shown) via the network 99 and installed.

[0302] In the first to third embodiments described above, the input units 16 and 26 are input devices such as a keyboard or a mouse, and the output units 17 and 27 are implemented as display devices such as a liquid crystal display. Alternatively, the input units 16 and 26 and the output units 17 and 27 may be integrated and implemented as a touch panel type display device. Alternatively, the output units 17 and 27 may be composed of a printer or the like, and the three-value image 83 of the analysis result or the tumor cell nucleus region enhanced image 84 of the cell nucleus may be printed and output.

[0303] In the first to third embodiments described above, the imaging device 300 is directly connected to the deep learning device 100A or the image analysis device 100B, but the imaging device 300 may be connected to the deep learning device 100A or the image analysis device 100B via the network 99. Similarly, the imaging device 400 is directly connected to the image analysis device 200A or the image analysis device 200B, but the imaging device 400 may be connected to the image analysis device 200A or the image analysis device 200B via the network 99.

[0304] <Example>

[0305] Next, embodiments of the present invention will be described to make the features of the present invention more clear.

[0306] [Example 1]

[0307] The independent system shown in the second embodiment is used to perform deep learning processing and image analysis processing. The tissues to be learned and analyzed are cancerous tissue and non-cancerous tissue obtained from the stomach. Non-cancerous tissue is an area of ​​tissue that is inflamed but not judged as cancer. Image analysis processing using the learned deep learning algorithm is performed on each of an image known to contain cancerous tissue and an image known to contain non-cancerous tissue, and the analysis results are confirmed to be suitable.

[0308] The details of the learning data and analysis data are as follows.

[0309] Learning data:

[0310] The whole slice image 512 is divided into 4 images (details: 2 for the area containing cancer cell nuclei and 2 for the area without cancer cell nuclei)

[0311] Verify the parsed data:

[0312] The whole slice image 512 is divided into two images (details: one for the area containing cancer cell nuclei and one for the area without cancer cell nuclei)

[0313] [Creation and learning of training data]

[0314] The whole slice image (WSI) of the bright field image of the gastric cancer tissue stained with HE and the fluorescent image of the gastric cancer tissue stained with DAPI is color photographed using a virtual slice scanner. The imaging magnification is 40 times. Thereafter, based on the bright field image, the color concentration values ​​of each color of R, G, and B are color-graded to make a color concentration symbolization diagram of each color of R, G, and B. In addition, the pathologist visually judges the whole whole slice image of the area containing the bright field image of the tissue containing the learning object, and in the whole slice image, the area of ​​cancer cells and the area without cancer cells are specified. Thereafter, by distinguishing the area designated as the nuclear area of ​​cancer cells and the area designated as the nuclear area without cancer cells, the value (binary value) of the distinguishing area is assigned to each area. The area of ​​the bright field image of the tissue containing the learning object is cut out from the whole slice image to generate the first preliminary training data and the third preliminary training data.

[0315] Furthermore, as the tissue to be learned, a region including the nuclei of cancer cells and a region not including the nuclei of cancer cells are used, and a plurality of regions are cut out from the whole slice image to obtain a binary image.

[0316] In addition, based on the fluorescent image stained with DAPI, the color density value is binarized into the region of the cell nucleus and the region other than the region using a preset threshold value to prepare the second preliminary training data and the fourth preliminary training data as the binary image. The second training data and the fourth training data as the true value image are prepared from the binary image showing the cancerous state prepared by visually judging the bright field image by the pathologist and the binary image showing the region of the cell nucleus prepared from the fluorescent image.

[0317] Thereafter, the first training data and the second training data, which are color density symbolization images of tumor tissue, are combined to generate tumor part training data. In addition, the third training data and the fourth training data, which are color density symbolization images of non-tumor tissue, are combined to generate non-tumor part training data. The generated training data is divided into a window size of 113×113 pixels, and the training data of the divided window size is used as the input layer to make the neural network learn. The 113×113 pixels used as the window size is, by way of example, a size that supports the overall shape of the nucleus region of at least one cell among a plurality of cells ranging from 2 to 9 being contained within the window.

[0318] [Creation of analysis target image]

[0319] Similar to the training data, the whole slice image of the bright field image of the HE-stained gastric cancer tissue was color photographed using a virtual slice scanner. The imaging magnification was 40 times. Afterwards, a color density symbolization map of each color of R, G, and B was prepared based on the photographed bright field image, and the prepared color density symbolization map of each color of R, G, and B was combined to prepare the analysis object image.

[0320] [Analysis results]

[0321] The analysis data with a window size of 113×113 pixels is generated with each pixel of the analysis target image as the center, and the generated analysis data with the window size is input to the learned neural network. Based on the analysis results output from the neural network, the areas are classified into cancer cell nuclei, non-cancerous cell nuclei, and areas other than cell nuclei. The outline of the cancer cell nuclei area is surrounded by white, and the outline of the non-cancerous cell nuclei area is surrounded by black. The analysis results are shown in Fig.18 and Fig.19 .

[0322] Fig.18 This is the analysis result of an image (HE staining) of a cancer tissue specimen obtained from the stomach. Fig.18 In the figure, (a) is a bright field image of a cancerous tissue stained with HE, which is an image of the object of analysis. (b) is an image in which the outline of the cancer cell nucleus and the outline of the non-cancerous cell nucleus obtained by the analysis process are overlapped on the bright field image of (a). (c) is an image of the area surrounded by the white dotted line in (b) enlarged. Fig.18 In (b) and (c), the area surrounded by white is the area of ​​the cancer cell nucleus, and the area surrounded by black is the area of ​​the non-cancerous cell nucleus.

[0323] Since the image to be analyzed is known to contain cancerous tissue, if the area of ​​the cancer cell nucleus is surrounded by white, the analysis is appropriate, and if the area of ​​the cancer cell nucleus is mistakenly surrounded by black, the analysis is inappropriate. Fig.18 As shown in (b), although some areas surrounded by black appear, the nucleus of cancer cells is mostly surrounded by white, and it can be confirmed that the pixels in the nucleus area to be judged as cancer cells are appropriately and correctly judged.

[0324] Fig.19 This is the analysis result of an image (HE staining) of a non-cancerous tissue specimen obtained from the stomach. Fig.19 In the figure, (a) is a bright field image of a non-cancerous tissue stained with HE, which is an image of the object of analysis. (b) is an image in which the outline of the cancer cell nucleus and the outline of the non-cancerous cell nucleus obtained by the analysis process are overlapped on the bright field image of (a). (c) is an image of the area surrounded by the white dotted line in (b) enlarged. Fig.19 In (b) and (c), the area surrounded by white is the area of ​​the cancer cell nucleus, and the area surrounded by black is the area of ​​the non-cancerous cell nucleus.

[0325] Since the image to be analyzed is known to contain non-cancerous tissue, if the area of ​​the non-cancerous cell nucleus is surrounded by black, the analysis is appropriate, and if the area of ​​the non-cancerous cell nucleus is mistakenly surrounded by white, the analysis is inappropriate. Fig.19As shown in (b), there is almost no area surrounded by white, and the area of ​​the non-cancer cell nucleus is almost entirely surrounded by black, which confirms that the pixel in the nucleus area to be judged as non-cancer cell is appropriately and correctly judged.

[0326] [Example 2]

[0327] The analysis results using the learned neural network created in the above-mentioned Example 1 and the analysis results using the existing machine learning algorithm were compared.

[0328] In the existing machine learning algorithm, a support vector machine is used. In the training of the support vector machine, the same training data as the training data used in the training of the neural network in the above-mentioned embodiment 1 is used. The image of the analysis object is the same as the image used in the above-mentioned embodiment 1. The analysis result is shown in Figure 20 to Figure 23 . Furthermore, in Fig. 20 and Fig.21 In FIG. 1 , the image displayed as the analysis result by the neural network is an image in which the label value indicating the cancerous state is separated into three images with respective values.

[0329] Fig. 20 This is the analysis result of an image (HE staining) of a cancer tissue specimen obtained from the stomach as the image to be analyzed by the learned neural network. Fig. 20 In the figure, (a) is a bright field image of the object to be analyzed. (b) is an image showing the nucleus of the area judged to be a cancer cell, (c) is an image showing the nucleus of the area judged to be a non-cancer cell, and (d) is an image showing the area other than the nucleus. Fig. 20 In (b), the white area is the area judged to be the nucleus of the cancer cell. Fig. 20 In (c), the white area is the nuclear area determined to be non-cancerous cell.

[0330] Since the image to be analyzed is known to contain cancerous tissue, Fig. 20 In (b), if the area of ​​the cancer cell nucleus is shown in white, the analysis is appropriate. Fig. 20 As shown in (b), the region of the cancer cell nucleus is appropriately indicated in white. Fig. 20 In (c), there are few areas that are erroneously represented in white. Therefore, it can be confirmed that the area of ​​cancer cell nuclei and the area of ​​non-cancerous cell nuclei can be appropriately and clearly classified for an image to be analyzed that is known to contain cancerous tissue.

[0331] Fig.21 This is the analysis result of an image (HE staining) of a non-cancerous tissue specimen obtained from the stomach as the analysis target by the learned neural network. Fig.21In the figure, (a) is a bright field image of the object to be analyzed. (b) is an image showing the nucleus of the area judged to be a cancer cell, (c) is an image showing the nucleus of the area judged to be a non-cancer cell, and (d) is an image showing the area other than the nucleus. Fig.21 In (b), the white area is the area judged to be the nucleus of the cancer cell. Fig.21 In (c), the white area is the nuclear area determined to be non-cancerous cell.

[0332] Since the image to be analyzed is known to contain non-cancerous tissue, Fig.21 In (b), if the non-cancerous cell nuclei are mistakenly represented in white, the analysis is inappropriate. Fig.21 In (b), the areas that are incorrectly represented in white are small. Fig.21 As shown in (c), the non-cancer cell nucleus region is appropriately indicated in white. Therefore, it can be confirmed that even for an image to be analyzed that is known to contain non-cancer tissue, the cancer cell nucleus region and the non-cancer cell nucleus region can be appropriately and clearly classified.

[0333] Fig. 22 The following is a result of analysis using a support vector machine, using an image of a cancer tissue specimen (HE staining) obtained from the stomach as an analysis target, as a comparative example. Fig. 22 In the figure, (a) is a bright field image of the object to be analyzed. (b) is an image showing a region of the nucleus determined to be a cancer cell, and (c) is an image showing a region of the nucleus determined to be a non-cancerous cell. In addition, in each of (a) to (c), an image of a portion of the image shown in (i) is enlarged and shown in (ii). The position of the enlarged region is common in (a) to (c).

[0334] Fig.23 The following is a result of analysis using a support vector machine, using an image of a non-cancerous tissue specimen (HE staining) obtained from the stomach as an analysis target, as a comparative example. Fig.23 In the figure, (a) is a bright field image of the object to be analyzed. (b) is an image showing a region of the nucleus determined to be a cancer cell, and (c) is an image showing a region of the nucleus determined to be a non-cancerous cell. In addition, in each of (a) to (c), an image of a portion of the image shown in (i) is enlarged and shown in (ii). The position of the enlarged region is common in (a) to (c).

[0335] Reference Fig. 22 (b) and (c) and Fig.23(b) and (c), it is considered that there is a tendency for either the cancer cell nucleus region or the non-cancerous cell nucleus region to be grasped. However, in the analysis by the support vector machine as a comparative example, the cancer cell nucleus region and the non-cancerous cell nucleus region cannot be clearly classified. Therefore, it can be confirmed that the analysis using the learned neural network prepared in Example 1 has an advantageous effect over the analysis using the support vector machine as an existing machine learning algorithm.

[0336]

Explanation of symbols

[0337] 10 (10A, 10B): Processing unit

[0338] 20 (20A, 20B, 20C): Processing unit

[0339] 11, 21: CPU

[0340] 12, 22: Memory

[0341] 13, 23: Recording Department

[0342] 14, 24: Bus

[0343] 15, 25: Interface

[0344] 16, 26: Input

[0345] 17, 27: Output section

[0346] 19, 29: GPU

[0347] 50: Neural Networks (Deep Learning Algorithms)

[0348] 50a: Input layer

[0349] 50b: Output layer

[0350] 50c: middle layer

[0351] 60: Completed Neural Network Learning (Completed Deep Learning Algorithm Learning)

[0352] 60a: Input layer

[0353] 60b: Output layer

[0354] 60c: Middle layer

[0355] 70W: Full slice image

[0356] 70C, 70NC: Bright field images for learning (first training image)

[0357] 70CL: 1st Preparatory Training Data

[0358] 70NCL: 3rd Preparatory Training Data

[0359] 71C, 71NC: Fluorescence images for learning (second training images)

[0360] 71CN, 71NCN: Second preparatory training data

[0361] 72r, 72g, 72b: Color density symbolization diagrams in single color images of R, G, and B colors of the bright field image for learning (first training data)

[0362] 73C, 73NC: True value images for learning (ternary images, second training data)

[0363] 74C, 74NC: training data

[0364] 74a: Color density value of the bright field image

[0365] 74b: Label value of the true value image

[0366] 75C, 75NC: training data of window size

[0367] 76: Color density value

[0368] 77: Label value of the true value image

[0369] 78: Brightfield images of parsed objects

[0370] 79r, 79g, 79b: Color density symbolization diagrams in single images of R, G, and B colors of the bright field image of the analysis object

[0371] 80: Data for analysis

[0372] 81: Color density value

[0373] 82: Determination result (estimated value of pixel)

[0374] 83: Ternary image of analytical results

[0375] 84: Tumor cell nuclear region emphasized image

[0376] 89(89a, 89b): Node

[0377] 98: Recording medium

[0378] 99: Network

[0379] 100: Supplier-side device

[0380] 100A: Deep Learning Device

[0381] 100B: Integrated image analysis device

[0382] 101: Training data generation section

[0383] 102: Training data input section

[0384] 103: Algorithm Update Department

[0385] 104: Window size database

[0386] 105: Algorithm Database

[0387] 200: User-side device

[0388] 200A: Image analysis device

[0389] 200B: Integrated image analysis device

[0390] 200C: Terminal device

[0391] 201: Analysis data generation unit

[0392] 202: Analysis data input unit

[0393] 203: Analysis Department

[0394] 204: Cell nuclear region detection unit

[0395] 300, 400: Camera device

[0396] 301, 401: Camera components

[0397] 302, 402: Fluorescence Microscopy

[0398] 308, 408: Sample organization

[0399] 309, 409: mirror table

[0400] W1: Window

[0401] W2: Window

Claims

1. An image analysis method, using a deep learning algorithm of a neural network structure to analyze an image of a tissue, the image analysis method comprising: generating analysis data corresponding to an analysis target image including a plurality of cell images obtained by imaging a case specimen in which a thin slice of an analysis target tissue is placed on a glass slide, inputting the analytical data into the deep learning algorithm, generating data representing the tumorigenic state of the tissue in the analysis target image by the deep learning algorithm, A tumor cell nucleus region enhanced image including the plurality of cell images, in which a tumor cell nucleus region and a region other than the tumor cell nucleus in the analysis target image can be distinguished, is displayed based on the analysis target image and the data indicating the tumorization state.

2. The image analysis method according to claim 1, wherein The analysis target image is a full slice image.

3. The image analysis method according to claim 1, wherein The data indicating the tumorigenic state is data indicating whether the nuclear regions of the plurality of cells included in the analysis target image are nuclear regions of tumor cells or nuclear regions of non-tumor cells.

4. The image analysis method according to claim 1, wherein The data representing the tumorigenic state are labels for distinguishing between nuclear regions of tumor cells, nuclear regions of non-tumor cells, and regions that are neither tumor cells nor non-tumor cells.

5. The image analysis method according to claim 1, wherein For one of the analysis target images, the analysis data is generated for each region having a designated number of pixels, and the generated plurality of analysis data are input to the deep learning algorithm.

6. The image analysis method according to claim 5, wherein The area with the designated number of pixels has a size capable of accommodating a plurality of cell images.

7. The image analysis method according to claim 5, wherein An input of a type of the tissue to be analyzed is received, and the designated number of pixels is set according to the type of the tissue.

8. The image analysis method according to claim 1, wherein The deep learning algorithm receives an input of a type of the tissue to be analyzed and acquires the deep learning algorithm, which is an input destination of the analysis data, from a database according to the type of the tissue.

9. The image analysis method according to claim 1, wherein The tissue to be analyzed is a tissue obtained from the stomach.

10. The image analysis method according to claim 1, wherein The deep learning algorithm is a deep learning algorithm trained using training data including first training data and second training data, wherein the first training data corresponds to an image of a tumor area included in a whole slice image obtained by photographing a case specimen in which a thin slice of tissue containing cancerous tissue is placed on a slide, and the second training data corresponds to an image of a non-tumor area included in the whole slice image.

11. An image analysis device that uses a deep learning algorithm with a neural network structure to analyze an image of a tissue, the image analysis device comprising: Processing Department; as well as Display unit, The processing unit: generating analysis data corresponding to an analysis target image including a plurality of cell images obtained by imaging a case specimen in which a thin slice of an analysis target tissue is placed on a glass slide, inputting the analytical data into the deep learning algorithm, generating data representing the tumorigenic state of the tissue in the analysis target image by the deep learning algorithm, generating a tumor cell nucleus region enhanced image including the plurality of cell images, the image capable of distinguishing a tumor cell nucleus region and a region other than the tumor cell nucleus in the image under analysis, based on the image under analysis and the data indicating the tumorization state; The display unit displays the tumor cell nuclear region enhanced image.

12. A computer program that uses a deep learning algorithm with a neural network structure to analyze an image of a tissue, the program causing a computer to perform the following processing: generating analysis data corresponding to an analysis target image including a plurality of cell images obtained by imaging a case specimen in which a thin slice of an analysis target tissue is placed on a glass slide, inputting the analytical data into the deep learning algorithm, generating data representing the tumorigenic state of the tissue in the analysis target image by the deep learning algorithm, A tumor cell nucleus region enhanced image including the plurality of cell images, capable of distinguishing a tumor cell nucleus region and a region other than the tumor cell nucleus in the analysis target image, is generated based on the analysis target image and the data indicating the tumorization state.

Images