Image analysis method, device, program, and method for manufacturing trained deep learning algorithm

Abstract

To provide a method for analyzing an image that generates data indicating the region of a cellular nucleus in an image of a tissue or a cell.SOLUTION: The method for analyzing an image is for analyzing an image of a tissue or a cell, using a deep learning algorithm 60 of a neural network structure. The method includes: generating analysis data 80 from an analysis target image 78 including the tissue or the cell to be analyzed; inputting the analysis data 80 into the deep learning algorithm 60; and generating data 82 and 83 showing the region of a cellular nucleus in the analysis target image 78 by the deep learning algorithm 60.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image analysis method, an apparatus, a program, and a method for producing a trained deep learning algorithm, and more particularly to an image analysis method that includes generating data indicating the area of ​​a cell nucleus for any position in an image of a tissue or cell. [Background technology]

[0002] Patent Document 1 discloses an image diagnosis support device that classifies and assesses tissue images in pathological tissue images into four groups: normal, benign tumor, precancerous state, and cancerous state. An image classification unit extracts a region of interest from image data, calculates feature quantities that indicate the characteristics of the region of interest, and classifies the group based on the calculated feature quantities. The feature quantities include the density of clusters per unit area in the cell nucleus, the density of the cluster area, the cluster area, the cluster thickness, and the cluster length. An image assessment unit learns the relationship between these feature quantities and assessment results and makes assessments based on the learned learning parameters. Learning is performed using machine learning algorithms such as support vector machines. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-203949 Summary of the Invention [Problem to be solved by the invention]

[0004] To definitively diagnose whether a tumor is malignant or not, histopathological diagnosis is performed using pathological tissue specimens. Histopathological diagnosis is also often performed as intraoperative rapid diagnosis to determine the area of ​​tissue containing malignant tumors to be resected during surgery. Intraoperative rapid diagnosis involves having the patient wait with the affected area incised during surgery, and using histopathological diagnosis to determine whether the tumor is malignant, whether any tumor remains at the margins of the resected tissue, whether there is lymph node metastasis, etc. The results of the intraoperative rapid diagnosis determine the direction of subsequent surgery for the patient.

[0005] Pathological tissue diagnosis is made by doctors, particularly pathologists, by observing tissue specimens under a microscope or other device. However, to be able to make an accurate and definitive diagnosis by observing tissue specimens, it is necessary to repeatedly observe tissue specimens from various cases over a long period of time under the supervision of an experienced pathologist, and the training of pathologists also requires a huge amount of time.

[0006] The shortage of pathologists is serious, and as a result, there are concerns that a definitive diagnosis of a patient's malignant tumor will be delayed, leading to delayed treatment, or that treatment will be started without a definitive diagnosis. Furthermore, because both regular tissue diagnosis and intraoperative rapid diagnosis are concentrated among a limited number of pathologists, the workload for each pathologist is enormous, and the pathologists' own working conditions are also becoming a problem. However, no solution to this problem has yet been found.

[0007] Therefore, the ability of devices to support pathological tissue diagnosis is expected to make a significant contribution to resolving the shortage of pathologists and improving their working conditions, especially when the diagnosis is closer to that made by the human eye.

[0008] In terms of the device supporting histopathological diagnosis, the invention described in the above-mentioned Patent Document 1 performs pathological judgment of sample tissue based on image analysis by machine learning. This method requires that features be created manually. There is a problem that the ability of the person in charge has a significant impact on the performance of image analysis.

[0009] For example, in tissue diagnosis or cytological diagnosis using a microscope, one of the objects of observation is the state of the cell nuclei, and benign tumors are distinguished from malignant tumors based on the size and shape of each cell nucleus, as well as the arrangement of multiple cell nuclei, etc. For this reason, the ability to accurately extract cell nuclei is extremely important in pathological tissue diagnosis and is the foundation of tissue diagnosis and cytological diagnosis.

[0010] An object of the present invention is to provide an image analysis method for generating data indicating the area of ​​a cell nucleus from an image of a tissue or cell. [Means for solving the problem]

[0011] One aspect of the present invention is an image analysis method. In this aspect, the image analysis method uses a deep learning algorithm (60) with a neural network structure to analyze an image of tissue or cells, and includes generating analysis data (80) from an image to be analyzed (78) containing the tissue or cell to be analyzed (S21 to S23), inputting the analysis data (80) into the deep learning algorithm (60) (S24), and generating data (82, 83) indicating the area of ​​a cell nucleus in the image to be analyzed (78) by the deep learning algorithm (60) (S25 to S28). This makes it possible to generate data indicating the area of ​​a cell nucleus for any position in the image of the tissue or cell.

[0012] It is preferable that the image to be analyzed is an image of a tissue diagnostic specimen, and that the image to be analyzed (78) contains a hue consisting of one primary color or a hue (R, G, B) that is a combination of two or more primary colors.

[0013] It is preferable that the image to be analyzed is an image of a cytological specimen, and that the image to be analyzed (78) contains a hue consisting of one primary color or a hue (R, G, B) that is a combination of two or more primary colors.

[0014] The data (82, 83) indicating the area of ​​the cell nucleus is preferably data for distinguishing and presenting the area of ​​the cell nucleus from other areas.

[0015] The data (82, 83) indicating the area of ​​the cell nucleus is preferably data indicating the boundary between the area of ​​the cell nucleus and other areas.

[0016] The deep learning algorithm (60) preferably determines whether a given location in the image (78) to be analyzed is a region of a cell nucleus.

[0017] It is preferable to generate a plurality of analysis data 80 for each region of a predetermined number of pixels for one analysis target image 78. This makes it possible to improve the discrimination accuracy of the neural network 60.

[0018] The analysis data (80) is preferably generated for each region of a predetermined number of pixels centered around a predetermined pixel and including surrounding pixels, and the deep learning algorithm (60) preferably generates a label indicating whether the predetermined pixel in the input analysis data (80) is a cell nucleus region or not, thereby improving the discrimination accuracy of the neural network (60).

[0019] It is preferable that the number of nodes in the input layer (60a) of the neural network (60) corresponds to the product of the predetermined number of pixels in the analysis data (80) and the number of combined primary colors, thereby improving the discrimination accuracy of the neural network (60).

[0020] Preferably, the specimen is a stained specimen, and the image to be analyzed (78) is an image of the stained specimen captured under a bright field microscope.

[0021] Preferably, the training data (74) used for learning the deep learning algorithm (60) is generated based on a bright-field image (70) obtained by capturing a stained image of a specimen prepared by staining a tissue sample collected from an individual or a sample containing cells collected from an individual with a bright-field observation stain under a bright-field microscope, and a fluorescent image (71) of a cell nucleus obtained by capturing a stained image of a specimen corresponding to or identical to the specimen, with a fluorescent nuclear stain under fluorescent observation under a fluorescent microscope, the position of the fluorescent image (71) within the specimen corresponding to the position of the acquired bright-field image (70) within the specimen.

[0022] For staining for bright-field observation, hematoxylin is preferably used for nuclear staining.

[0023] Preferably, when the specimen is a tissue sample, the stain for brightfield observation is hematoxylin and eosin stain, and when the specimen is a cell-containing sample, the stain for brightfield observation is Papanicolaou stain.

[0024] The training data (74) preferably includes label values ​​indicating the regions of cell nuclei extracted from the bright-field image (70) and the fluorescent image (71), thereby enabling the neural network (50) to learn the label values ​​indicating the regions of cell nuclei.

[0025] The training data (74) preferably includes a label value for each pixel in the bright-field image (70), allowing the neural network (50) to learn label values ​​that indicate the area of ​​the cell nucleus.

[0026] It is preferable that the training data (74) be generated for each region of a predetermined number of pixels in the bright-field image (70), thereby enabling the neural network (50) to learn label values ​​indicating cell nucleus regions with high accuracy.

[0027] Preferably, the deep learning algorithm (60) classifies the analysis data (80) into classes that indicate the areas of cell nuclei contained in the image (78) to be analyzed, thereby making it possible to classify any position in the tissue image or image containing cells to be analyzed into areas of cell nuclei and other areas.

[0028] The output layer (60b) of the neural network (60) preferably has nodes with a softmax function as an activation function, which enables the neural network (60) to classify any position in an image containing tissue or cells to be analyzed into a finite number of classes.

[0029] Each time analysis data 80 is input, the deep learning algorithm 60 preferably generates data 82 for each unit pixel indicating whether the pixel is a cell nucleus region included in the analysis target image 78. This makes it possible to classify each unit pixel (one pixel) of the tissue image or image containing cells to be analyzed into a cell nucleus region and other regions.

[0030] It is preferable that the deep learning algorithm (60) is generated depending on the type of tissue sample or the type of sample containing cells, which makes it possible to use different deep learning algorithms (60) depending on the type of tissue image or image containing cells to be analyzed, thereby improving the discrimination accuracy of the neural network (60).

[0031] Furthermore, multiple deep learning algorithms are used depending on the type of tissue sample or the type of sample containing cells. It is preferable to process the analysis data (80) using a deep learning algorithm (60) that corresponds to the type of sample, selected from the algorithms (60). This makes it possible to use different deep learning algorithms (60) depending on the type of tissue image or image containing cells to be analyzed, thereby improving the discrimination accuracy of the neural network (60).

[0032] One aspect of the present invention is an image analysis device. In this aspect, the image analysis device (200A) is an image analysis device that analyzes an image of tissue or cells using a deep learning algorithm (60) with a neural network structure, and includes a processing unit (20A) that generates analysis data (80) from an analysis target image (78) containing the tissue or cells to be analyzed, inputs the analysis data (80) to the deep learning algorithm (60), and generates data (82, 83) indicating the region of a cell nucleus in the analysis target image (78) using the deep learning algorithm (60). This makes it possible to generate data indicating whether an arbitrary position in an image of tissue or cells is the region of a cell nucleus.

[0033] One aspect of the present invention is a computer program. In this aspect, the computer program is a computer program for analyzing an image of tissue or cells using a deep learning algorithm (60) with a neural network structure, and causes a computer to execute the following processes: generating analysis data (80) from an image to be analyzed (78) including the tissue or cell to be analyzed; inputting the analysis data (80) into the deep learning algorithm (60); and generating data (82, 83) indicating the area of ​​a cell nucleus in the image to be analyzed (78) using the deep learning algorithm (60). This makes it possible to generate data indicating the area of ​​a cell nucleus for any position in the image of the tissue or cell.

[0034] One aspect of the present invention is a method for manufacturing a trained deep learning algorithm. In this aspect, the method for manufacturing a trained deep learning algorithm (60) includes a first acquisition step of acquiring first training data (72r, 72g, 72b) corresponding to a first training image (70) of tissue or cells, a second acquisition step of acquiring second training data (73) corresponding to a second training image (71) showing a region of a cell nucleus in the first training image (70), and a learning step (S13 to S19) of training a neural network (50) to learn the relationship between the first training data (72r, 72g, 72b) and the second training data (73). This makes it possible to manufacture a deep learning algorithm for generating data showing a region of a cell nucleus for any position in an image of tissue or cells.

[0035] It is preferable that the first training data (72r, 72g, 72b) be used as the input layer (50a) of the neural network (50), and the second training data (73) be used as the output layer (50b) of the neural network (50).

[0036] Preferably, the method further includes a step (S11) of generating first training data (72r, 72g, 72b) from a first training image (70) before the first acquisition step, and a step (S12) of generating second training data (73) from a second training image (71) before the second acquisition step, thereby enabling the development of a deep learning algorithm for generating data indicating the area of ​​a cell nucleus for any position in an image of a tissue or cell.

[0037] The first training image (70) is a bright-field image (70) of a stained image of a specimen prepared by staining a tissue sample collected from an individual or a sample containing cells collected from an individual with a stain for bright-field observation, captured under a bright-field microscope; the second training image (71) is a fluorescent image (71) of a stained image of a specimen prepared by staining a specimen with a fluorescent nuclear stain, captured under fluorescent microscope observation. ) and the fluorescent image (71) is preferably a fluorescent image (71) whose position within the specimen corresponds to the position within the specimen of the acquired bright-field image (70).

[0038] One aspect of the present invention is a trained deep learning algorithm (60), which is trained using first training data (72r, 72g, 72b) as an input layer (50a) of a neural network (50) and second training data (73) as an output layer (50b) of the neural network (50), wherein the first training data (72r, 72g, 72b) are generated from first training images (70) of tissues or cells, and the second training data (73) indicates the regions of cell nuclei in the first training images. [Effects of the Invention]

[0039] According to the present invention, data indicating the area of ​​the cell nucleus can be generated for any position in an image of a tissue or cell. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a schematic diagram for explaining an overview of a deep learning method. [Figure 2] FIG. 10 is a schematic diagram for explaining details of training data. [Figure 3] FIG. 1 is a schematic diagram for explaining an outline of an image analysis method. [Figure 4] 1 is a schematic configuration diagram of an image analysis system according to a first embodiment. [Figure 5] 2 is a block diagram showing the hardware configuration of a vendor-side device 100. FIG. [Figure 6] FIG. 2 is a block diagram showing the hardware configuration of a user device 200. [Figure 7] FIG. 2 is a block diagram for explaining the functions of a deep learning device 100A according to the first embodiment. [Figure 8] 1 is a flowchart showing the procedure of deep learning processing. [Figure 9] FIG. 10 is a schematic diagram for explaining details of learning by a neural network. [Figure 10] FIG. 2 is a block diagram for explaining the functions of an image analyzing device 200A according to the first embodiment. [Figure 11] 10 is a flowchart showing the procedure of an image analysis process. [Figure 12] FIG. 10 is a schematic configuration diagram of an image analysis system according to a second embodiment. [Figure 13] FIG. 10 is a block diagram for explaining functions of an integrated image analyzing device 200B according to a second embodiment. [Figure 14] FIG. 10 is a schematic configuration diagram of an image analysis system according to a third embodiment. [Figure 15] FIG. 10 is a block diagram for explaining functions of an integrated image analyzing device 100B according to a third embodiment. [Figure 16] 1 shows bright-field images, fluorescent images, and binarized images created from the fluorescent images used to create training data in Example 1. [Figure 17] 1 shows the results of analyzing an image of the first gastric cancer tissue specimen (HE stained) in Example 1. [Figure 18] 1 shows the results of analyzing an image of the second gastric cancer tissue specimen (HE stained) in Example 1. [Figure 19] 1 shows the results of analyzing images of imprint specimens (Papanicolaou stained) of stomach cancer areas in Example 2. [Figure 20] 1 shows the results of analyzing images of imprint specimens (Papanicolaou stained) of non-cancerous stomach areas in Example 2. DETAILED DESCRIPTION OF THE INVENTION

[0041] The outline and embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the following description and drawings, the same reference numerals will denote the same or similar components, and therefore, descriptions of the same or similar components will be omitted.

[0042] The present invention relates to an image analysis method for analyzing images of tissues or cells, which uses a deep learning algorithm with a neural network structure.

[0043] In the present invention, a tissue or cell image is an image obtained from a tissue sample specimen or a cell-containing sample specimen. The tissue sample specimen or cell-containing sample specimen is collected from an individual. The individual is not particularly limited, but is preferably a mammal, more preferably a human. The individual may be alive or dead when the sample is collected from the individual. The tissue is not limited as long as it is present within the individual. Examples of tissue collected from the individual include surgically resected tissue and biopsy tissue. The cell-containing sample is not limited as long as it is collected from an individual. Examples include sputum, pleural effusion, ascites, urine, cerebrospinal fluid, bone marrow, blood, and cyst fluid.

[0044] The specimen is intended to be a tissue sample or a cell-containing sample that has been processed to enable observation under a microscope, such as a slide. The specimen can be prepared according to known methods. For example, in the case of a tissue specimen, after collecting tissue from the individual, the tissue is fixed with a specified fixative (formalin fixation, etc.), the fixed tissue is embedded in paraffin, and the paraffin-embedded tissue is thinly sliced. The thin slices are placed on a glass slide. The slide with the slices is stained for observation under an optical microscope, i.e., for brightfield observation, and the specimen is completed by a specified mounting process. A typical example of a tissue specimen is a tissue diagnostic specimen (pathological specimen), and the stain is hematoxylin-eosin (HE) staining.

[0045] For example, in the case of a specimen of a sample containing cells, the cells in the sample are attached to a slide glass by centrifugation, smearing, etc., fixed with a predetermined fixative (ethanol, etc.), stained for bright-field observation, and mounted as required to complete the specimen. A typical example of a specimen of a sample containing cells is a cytological specimen (cytological diagnosis specimen), which is stained with Papanicolaou staining. The cytological specimen also includes a stamped specimen of tissue collected for the tissue specimen.

[0046] Hematoxylin is used as the nuclear stain in both HE staining and Papanicolaou staining. Hematoxylin is widely used as a nuclear stain in tissue and cell staining (e.g., immunostaining, lectin staining, sugar staining, fat staining, collagen fiber staining, etc.). Therefore, the present invention can be applied to all specimens that use hematoxylin for nuclear staining.

[0047] In the present invention, two types of training images are used during deep learning. One of the training images (first training image) is an image containing tissue or cells contained in a tissue specimen collected from an individual or a cell-containing specimen collected from an individual. This image is acquired from a specimen stained so that the tissue structure or cellular structure can be recognized by microscopic observation. The staining is not limited as long as the tissue structure or cellular structure can be recognized, but is preferably a staining for bright-field observation. The staining for bright-field observation is not limited as long as at least cell nuclei and regions other than cell nuclei can be stained in a distinguishable hue. When the specimen is a mammalian tissue specimen, for example, HE staining can be used. Furthermore, when the specimen is a specimen containing mammalian cells, for example, Papanicolaou staining can be used.

[0048] The second training image (second training image) is an image showing which region in the first training image is the cell nucleus region, i.e., which region in the first training image is the correct answer "cell nucleus region." This image is an image captured by applying a fluorescent nuclear stain that selectively stains cell nuclei to the same specimen as the specimen from which the first training image was obtained, or a specimen corresponding to the specimen from which the first training image was obtained (e.g., a serial section specimen). Examples of the fluorescent nuclear stain include, but are not limited to, 4',6-diamidino-2-phenylindole (DAPI) stain. Color can be mentioned.

[0049] In the example shown in Figure 1, a first training image 70 of HE-stained tissue is used as the first training image, and a second training image 71 of DAPI-stained tissue is used as the second training image, with the specimen position corresponding to the specimen position shown in the first training image 70. Analysis target image 78 of tissue stained for bright field staining, the same as the first training image shown in Figure 3, is used as analysis data for the analysis target used in the image analysis process. The discrimination target that neural network 50 is trained to determine as a correct answer is the region of a cell nucleus contained in a tissue specimen or a specimen containing cells.

[0050] In the overview and embodiments of the present invention, a case will be described as an example in which the area of ​​a cell nucleus contained in an image of an HE-stained tissue specimen is identified using a deep learning algorithm.

[0051] [Overview of deep learning and image analysis methods] Below, we will first provide an overview of the deep learning method and the image analysis method, and then we will provide a detailed description of each of the embodiments of the present invention.

[0052] Overview of deep learning methods As shown in Figure 1, the deep learning method uses training data generated from the first and second training images described above. The first training images are captured as color images of HE-stained specimens under bright-field microscope observation, and therefore contain multiple hues.

[0053] The first training image (bright-field image) 70 can be acquired in advance using an image acquisition device such as a known optical microscope, a fluorescence microscope, or a virtual slide scanner. Illustratively, in this embodiment, the color image acquired from the image acquisition device is preferably in 24-bit RGB color space. In 24-bit RGB color, the intensity (color density) of each of the red, green, and blue colors is preferably expressed in 8 bits (256 levels). The first training image (bright-field image) 70 may be any image that includes one or more primary colors.

[0054] In the present invention, hue is illustratively defined as a combination of the three primary colors of light or the three primary colors of color. The first training data is generated from the first training image 70 by separating the hues appearing in the first training image 70 into individual primary colors, and data is generated for each primary color, and the data is represented by a code corresponding to its density. In FIG. 1, single-color images 72R, 72G, and 72B are obtained by separating each of the three primary colors of light, red (R), green (G), and blue (B).

[0055] When the color density of each pixel in the single-color images 72R, 72G, and 72B is encoded, the entire image becomes coded maps 72r, 72g, and 72b corresponding to the color density of each pixel for each R, G, and B image. Color density may be coded using a numerical value representing 256 levels of each color. Alternatively, the numerical value representing 256 levels of color density may be further preprocessed to code the color density of each pixel using a number representing eight levels, for example, from 0 to 7. The color density coded maps 72r, 72g, and 72b for the R, G, and B single-color images shown in FIG. 1 represent the color density of each pixel using a code representing eight levels, from 0 to 7 (three levels in gradation). The code representing the color density is also referred to herein as a color density value.

[0056] The second training image 71 is an image of a fluorescently stained nuclear specimen captured in two or more grayscale levels or in color under fluorescent observation with a fluorescence microscope. The second training image 71 can be acquired in advance using, for example, a known fluorescence microscope or a bright-field image acquisition device such as a virtual slide scanner.

[0057] The second training data is a true image 73 showing the region of the cell nucleus of the tissue to be trained, generated from a second training image 71 obtained by capturing an image of the tissue to be trained. The first training image 70 and the second training image 71 are images captured of the same region or corresponding regions of the tissue on the specimen.

[0058] The second training data is generated by binarizing second training images 71, which are either in two or more grayscale levels or in color, into black and white fluorescent images, which are then used as true image 73 to be learned by the neural network 50 as the correct answer. When the object to be discriminated by the neural network 60 is the region of a cell nucleus, the true image 73 is data indicating the region of the cell nucleus, i.e., the correct answer. By binarizing the second training images 71, the region of the cell nucleus is distinguished from other regions, and the region of the cell nucleus is identified. The determination of whether the region is a cell nucleus or other regions is made, for example, by comparing the color density of each pixel in the image with a predetermined condition (for example, a color density threshold).

[0059] In the deep learning method, color density coded maps 72r, 72g, and 72b (also referred to as first training data) and a true value image 73 (also referred to as second training data) are used as training data 74, and are trained on a neural network 50 having the color density coded maps 72r, 72g, and 72b as the input layer 50a and the true value image 73 as the output layer 50b. That is, pairs of the color density coded maps 72r, 72g, and 72b for each of R, G, and B and the true value image 73 are used as training data 74 for learning by the neural network 50.

[0060] A method for generating training data 74 will be described with reference to Figures 2(a) to 2(c). The training data 74 is data that combines color density coded maps 72r, 72g, and 72b for each of R, G, and B, and a true value image 73. In Figure 2(a), the image size (size per training data item) of the training data 74 is simplified for ease of explanation, and the color density coded maps 72r, 72g, and 72b and the true value image 73 each have a total of 81 pixels, 9 pixels vertically and 9 pixels horizontally.

[0061] FIG. 2(b) shows an example of pixels constituting the training data 74. The three values ​​74a shown in the upper row of FIG. 2(b) are the density values ​​of the R, G, and B colors for each pixel. For example, the three values ​​are stored in the order of red (R), green (G), and blue (B). Each pixel in the color density encoding maps 72r, 72g, and 72b is represented by eight color density values ​​ranging from 0 to 7. This is an example of image preprocessing, in which the brightness of each color image 72R, 72G, and 72B, which is represented in 256 levels when captured, is converted into eight color density values. For example, the lowest brightness (a group of gradations with low brightness values ​​when expressed in the 256-level RGB color system) is assigned a color density value of 0, and gradually higher values ​​are assigned as the brightness increases, with the highest brightness (a group of gradations with high brightness values ​​when expressed in the 256-level RGB color system) being assigned a color density value of 7. The value 74b shown in the lower part of Figure 2(b) is the binary data of the true image 73. The binary data 74b of the true image 73 is also called a label value. For example, a label value of 1 indicates the region of the cell nucleus, and a label value of 0 indicates the other region. That is, in the true image 73 shown in Figure 1, the position of a pixel where the label value changes from 1 to 0 or from 0 to 1 corresponds to the boundary between the region of the cell nucleus and the other region.

[0062] The training data 75 shown in FIG. 2(c) is data cut out from a region of a predetermined number of pixels (hereinafter referred to as "window size") of the training data 74 shown in FIG. 2(a). The window size of the training data 75 is also shown simplified to 3×3 pixels for ease of explanation, but an actual preferable window size is, for example, about 113×113 pixels, and a size that can contain about 3×3 nuclei of normal gastric epithelial cells is preferable from the viewpoint of learning efficiency. For example, as shown in FIG. 2(c), a window W1 of 3×3 pixels is set, and the training data 74 is The center of the window W1 is located at one of the pixels in the training data 74, and for example, the training data 74 within the window W1 indicated by the black frame is extracted as training data 75 of the window size. The extracted training data 75 of the window size is used for training the neural network 50 shown in FIG. 1.

[0063] As shown in Figure 1, the number of nodes in the input layer 50a of the neural network 50 corresponds to the product of the number of pixels in the input window-sized training data 75 and the number of primary colors contained in the image (for example, R, G, and B for the three primary colors of light). Color density value data 76 of each pixel in the window-sized training data 75 is used as the input layer 50a of the neural network, and binary data 77 of the central pixel among binary data 74b corresponding to the true image 73 of each pixel in the training data 75 is used as the output layer 50b of the neural network, and the neural network 50 is trained. The color density value data 76 of each pixel is a collection of color density values ​​74a of R, G, and B for each pixel in the training data 75. As an example, if the window size of the training data 75 is 3x3 pixels, one color density value 74a is assigned for each of R, G, and B for each pixel, so the number of color density values ​​in the color density value data 76 is "27" (3x3x3=27), and the number of nodes in the input layer 50a of the neural network 50 is also "27."

[0064] In this way, the training data 75 of the window size to be input to the neural network 50 can be automatically created by the computer without the user having to create it, thereby promoting efficient deep learning of the neural network 50.

[0065] As shown in FIG. 2(c), in the initial state, the center of window W1 is located in the upper left corner of training data 74. Thereafter, training data 75 of the window size is extracted using window W1, and the position of window W1 is moved each time training of neural network 50 is performed. Specifically, window W1 is moved by one pixel at a time so that the center of window W1 scans, for example, all pixels of training data 74. As a result, training data 75 of the window size extracted from all pixels of training data 74 is used for training of neural network 50. This improves the learning level of neural network 50, and as a result of deep learning, a deep learning algorithm having the neural network 60 structure shown in FIG. 3 is obtained.

[0066] Overview of image analysis methods As shown in FIG. 3, the image analysis method generates analysis data 80 from an analysis target image (bright-field image) 78, which is an image of a specimen containing tissue or cells to be analyzed. The specimen is preferably stained in the same manner as the first training image. The analysis target image 78 can also be acquired, for example, as a color image using a known microscope or virtual slide scanner. The analysis target image (bright-field image) 78 may be an image containing one or more primary colors. By encoding the color density values ​​of R, G, and B for each pixel of the color analysis target image 78, the entire image can be represented as a coded map of the color density values ​​for each pixel of R, G, and B (analysis color density coded maps 79r, 79g, and 79b). The color density coded maps 79r, 79g, and 79b, which show the codes of color density in the single-color images of R, G, and B as exemplified in FIG. 3, display the color density values ​​coded in eight levels from 0 to 7 instead of the images 79R, 79G, and 79B of the three primary colors.

[0067] The analysis data 80 is data obtained by cutting out a region of a predetermined number of pixels (i.e., window size) from the color density coding maps 79r, 79g, and 79b, and is data containing color density values ​​of tissues or cells contained in the analysis target image 78. As with the training data 75, the analysis data 80 of the window size is shown simplified to 3×3 pixels for ease of explanation, but an actual preferable window size is illustratively about 113×113 pixels, and within that, From the standpoint of discrimination accuracy, a size large enough to accommodate approximately 3 × 3 normal gastric epithelial cell nuclei is preferable, for example, approximately 113 × 113 pixels at a 40x field of view. For example, a 3 × 3 pixel window W2 is set and moved relative to the color density coded maps 79r, 79g, and 79b. The center of the window W2 is located at one of the pixels in the color density coded maps 79r, 79g, and 79b. Cutting out the color density coded maps 79r, 79g, and 79b using the window W2, indicated by a black frame of 3 × 3 pixels, yields analysis data 80 of the window size. In this way, analysis data 80 is generated from the color density coded maps 79r, 79g, and 79b for each region centered on a specific pixel and including surrounding pixels. The predetermined pixel refers to the pixel of the color density coding map 79r, 79g, 79b located at the center of the window W2, and the surrounding pixels refer to the pixels of the color density coding map 79r, 79g, 79b included within the range of the window size centered on the predetermined pixel. In the analysis data 80, as in the training data 74, the color density values ​​for each pixel are stored in the order of red (R), green (G), and blue (B).

[0068] In the image analysis method, analysis data 80 is processed using a deep learning algorithm 60 having a neural network trained using training data 75 with the window size shown in Figure 1. By processing the analysis data 80, data 83 is generated that indicates whether or not the region is a cell nucleus in the tissue or cell being analyzed.

[0069] Referring back to FIG. 3 , analysis data 80 extracted from the color density coding maps 79r, 79g, and 79b for R, G, and B are input to the neural network 60, which constitutes a deep learning algorithm. The number of nodes in the input layer 60a of the neural network 60 corresponds to the product of the number of input pixels and the number of primary colors contained in the image. When color density value data 81 for each pixel of the analysis data 80 is input to the neural network 60, an estimated value 82 (binary value) of the pixel located at the center of the analysis data 80 is output from the output layer 60b. For example, an estimated value of 1 indicates a cell nucleus region, and an estimated value of 0 indicates other regions. In other words, the estimated value 82 output from the output layer 60b of the neural network 60 is data generated for each pixel of the image to be analyzed and indicates whether the pixel is a cell nucleus region in the image to be analyzed. The estimated value 82 distinguishes between cell nucleus regions and other regions, for example, by using values ​​1 and 0. The estimated value 82 is also called a label value, and will also be called a class in the description of neural networks to be described later. The neural network 60 generates a label for the pixel located at the center of the input analysis data 80, indicating whether it is a cell nucleus region or not, for the pixel located at the center of the analysis data 80. In other words, the neural network 60 classifies the analysis data 80 into classes indicating cell nucleus regions included in the image to be analyzed. Note that the color density value data 81 of each pixel is a collection of data on the color density values ​​of R, G, and B of each pixel of the analysis data 80.

[0070] Thereafter, the window W2 is moved pixel by pixel so that the center of the window W2 scans all pixels of the R, G, and B color density coding maps 79r, 79g, and 79b, respectively, and analysis data 80 is extracted to the window size. The extracted analysis data 80 is input to the neural network 60. This process yields binary data 83 indicating whether or not a region in the image to be analyzed is a cell nucleus. In the example shown in FIG. 3, a cell nucleus region detection process is further performed on the binary data 83 to obtain a cell nucleus region-enhanced image 84 indicating the cell nucleus region. Specifically, the cell nucleus region detection process is, for example, a process of detecting pixels whose estimated value 82 is 1, thereby actually identifying the cell nucleus region. The cell nucleus region-enhanced image 84 is an image in which the cell nucleus region obtained by the image analysis process is superimposed on the analysis target image 78. After identifying the cell nucleus region, a process may be performed in which the cell nucleus and other regions are distinguishably displayed on a display device. For example, the area of ​​the cell nucleus is filled with a color, a line is drawn between the area of ​​the cell nucleus and the other areas, or the like, and these are displayed on a display device in a distinguishable manner.

[0071] First Embodiment In the first embodiment, a specific description will be given of the configuration of a system that implements the deep learning method and image analysis method outlined above.

[0072] [Configuration Summary] Referring to FIG. 4, the image analysis system according to the first embodiment includes a deep learning device 100A and an image analysis device 200A. The vendor-side device 100 operates as the deep learning device 100A, and the user-side device 200 operates as the image analysis device 200A. The deep learning device 100A trains a neural network 50 using training data and provides the deep learning algorithm 60 trained by the training data to the user. The deep learning algorithm configured from the trained neural network 60 is provided from the deep learning device 100A to the image analysis device 200A via a recording medium 98 or a network 99. The image analysis device 200A analyzes the image to be analyzed using the deep learning algorithm configured from the trained neural network 60.

[0073] Deep learning device 100A is configured, for example, by a general-purpose computer and performs deep learning processing based on the flowchart described below. Image analysis device 200A is configured, for example, by a general-purpose computer and performs image analysis processing based on the flowchart described below. Recording medium 98 is a computer-readable, non-transitory, tangible recording medium such as a DVD-ROM or USB memory.

[0074] The deep learning device 100A is connected to an imaging device 300. The imaging device 300 includes an image sensor 301 and a fluorescence microscope 302, and captures bright-field images and fluorescent images of a training specimen 308 set on a stage 309. The training specimen 308 has been stained as described above. The deep learning device 100A acquires a first training image 70 and a second training image 71 captured by the imaging device 300.

[0075] The image analysis device 200A is connected to an imaging device 400. The imaging device 400 includes an imaging element 401 and a fluorescence microscope 402, and captures a bright-field image of a specimen 408 to be analyzed that is set on a stage 409. The specimen 408 to be analyzed has been stained in advance as described above. The image analysis device 200A acquires an image 78 to be analyzed that has been captured by the imaging device 400.

[0076] A known fluorescence microscope, virtual slide scanner, or the like having the function of capturing an image of a sample can be used as the imaging devices 300 and 400. The imaging device 400 may also be an optical microscope as long as it has the function of capturing an image of a sample.

[0077] [Hardware configuration] Referring to FIG. 5, the vendor-side device 100 (100A, 100B) includes a processing unit 10 (10A, 10B), an input unit 16, and an output unit 17.

[0078] The processing unit 10 includes a CPU (Central Processing Unit) 11 that performs data processing, which will be described later. The processor 10 comprises a memory 12 used as a working area for data processing, a recording unit 13 for recording programs and processing data (described later), a bus 14 for transmitting data between the various units, an interface unit 15 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 19. An input unit 16 and an output unit 17 are connected to the processor 10. Illustratively, the input unit 16 is an input device such as a keyboard or a mouse, and the output unit 17 is a display device such as a liquid crystal display. The GPU 19 functions as an accelerator that assists the arithmetic processing (for example, parallel arithmetic processing) performed by the CPU 11. That is, in the following description, the CPU 11 will be referred to as a The processing performed by the CPU 11 includes processing performed by the CPU 11 using the GPU 19 as an accelerator.

[0079] In order to perform the processing of each step described below in FIG. 8, the processing unit 10 pre-records the program according to the present invention and the pre-trained neural network 50 in, for example, an executable format in the recording unit 13. The executable format is, for example, a format generated by conversion from a programming language by a compiler. The processing unit 10 performs processing using the program and the pre-trained neural network 50 recorded in the recording unit 13.

[0080] In the following description, unless otherwise specified, the processing performed by processing unit 10 refers to the processing performed by CPU 11 based on the programs and neural network 50 stored in recording unit 13 or memory 12. CPU 11 uses memory 12 as a working area to temporarily store necessary data (such as intermediate data during processing), and records data to be stored long-term, such as calculation results, in recording unit 13 as appropriate.

[0081] Referring to FIG. 6, the user device 200 (200A, 200B, 200C) includes a processing unit 20 (20A, 20B, 20C), an input unit 26, and an output unit 27.

[0082] The processing unit 20 includes a CPU (Central Processing Unit) 21 that performs data processing, which will be described later. The CPU 20 includes a memory 22 used as a working area for data processing, a recording unit 23 for recording programs and processing data (described later), a bus 24 for transmitting data between the various units, an interface unit 25 for inputting and outputting data to and from external devices, and a GPU (Graphics Processing Unit) 29. An input unit 26 and an output unit 27 are connected to the processing unit 20. Illustratively, the input unit 26 is an input device such as a keyboard or a mouse, and the output unit 27 is a display device such as a liquid crystal display. The GPU 29 functions as an accelerator that assists the arithmetic processing (e.g., parallel arithmetic processing) performed by the CPU 21. In other words, in the following description, the processing performed by the CPU 21 also includes processing performed by the CPU 21 using the GPU 29 as an accelerator.

[0083] Furthermore, in order to perform the processing of each step described below in FIG. 11, the processing unit 20 pre-records the program according to the present invention and a deep learning algorithm 60 of a trained neural network structure in, for example, an executable format in the recording unit 23. The executable format is, for example, a format generated by conversion from a programming language by a compiler. The processing unit 20 performs processing using the program and deep learning algorithm 60 recorded in the recording unit 23.

[0084] In the following description, unless otherwise specified, the processing performed by the processing unit 20 actually means processing performed by the CPU 21 of the processing unit 20 based on the program and deep learning algorithm 60 stored in the recording unit 23 or memory 22. The CPU 21 uses the memory 22 as a working area to temporarily store necessary data (such as intermediate data during processing), and records data to be stored long-term, such as calculation results, in the recording unit 23 as appropriate.

[0085] [Function blocks and processing procedures] Deep learning processing 7, the processing unit 10A of the deep learning device 100A according to the first embodiment includes a training data generation unit 101, a training data input unit 102, and an algorithm update unit 103. These functional blocks are realized by installing a program that causes a computer to execute deep learning processing in the recording unit 13 or memory 12 of the processing unit 10A and executing this program by the CPU 11. The window size database 104 and the algorithm database 105 are recorded in the recording unit 13 or memory 12 of the processing unit 10A.

[0086] The first training image 70 and the second training image 71 of the learning specimen are assumed to be captured in advance by the imaging device 300 and stored in advance in the recording unit 13 of the processing unit 10A or the memory 12. The neural network 50 is stored in advance in the algorithm database 105 in association with, for example, the type of tissue sample (e.g., tissue name) from which the specimen to be analyzed is derived or the type of sample containing cells.

[0087] Processing unit 10A of deep learning device 100A performs the processing shown in Fig. 8. Explaining this using the functional blocks shown in Fig. 7, the processing of steps S11 to S13, S18, and S19 is performed by training data generation unit 101. The processing of step S14 is performed by training data input unit 102. The processing of steps S15 to S17 is performed by algorithm update unit 103.

[0088] Steps S11 to S19 described below describe deep learning processing for a pair of a first training image 70 (bright-field image) and a second training image (second training image 71).

[0089] In step S11, the processing unit 10A generates color density coded maps 72r, 72g, and 72b for each of the R, G, and B colors from the input first training image 70. The color density coded maps 72r, 72g, and 72b are created by expressing the color density values ​​of each of the R, G, and B colors of each pixel of the first training image 70 in stages. In this embodiment, the color density values ​​are assigned in eight stages from 0 to 7, and the color density coded maps 72r, 72g, and 72b are created for each of the R, G, and B gradation images. The color density values ​​are assigned, for example, by assigning a color density value of 0 to the lowest brightness and gradually increasing values ​​as the brightness level increases, with the highest brightness being assigned a color density value of 7.

[0090] In step S12, the processing unit 10A binarizes the gradation of each pixel of the input second training image 71 to generate a true image 73. The true image 73 (binarized image 73) is used to generate training data that the neural network 50 learns as a correct answer. The binarization process is performed, for example, by comparing the gradation of each pixel in the image with a predetermined threshold value.

[0091] In step S13, the processing unit 10A receives input of the type of tissue to be used for training from the operator of the deep learning device 100A via the input unit 16. Based on the input tissue type, the processing unit 10A sets the window size by referencing the window size database 104 and sets the neural network 50 to be used for training by referencing the algorithm database 105. In this embodiment, in which a gastric tissue specimen is analyzed, the window size is, for example, 113 × 113 pixels. This pixel size is the size of an image captured at, for example, 40x magnification. For example, the window size is a size that supports the entire shape of the cell nucleus region of at least one cell out of a plurality of cells, approximately two to nine, being included within the window. The window size is the unit of training data input to the neural network 50 at one input. The product of the number of pixels of the training data 75 in the window size and the number of primary colors included in the image corresponds to the number of nodes in the input layer 50a. The window size is associated with the type of tissue sample or the type of sample containing cells and is pre-recorded in the window size database 104.

[0092] In step S14, the processing unit 10A generates training data 75 of a window size from the color density coded maps 72r, 72g, and 72b and the true value image 73. Specifically, as described in the above "Outline of Deep Learning Method" with reference to Figures 2(a) to 2(c), the processing unit 10A generates training data 75 of a window size by using the window W1 from training data 74 that combines the color density coded maps 72r, 72g, and 72b and the true value image 73.

[0093] 8, processing unit 10A trains neural network 50 using window-sized training data 75. The training results of neural network 50 are accumulated every time neural network 50 is trained using window-sized training data 75.

[0094] In the image analysis method according to the embodiment, a convolutional neural network is used, and a stochastic gradient descent method is employed. Therefore, in step S16, the processing unit 10A determines whether or not learning results for a predetermined number of trials have been accumulated. If learning results for the predetermined number of trials have been accumulated, the processing unit 10A performs the process of step S17. If learning results for the predetermined number of trials have not been accumulated, the processing unit 10A performs the process of step S18.

[0095] When the learning results have been accumulated for the predetermined number of trials, in step S17, processing unit 10A updates the connection weights w of neural network 50 using the learning results accumulated in step S15. Since the image analysis method according to the embodiment uses the stochastic gradient descent method, the connection weights w of neural network 50 are updated when the learning results for the predetermined number of trials have been accumulated. Specifically, the process of updating the connection weights w is a process of performing calculations using the gradient descent method shown in (Equation 11) and (Equation 12) described below.

[0096] In step S18, the processing unit 10A determines whether a predetermined number of pixels in the input image have been processed. The input image is training data 74, and if the series of processes from step S14 to step S17 have been performed on the predetermined number of pixels in the training data 74, the deep learning process is terminated. Neural network learning does not necessarily have to be performed on all pixels in the input image; the processing unit 10A can process and learn a portion of the pixels in the input image, i.e., a predetermined number of pixels. The predetermined number of pixels may be all pixels in the input image.

[0097] If the predetermined number of pixels in the input image have not been processed, the processing unit 10A moves the center position of the window in one pixel unit within the training data 74 in step S19, as shown in FIG. 2(c). Thereafter, the processing unit 10A performs a series of processes from step S14 to step S17 at the new window position after the movement. That is, in step S14, the processing unit 10A cuts out the training data 74 at the new window position after the movement by the window size. Subsequently, in step S15, the processing unit 10A trains the neural network 50 using the training data 75 of the newly cut window size. In step S16, if learning results for a predetermined number of trials have been accumulated, the processing unit 10A updates the connection weight w of the neural network 50 in step S17. This training of the neural network 50 for each window size is performed on the predetermined number of pixels in the training data 74.

[0098] The deep learning process of steps S11 to S19 for one pair of input images described above is repeated for multiple pairs of different input images to improve the level of learning of the neural network 50. This results in a deep learning algorithm 60 with a neural network structure shown in FIG.

[0099] Neural network structure As shown in Fig. 9(a), the first embodiment uses a deep learning type neural network. The deep learning type neural network includes an input layer 50a, an output layer 50b, and an intermediate layer 50c between the input layer 50a and the output layer 50b, and the intermediate layer 50c is configured with multiple layers, as in the neural network 50 shown in Fig. 9. The number of layers that make up the intermediate layer 50c can be, for example, five or more.

[0100] In the neural network 50, multiple nodes 89 are arranged in layers and connected between the layers. This allows information to propagate in only one direction, as indicated by arrow D in the figure, from the input-side layer 50a to the output-side layer 50b. In this embodiment, the number of nodes in the input layer 50a corresponds to the product of the number of pixels in the input image, i.e., the number of pixels in the window W1 shown in FIG. 2(c), and the number of primary colors contained in each pixel. Since pixel data (color density values) of an image can be input to the input layer 50a, the user can input the input image to the input layer 50a without having to separately calculate feature values ​​from the input image.

[0101] · Calculations at each node Figure 9(b) is a schematic diagram showing the operations at each node. Each node 89 receives multiple inputs and calculates one output (z). In the example shown in Figure 9(b), node 89 receives four inputs. The total input (u) received by node 89 is expressed by the following (Equation 1).

number

[0102] Each input is multiplied by a different weight. In (Equation 1), b is a value called the bias. The output (z) of the node is the output of a given function f for the total input (u) expressed in (Equation 1), and is expressed in the following (Equation 2). The function f is called the activation function.

number

[0103] FIG. 9(c) is a schematic diagram showing the operations between nodes. In the neural network 50, nodes that output a result (z) expressed by (Equation 2) for a total input (u) expressed by (Equation 1) are arranged in layers. The output of a node in the previous layer becomes the input of a node in the next layer. In the example shown in FIG. 9(c), the output of node 89a in the layer on the left side of the figure becomes the input of node 89b in the layer on the right side of the figure. Each node 89b in the right layer receives the output from node 89a in the layer on the left side. A different weight is applied to each connection between each node 89a in the left layer and each node 89b in the right layer. If the outputs of each of the multiple nodes 89a in the left layer are x1 to x4, the inputs to each of the three nodes 89b in the right layer are expressed by the following (Equation 3-1) to (Equation 3-3).

number

[0104] Generalizing these (Equation 3-1) to (Equation 3-3), we obtain (Equation 3-4), where i=1,···I, j=1,···J.

number

[0105] Applying (Equation 3-4) to the activation function gives the output, which is expressed as (Equation 4) below.

number

[0106] Activation function In the image analysis method according to the embodiment, a rectified linear unit function is used as the activation function. The rectified linear unit function is expressed by the following (Equation 5).

number

[0107] (Equation 5) is a linear function of z=u, where the part of u<0 is set to u=0. In the example shown in FIG. 9(c), the output of the node j=1 is expressed by the following equation using (Equation 5):

number

[0108] Neural network training If the function expressed using a neural network is y(x:w), the function y(x:w) will change when the parameter w of the neural network is changed. Adjusting the function y(x:w) so that the neural network selects the parameter w that is more suitable for the input x is called neural network learning. Suppose multiple pairs of input and output of the function expressed using a neural network are given. If the desired output for a certain input x is d, the input / output pair is {(x1,d1),(x2,d2),...,(x n ,d n )}. The set of pairs represented by (x, d) is called training data. Specifically, the set of pairs of color density values ​​for each pixel in the single-color images of R, G, and B and labels of the true-value image shown in Figure 2(b) is the training data shown in Figure 2(a).

[0109] Learning a neural network involves determining what input and output pairs (x n ,d n ), the input x n The output of the neural network when given y(x n :w) but the output is d n This means adjusting the weights w so that they are as close as possible to the error function. The error function is the degree of similarity between the function expressed using a neural network and the training data.

number

number

[0110] A method for calculating the cross entropy of (Equation 6) will be explained. In the output layer 50b of the neural network 50 used in the image analysis method according to the embodiment, i.e., in the final layer of the neural network, an activation function is used to classify the input x into a finite number of classes according to its content. The activation function is called a softmax function, and is expressed by the following (Equation 7). It is assumed that the output layer 50b has the same number of nodes as the number of classes k. The total input u of each node k (k=1,...K) of the output layer L is calculated by subtracting u from the output of the previous layer L-1. k (L) As a result, the output of the kth node in the output layer is This is expressed by (Equation 7).

number

[0111] (Equation 7) is the softmax function. The output y1, ,y K The sum is always 1.

[0112] Let each class be C1,...,C K Then, the output y of node k in the output layer L is K (i.e., u k (L) ) is the probability that a given input x is of class C K represents the probability that the input x belongs to the class with the highest probability expressed by (Equation 8).

number

[0113] In neural network training, the function represented by the neural network is considered as a model of the posterior probability of each class, and the function of such a probability model is Below, we evaluate the likelihood of the weights w given the training data and select the weights w that maximize the likelihood.

[0114] The target output d by the softmax function of (Eq. 7) n Let d be 1 only if the output is the correct class, and 0 otherwise. Let d be the target output. n =[d n1 ,···,d nK ], for example, input x n If the correct class of is C3, the target output d n3 Only the target output is 1, and the other target outputs are 0. When encoded in this way, the posterior distribution is expressed by the following (Equation 9).

number

[0115] training data {(x n ,d n )}(n=1, ,N), the likelihood L(w) of weight w is expressed by the following (Equation 10): Taking the logarithm of the likelihood L(w) and inverting the sign leads to the error function of (Equation 6).

number

[0116] Learning means minimizing the error function E(w) calculated based on training data with respect to the parameter w of the neural network. In the image analysis method according to the embodiment, the error function E(w) is expressed by Equation 6.

[0117] Minimizing the error function E(w) with respect to the parameter w is the same as finding a local minimum of the function E(w). The parameter w is the weight of the connection between nodes. The minimum of the weight w is found by iterative calculations that start with an arbitrary initial value and repeatedly update the parameter w. An example of such calculations is the gradient descent method. hod).

[0118] The gradient descent method uses a vector expressed by the following (Equation 11).

number

[0119] Gradient descent involves repeatedly moving the current value of the parameter w in the negative gradient direction (i.e., -∇E). (t) The weight after the movement is w (t+1) Then, the calculation by the gradient descent method is expressed by the following (Equation 12): The value t indicates the number of times the parameter w is moved.

number

[0120] symbol

number

[0121] The calculation by (Equation 12) may be performed on all training data (n = 1, , N) or only on a part of the training data. The gradient descent method performed on only a part of the training data is called stochastic gradient descent. The image analysis method according to the embodiment uses a stochastic gradient descent method.

[0122] Image analysis processing 10, processing unit 20A of image analysis device 200A according to the first embodiment includes analysis data generation unit 201, analysis data input unit 202, analysis unit 203, and cell nucleus region detection unit 204. These functional blocks are realized by installing a program for causing a computer according to the present invention to execute image analysis processing in recording unit 23 or memory 22 of processing unit 20A and executing this program by CPU 21. Window size database 104 and algorithm database 105 are provided from deep learning device 100A via recording medium 98 or network 99 and are recorded in recording unit 23 or memory 22 of processing unit 20A.

[0123] The analysis target image 78 of the tissue to be analyzed is assumed to be captured in advance by the imaging device 400 and pre-recorded in the recording unit 23 or memory 22 of the processing unit 20A. The deep learning algorithm 60 including the trained connection weights w is stored in the algorithm database 105 in association with the type of tissue sample (e.g., tissue name) from which the tissue specimen to be analyzed is derived or the type of sample containing cells, and functions as a program module that is part of a program that causes a computer to execute image analysis processing. That is, the deep learning algorithm 60 is used in a computer equipped with a CPU and memory, and causes the computer to perform calculations or processing of specific information according to the intended use, such as outputting data indicating whether or not the region in the tissue to be analyzed is a cell nucleus region. Specifically, the CPU 21 of the processing unit 20A performs calculations of the neural network 60 based on the trained connection weights w according to the algorithm defined in the deep learning algorithm 60 recorded in the recording unit 23 or memory 22. The CPU 21 of the processing unit 20A calculates the solution input to the input layer 60a. An operation is performed on an analysis target image 78 obtained by capturing an image of the tissue to be analyzed, and a binary image 83 is output from the output layer 60b as data indicating whether or not the tissue to be analyzed is a cell nucleus region.

[0124] 11, processing unit 20A of image analyzing device 200A performs the processing shown in Fig. 11. Explaining this using the functional blocks shown in Fig. 10, the processing of steps S21 and S22 is performed by analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by analysis data input unit 202. The processing of steps S25 and S28 is performed by analysis unit 203. The processing of step S29 is performed by cell nucleus region detection unit 204.

[0125] In step S21, the processing unit 20A generates color density coded maps 79r, 79g, and 79b for each of the colors R, G, and B from the input analysis target image 78. The method for generating the color density coded maps 79r, 79g, and 79b is the same as the generation method in step S11 during the deep learning process shown in FIG.

[0126] In step S22 shown in FIG. 11 , the processing unit 20A receives an input of a tissue type from a user of the image analysis device 200A via the input unit 26 as an analysis condition. Based on the input tissue type, the processing unit 20A references the window size database 104 and the algorithm database 105 to set a window size to be used for analysis and acquire a deep learning algorithm 60 to be used for analysis. The window size is a unit of analysis data input to the neural network 60 at one time. The product of the number of pixels of the analysis data 80 in the window size and the number of primary colors included in the image corresponds to the number of nodes in the input layer 60a. The window size is associated with the tissue type and pre-recorded in the window size database 104. The window size is, for example, 3 × 3 pixels, as in the window W2 shown in FIG. 3 . The deep learning algorithm 60 is also associated with the tissue sample type or the cell-containing sample type and pre-recorded in the algorithm database 105 shown in FIG. 10 .

[0127] In step S23 shown in FIG. 11, the processing unit 20A generates analysis data 80 for the window size from the color density coded diagrams 79r, 79g, and 79b.

[0128] In step S24, the processing unit 20A inputs the analysis data 80 shown in FIG. 3 to the deep learning algorithm 60. As in step S15 during deep learning processing, the initial position of the window is, for example, a position where the pixel located at the center of the 3×3 pixels in the window corresponds to the upper left corner of the image to be analyzed. When the processing unit 20A inputs data 81 of a total of 27 color density values ​​(3×3 pixels×3 primary colors) included in the analysis data 80 of the window size to the input layer 60a, the deep learning algorithm 60 outputs a discrimination result 82 to the output layer 60b.

[0129] In step S25 shown in Fig. 11, the processing unit 20A records the discrimination result 82 output to the output layer 60b shown in Fig. 3. The discrimination result 82 is an estimated value (binary) of the pixel located at the center of the color density coding diagrams 79r, 79g, and 79b that are the analysis target. For example, an estimated value of 1 indicates a cell nucleus region, and an estimated value of 0 indicates other regions.

[0130] In step S26 shown in Fig. 11, the processing unit 20A determines whether all pixels in the input image have been processed. The input image is the color density coded maps 79r, 79g, and 79b shown in Fig. 3, and if the series of processes from step S23 to step S25 shown in Fig. 11 have been performed for all pixels in the color density coded maps 79r, 79g, and 79b, the processing unit 20A performs the process of step S28.

[0131] If all pixels in the input image have not been processed, in step S27, the processing unit 20A moves the center position of the window W2 by one pixel within the color density encoding maps 79r, 79g, and 79b shown in FIG. 3, similar to step S19 during deep learning processing. Then, the processing unit 20A performs a series of processes from step S23 to step S25 at the new position of the window W2 after the movement. In step S25, the processing unit 20A records the discrimination result 82 corresponding to the new window position after the movement. By recording the discrimination result 82 for each window size in this manner for all pixels in the image to be analyzed, a binary image 83 of the analysis result is obtained. The image size of the binary image 83 of the analysis result is the same as the image size of the image to be analyzed. Here, the binary image 83 may be numerical data in which estimated values ​​1 and 0 are assigned to each pixel, or may be an image displayed in display colors corresponding to the values ​​1 and 0, respectively, instead of the estimated values ​​1 and 0.

[0132] In step S28 shown in FIG. 11, the processing unit 20A outputs a binary image 83 of the analysis result to the output unit 27.

[0133] In step S29, following step S28, the processing unit 20A further performs cell nucleus region detection processing on the cell nucleus region for the binary image 83 of the analysis result. In the binary image 83, the cell nucleus region and other regions are represented by different values. Therefore, the cell nucleus region can be identified by detecting the position of a pixel in the binary image 83 where the estimated pixel value changes from 1 to 0 or from 0 to 1. In another aspect, the boundary between the cell nucleus region and other regions, i.e., the cell nucleus region, can be detected.

[0134] Optionally, the processing unit 20A creates a cell nucleus region-enhanced image 84 by superimposing the obtained cell nucleus region on the analysis target image 78. The processing unit 20A outputs the created cell nucleus region-enhanced image 84 to the output unit 27, and ends the image analysis process.

[0135] As described above, a user of image analyzing device 200A can obtain binary image 83 as an analysis result by inputting analysis target image 78 of tissue to be analyzed into image analyzing device 200A. Binary image 83 represents the area of ​​cell nuclei and other areas in the specimen to be analyzed, and the user can distinguish the area of ​​cell nuclei in the specimen to be analyzed.

[0136] Furthermore, the user of the image analyzing device 200A can obtain a cell nucleus region-enhanced image 84 as the analysis result. The cell nucleus region-enhanced image 84 is generated, for example, by filling in the cell nucleus region with color in the analysis target image 78 of the analysis target. In another aspect, it is generated by overlapping the boundary line between the cell nucleus region and other regions. This allows the user to grasp at a glance the cell nucleus region in the tissue of the analysis target.

[0137] Showing the area of ​​the cell nucleus in the specimen to be analyzed helps those who are not familiar with looking at specimens to understand the state of the cell nucleus.

[0138] <Second embodiment> The image analysis system according to the second embodiment will be described below, focusing on the differences from the image analysis system according to the first embodiment.

[0139] [Configuration Summary] Referring to FIG. 12, the image analysis system according to the second embodiment includes a user-side device 200, which operates as an integrated image analysis device 200B. The image analysis device 200B is configured, for example, by a general-purpose computer, and performs both the deep learning processing and the image analysis processing described in the first embodiment. In other words, the image analysis system according to the second embodiment is a standalone system in which deep learning and image analysis are performed on the user side. The image analysis system according to the second embodiment differs from the image analysis system according to the first embodiment in that the integrated image analysis device 200B installed on the user side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment.

[0140] Image analysis device 200B is connected to imaging device 400. During deep learning processing, imaging device 400 acquires first training image 70 and second training image 71 of tissue for learning, and during image analysis processing, acquires analysis target image 78 of tissue to be analyzed.

[0141] [Hardware configuration] The hardware configuration of the image analyzing device 200B is the same as the hardware configuration of the user side device 200 shown in FIG.

[0142] [Function blocks and processing procedures] Referring to FIG. 13 , the processing unit 20B of the image analysis device 200B according to the second embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a cell nucleus detection unit 204. These functional blocks are implemented by installing a program for causing a computer to execute deep learning processing and image analysis processing in the recording unit 23 or memory 22 of the processing unit 20B and executing the program by the CPU 21. The window size database 104 and the algorithm database 105 are recorded in the recording unit 23 or memory 22 of the processing unit 20B and are both used during deep learning and image analysis processing. A trained neural network 60 is associated with a tissue type or a cell-containing sample type and pre-stored in the algorithm database 105. The connection weight w is updated by the deep learning processing and stored in the algorithm database 105 as a deep learning algorithm 60. It is assumed that the first training image 70 and the second training image 71, which are first training images for learning, are captured in advance by the imaging device 400 and stored in advance in the recording unit 23 or memory 22 of the processing unit 20B. The analysis target image 78 of the specimen to be analyzed is also captured in advance by the imaging device 400 and stored in advance in the recording unit 23 or memory 22 of the processing unit 20B.

[0143] Processing unit 20B of image analysis device 200B performs the processing shown in FIG. 8 during deep learning processing, and performs the processing shown in FIG. 11 during image analysis processing. Explaining this using the functional blocks shown in FIG. 13, during deep learning processing, the processing of steps S11 to S13, S18, and S19 is performed by training data generation unit 101. The processing of step S14 is performed by training data input unit 102. The processing of steps S15 to S17 is performed by algorithm update unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by analysis data input unit 202. The processing of steps S25 and S28 is performed by analysis unit 203. The processing of step S29 is performed by cell nucleus region detection unit 204.

[0144] The deep learning process and image analysis process performed by image analysis device 200B according to the second embodiment are similar to the processes performed by deep learning device 100A and image analysis device 200A according to the first embodiment. Note that image analysis device 200B according to the second embodiment differs from deep learning device 100A and image analysis device 200A according to the first embodiment in the following respects.

[0145] In step S13 during deep learning processing, processing unit 20B receives an input of the type of tissue to be used for learning from the user of image analyzing device 200B via input unit 26. Based on the input type of tissue, processing unit 20B sets a window size with reference to window size database 104, and sets neural network 50 to be used for learning with reference to algorithm database 105.

[0146] As described above, the user of image analyzing device 200B can obtain binary image 83 as the analysis result by inputting analysis target image 78 into image analyzing device 200B. Furthermore, the user of image analyzing device 200B can obtain cell nucleus region-enhanced image 84 as the analysis result.

[0147] Image analyzing device 200B according to the second embodiment allows a user to use a type of tissue selected by the user as a training tissue. This means that the training of neural network 50 is not left to the vendor, but the user can improve the level of training of neural network 50.

[0148] <Third embodiment> The image analysis system according to the third embodiment will be described below, focusing on the differences from the image analysis system according to the second embodiment.

[0149] [Configuration Summary] Referring to FIG. 14, the image analysis system according to the third embodiment includes a vendor-side device 100 and a user-side device 200. The vendor-side device 100 operates as an integrated image analysis device 100B, and the user-side device 200 operates as a terminal device 200C. The image analysis device 100B is configured, for example, by a general-purpose computer and is a cloud server-side device that performs both the deep learning processing and the image analysis processing described in the first embodiment. The terminal device 200C is configured, for example, by a general-purpose computer and is a user-side terminal device that transmits images to be analyzed to the image analysis device 100B via a network 99 and receives images of the analysis results from the image analysis device 100B via the network 99.

[0150] The image analysis system according to the third embodiment is similar to the image analysis system according to the second embodiment in that an integrated image analysis device 100B installed on the vendor side performs the functions of both the deep learning device 100A and the image analysis device 200A according to the first embodiment. On the other hand, the image analysis system according to the third embodiment differs from the image analysis system according to the second embodiment in that it includes a terminal device 200C, which provides an input interface for images to be analyzed and an output interface for images of analysis results to the user-side terminal device 200C. In other words, the image analysis system according to the third embodiment is a cloud service-type system in which the vendor side, which performs deep learning processing and image analysis processing, provides the user side with input / output interfaces for images to be analyzed and images of analysis results.

[0151] Image analysis device 100B is connected to imaging device 300, and acquires first training image 70 and second training image 71 of a tissue for learning, which are captured by imaging device 300.

[0152] The terminal device 200C is connected to the imaging device 400, and acquires an analysis target image 78 of the tissue to be analyzed, which is imaged by the imaging device 400.

[0153] [Hardware configuration] The hardware configuration of the image analyzing device 100B is the same as the hardware configuration of the vendor-side device 100 shown in FIG. 5. The hardware configuration of the terminal device 200C is the same as the hardware configuration of the user-side device 100 shown in FIG. The hardware configuration is the same as that of the device 200.

[0154] [Function blocks and processing procedures] Referring to FIG. 15 , a processing unit 10B of an image analysis device 100B according to the third embodiment includes a training data generation unit 101, a training data input unit 102, an algorithm update unit 103, an analysis data generation unit 201, an analysis data input unit 202, an analysis unit 203, and a cell nucleus region detection unit 204. These functional blocks are implemented by installing a program for causing a computer to execute deep learning processing and image analysis processing in the recording unit 13 or memory 12 of the processing unit 10B and executing the program by the CPU 11. A window size database 104 and an algorithm database 105 are recorded in the recording unit 13 or memory 12 of the processing unit 10B and are both used during deep learning and image analysis processing. A neural network 50 is associated with a tissue type and pre-stored in the algorithm database 105. The connection weight w is updated by the deep learning processing and stored in the algorithm database 105 as a deep learning algorithm 60.

[0155] It is assumed that the first training image 70 and the second training image 71 for learning are captured in advance by the imaging device 300 and stored in advance in the recording unit 13 or memory 12 of the processing unit 10B. The analysis target image 78 of the tissue to be analyzed is also captured in advance by the imaging device 400 and stored in advance in the recording unit 23 or memory 22 of the processing unit 20C of the terminal device 200C.

[0156] Processing unit 10B of image analysis device 100B performs the processing shown in FIG. 8 during deep learning processing, and performs the processing shown in FIG. 11 during image analysis processing. Explaining this using the functional blocks shown in FIG. 15, during deep learning processing, the processing of steps S11 to S13, S18, and S19 is performed by training data generation unit 101. The processing of step S14 is performed by training data input unit 102. The processing of steps S15 to S17 is performed by algorithm update unit 103. During image analysis processing, the processing of steps S21 and S22 is performed by analysis data generation unit 201. The processing of steps S23, S24, S26, and S27 is performed by analysis data input unit 202. The processing of steps S25 and S28 is performed by analysis unit 203. The processing of step S29 is performed by cell nucleus region detection unit 204.

[0157] The deep learning process and image analysis process performed by image analysis device 100B according to the third embodiment are similar to the processes performed by deep learning device 100A and image analysis device 200A according to the first embodiment. Image analysis device 100B according to the third embodiment differs from deep learning device 100A and image analysis device 200A according to the first embodiment in the following four points.

[0158] 11, the processing unit 10B receives an analysis target image 78 of the tissue to be analyzed from the user terminal device 200C, and generates color density coded maps 79r, 79g, and 79b for each of the colors R, G, and B from the received analysis target image 78. The method for generating the color density coded maps 79r, 79g, and 79b is the same as the method for generating them in step S11 during the deep learning process shown in FIG.

[0159] 11, the processing unit 10B receives an input of a tissue type from the user of the terminal device 200C as an analysis condition via the input unit 26 of the terminal device 200C. Based on the input tissue type, the processing unit 10B references the window size database 104 and the algorithm database 105 to set a window size to be used in the analysis and acquires the deep learning algorithm 60 to be used in the analysis.

[0160] In step S28 of the image analysis process, the processing unit 10B transmits the binary image 83 of the analysis result to the user-side terminal device 200C. In the user-side terminal device 200C, the processing unit 20C outputs the received binary image 83 of the analysis result to the output unit 27.

[0161] In step S29 of the image analysis process, the processing unit 10B, following step S28, further performs a process of detecting the cell nucleus region on the binary image 83 that is the analysis result. The processing unit 10B creates a cell nucleus region-enhanced image 84 by superimposing the obtained cell nucleus region on the analysis target image 78 that is the analysis target. The processing unit 10B transmits the created cell nucleus region-enhanced image 84 to the user-side terminal device 200C. In the user-side terminal device 200C, the processing unit 20C outputs the received cell nucleus region-enhanced image 84 to the output unit 27, and the image analysis process ends.

[0162] As described above, the user of terminal device 200C can obtain, as the analysis result, binary image 83 by transmitting analysis target image 78 of the tissue to be analyzed to image analyzing device 100B. Furthermore, the user of terminal device 200C can obtain, as the analysis result, cell nucleus region-enhanced image 84.

[0163] According to the image analyzing device 100B of the third embodiment, a user can enjoy the results of the image analysis processing without acquiring the window size database 104 and the algorithm database 105 from the deep learning device 100A. This makes it possible to provide a service for identifying the region of a cell nucleus as a cloud service for analyzing tissue to be analyzed.

[0164] There is a nationwide shortage of pathologists who perform cytological diagnosis. While pathologists are employed at large hospitals in urban areas, they are rarely employed at medical institutions in remote areas or relatively small medical institutions such as clinics even in urban areas. The cloud services provided by image analysis device 100B and terminal device 200C assist with tissue and cytological diagnosis at such remote or relatively small medical institutions.

[0165] <Other forms> Although the present invention has been described above with reference to the outline and specific embodiments, the present invention is not limited to the outline and each embodiment described above.

[0166] In the above first to third embodiments, the case of stomach cancer is described as an example, but the specimen to be processed is not limited to this, and specimens of the aforementioned tissue samples or specimens containing cells can be used.

[0167] In the first to third embodiments, in step S13, the processing units 10A, 20B, and 10B set the number of pixels of the window size by referring to the window size database 104. However, the operator or user may directly set the window size. In this case, the window size database 104 is not necessary.

[0168] In the first to third embodiments, in step S13, the processing units 10A, 20B, and 10B set the number of pixels of the window size based on the input tissue type, but the tissue size may be input instead of the input tissue type. The processing units 10A, 20B, and 10B may set the number of pixels of the window size by referring to the window size database 104 based on the input tissue size. In step S22, as in step S13, the tissue size may be input instead of the input tissue type. The processing units 20A, 20B, and 10B may set the number of pixels of the window size by referring to the window size database 104 and the algorithm database 105 based on the input tissue size. All you have to do is set the number of pixels and acquire the neural network 60.

[0169] Regarding the manner in which the tissue size is input, the size may be input directly as a numerical value, or, for example, the input user interface may be a pull-down menu in which the user is prompted to select a predetermined numerical range corresponding to the size that the user wishes to input.

[0170] Furthermore, in steps S13 and S22, in addition to the type of tissue or the size of the tissue, the imaging magnification at which the first training image 70, the analysis target image 78, and the second training image 71 of the tissue were captured may be input. The imaging magnification may be input directly as a numerical value, or, for example, the input user interface may be a pull-down menu, allowing the user to select a predetermined numerical range corresponding to the magnification that the user wishes to input.

[0171] In the first to third embodiments, the window size is set to 3 × 3 pixels for the convenience of explanation during deep learning processing and image analysis processing, but the number of pixels in the window size is not limited to this. The window size may be set according to, for example, the type of tissue sample or the type of sample containing cells. In this case, the product of the number of pixels in the window size and the number of primary colors included in the image only needs to correspond to the number of nodes in the input layer 50 a, 60 a of the neural network 50, 60.

[0172] In step S13, the processing units 10A, 20B, and 10C may acquire the number of pixels of the window size, and may further correct the acquired number of pixels of the window size based on the input imaging magnification.

[0173] In the first to third embodiments, in step S17, the processing units 10A, 20B, and 10B associate the deep learning algorithms 60 with the tissue types in a one-to-one correspondence and record them in the algorithm database 105. Alternatively, in step S17, the processing units 10A, 20B, and 10B may associate one deep learning algorithm 60 with multiple tissue types and record them in the algorithm database 105.

[0174] In the first to third embodiments, the hue is defined as a combination of the three primary colors of light or the three primary colors of color. However, the number of hues is not limited to three. The number of hues may be four, i.e., red (R), green (G), blue (B), and yellow (Y), or two, i.e., two hues, i.e., one of the three primary colors (e.g., green (G)) removed from red (R), green (G), and blue (B). Alternatively, the bright-field image 70 and the analysis target image 78 acquired using a known microscope or virtual slide scanner are not limited to three primary color images (e.g., red (R), green (G), and blue (B)). They may be two-primary color images, or may contain one or more primary colors.

[0175] In the first to third embodiments, in step S11, the processing units 10A, 20B, and 10B generate the color density coded maps 72r, 72g, and 72b as single-color images with three gradations for each primary color. However, the gradations of the primary colors of the color density coded maps 72r, 72g, and 72b are not limited to three gradations. The gradations of the color density coded maps 72r, 72g, and 72b may be two-gradation images, or may be images with one or more gradations. Similarly, in step S21, the processing units 20A, 20B, and 10B generate the color density coded maps 79r, 79g, and 79b as single-color images for each primary color. However, the gradations of the primary colors used to create the color density coded maps are not limited to three gradations. The gradations of the primary colors used to create the color density coded maps may be two-gradation images, or may be images with one or more gradations. For example, the gradation of the color density coding schemes 72r, 72g, 72b, 79r, 79g, and 79b can be set to 256 levels (8 gradations) of color density values ​​ranging from 0 to 255.

[0176] In the first to third embodiments, in step S11, the processing units 10A, 20B, and 10B generate color density coded maps 72r, 72g, and 72b for R, G, and B, respectively, from the input first training image 70. However, the input first training image 70 may be pre-gradated. That is, the processing units 10A, 20B, and 10B may directly acquire the color density coded maps 72r, 72g, and 72b for R, G, and B, respectively, from, for example, a virtual slide scanner. Similarly, in step S21, the processing units 20A, 20B, and 10B generate color density coded maps 79r, 79g, and 79b for R, G, and B, respectively, from the input analysis target image 78. However, the input analysis target image 78 may be pre-gradated. That is, the processing units 20A, 20B, and 10B may directly acquire the color density coded maps 79r, 79g, and 79b of the R, G, and B colors from, for example, a virtual slide scanner.

[0177] In the first to third embodiments, RGB is used as the color space when generating the color density coding maps 72 and 79 from the color first training images 70 and 78, but the color space is not limited to RGB. In addition to RGB, YUV, CMY, and CIE L * a * b * Various color spaces such as the above can be used.

[0178] In the first to third embodiments, the density values ​​for each pixel are stored in the order of red (R), green (G), and blue (B) in the training data 74 and the analysis data 80. However, the order in which the density values ​​are stored and handled is not limited to this. For example, the density values ​​may be stored in the order of blue (B), green (G), and red (R), as long as the order of the density values ​​in the training data 74 and the analysis data 80 is the same.

[0179] In the above first to third embodiments, in step S12, the processing units 10A, 20B, and 10B binarize the gradation of each pixel of the input second training image 71 to generate the true value image 73, but it is also possible to acquire a true value image 73 that has been binarized in advance.

[0180] In the first to third embodiments, the processing units 10A and 10B are realized as an integrated device, but the processing units 10A and 10B do not need to be an integrated device, and the CPU 11, memory 12, recording unit 13, GPU 19, etc. may be located in different locations and connected via a network. The processing units 10A and 10B, the input unit 16, and the output unit 17 do not necessarily need to be located in the same location, and may be located in different locations and connected to each other so that they can communicate with each other via a network. The processing units 20A, 20B, and 20C are similar to the processing units 10A and 10B.

[0181] In the first to third embodiments, the functional blocks of the training data generation unit 101, the training data input unit 102, the algorithm update unit 103, the analysis data generation unit 201, the analysis data input unit 202, the analysis unit 203, and the cell nucleus region detection unit 204 are executed by a single CPU 11 or a single CPU 21. However, these functional blocks do not necessarily have to be executed by a single CPU, and may be executed in a distributed manner by multiple CPUs. Furthermore, these functional blocks may be executed in a distributed manner by multiple GPUs, or in a distributed manner by multiple CPUs and multiple GPUs.

[0182] In the second and third embodiments, a program for performing the processing of each step described in Figures 8 and 11 is pre-recorded in the recording units 13, 23. Alternatively, the program may be installed in the processing units 10B, 20B from a computer-readable, non-transitory, tangible recording medium 98, such as a DVD-ROM or a USB memory. Alternatively, the processing units 10B, 20B may be connected to a network 99, and the program may be downloaded from, for example, an external server (not shown) via the network 99 and installed.

[0183] In the first to third embodiments, the input units 16, 26 are input devices such as a keyboard or a mouse, and the output units 17, 27 are realized as display devices such as a liquid crystal display. Alternatively, the input units 16, 26 and the output units 17, 27 may be integrated into a touch panel display device. Alternatively, the output units 17, 27 may be configured with a printer or the like, and a binary image 83 of the analysis results or a cell nucleus region-enhanced image 84 of the cell nucleus may be printed and output.

[0184] In the first to third embodiments described above, imaging device 300 is directly connected to deep learning device 100A or image analysis device 100B, but imaging device 300 may be connected to deep learning device 100A or image analysis device 100B via network 99. Similarly, imaging device 400 is directly connected to image analysis device 200A or image analysis device 200B, but imaging device 400 may be connected to image analysis device 200A or image analysis device 200B via network 99.

[0185] <Example> Examples of the present invention will be described below to clarify the features of the present invention. [Example]

[0186] Deep learning processing and image analysis processing were performed using the standalone system described in the second embodiment. The tissues targeted for learning and analysis were gastric cancer tissues. The analysis processing was performed on two different gastric cancer tissue specimens.

[0187] [Creating training data and learning] Bright-field images of HE-stained gastric cancer tissue and whole-slide images (WSIs) of DAPI-stained gastric cancer tissue were scanned in color using a virtual slide scanner. The imaging magnification was 40x. The R, G, and B color density values ​​of the bright-field images were then scaled to create a color density coded map for each color. Furthermore, the DAPI-stained fluorescent images were binarized using a preset threshold to separate the cell nucleus region from the rest of the image, creating a binary image. The bright-field and fluorescent images are shown in Figure 16(a) and (b), respectively, and the binary image created from the fluorescent image is shown in Figure 16(c).

[0188] We then created training data by combining the color intensity coded image and the binarized image. The created training data was divided into windows of 113 x 113 pixels, and the training data of each window size was used as the input layer for neural network training. The window size of 113 x 113 pixels was chosen to support the inclusion of the entire shape of the cell nucleus region of at least one cell out of a set of, for example, two to nine cells, within the window.

[0189] [Create an image to be analyzed] Similar to the training data, whole-slide images of HE-stained gastric cancer tissue were scanned in color using a virtual slide scanner at a magnification of 40x. R, G, and B color density coded maps were then created based on the bright-field images, and the images to be analyzed were then combined.

[0190] [Analysis results] Analysis data of a 113 x 113 pixel window size was created around each pixel of the image to be analyzed, and the analysis data of the created window size was input into the trained neural network. Based on the analysis results output from the neural network, the cell nucleus The areas were classified into cell nuclei and other areas, and the outlines of the cell nuclei areas were circled in white. The analysis results are shown in Figures 17 and 18.

[0191] Figure 17 shows the analysis results of the first image of a stomach cancer tissue specimen. Figure 17(a) is a bright-field image of stomach cancer tissue stained with HE, and Figure 17(b) is an image in which the outline of the cell nucleus region obtained by analysis processing is superimposed on the bright-field image of Figure 17(a). The region surrounded by white in Figure 17(b) is the cell nucleus region.

[0192] Figure 18 shows the analysis results of the second image of the stomach cancer tissue specimen. Figure 18(a) is a bright-field image of the stomach cancer tissue stained with HE, and Figure 18(b) is an image in which the outline of the cell nucleus region obtained by the analysis process is superimposed on the bright-field image of Figure 18(a). The region surrounded by white in Figure 18(b) is the cell nucleus region.

[0193] As shown in Figures 17 and 18, it was possible to determine whether or not a region was a cell nucleus at any position in two different types of pathological tissue images. The accuracy rate for determining cell nucleus regions was 85% or higher. [Example]

[0194] The cells from the stamped stomach tissue were stained with Papanicolaou staining to prepare a specimen. The specimen was subjected to the same analysis process as in Example 1 above using a trained neural network. Cells were stamped from both cancerous and non-cancerous stomach areas. The analysis results are shown in Figures 19 and 20.

[0195] Figure 19 shows the analysis results of a stamped specimen of a stomach cancer area. Figure 19(a) is a bright-field image of a stained stamped specimen of a stomach cancer area, and Figure 19(b) is an image in which the outline of the cell nucleus area obtained by analysis processing is superimposed on the bright-field image of Figure 19(a). The area surrounded by white in Figure 19(b) is the cell nucleus area.

[0196] Figure 20 shows the analysis results of a stamped specimen of a non-cancerous stomach area. Figure 20(a) is a bright-field image of a stained stamped specimen of a non-cancerous stomach area, and Figure 20(b) is an image in which the outline of the cell nucleus region obtained by the analysis process is superimposed on the bright-field image of Figure 20(a). The region surrounded by white in Figure 20(b) is the cell nucleus region.

[0197] As shown in FIGS. 19 and 20, it was possible to determine whether or not a region was a cell nucleus at any position of the imprint specimen, which had a different staining pattern from that of Example 1 described above. [Explanation of symbols]

[0198] 10 (10A, 10B) Processing section 20(20A, 20B, 20C) Processing section 11,21 CPU 12,22 memory 13,23 Recording section 14,24 Bus 15,25 Interface section 16,26 Input section 17,27 Output section 19,29 GPU 50 Neural Networks (Deep Learning Algorithms) 50a Input layer 50b output layer 50c middle class 60 Trained Neural Networks (Trained Deep Learning Algorithms) 60a Input layer 60b output layer 60c middle class 70 brightfield images for training (first training image) 71 Fluorescence image for training (second training image) 72r, 72g, 72b Color density coding diagrams for the R, G, and B single-color images of bright-field images for learning (first training data) 73 True image for training (binarized image, second training data) 74 training data 74a Gradient color density values ​​of brightfield image 74b True image binary data 75 training data window size 76 color density values 77 True Image Binary Data 78 Brightfield image of the object to be analyzed 79r, 79g, 79b Color density coding diagram for a single image of each of the R, G, and B colors of the bright-field image to be analyzed 80 Analysis Data 81 color density values 82 Classification result (pixel estimate) 83 Binary image of analysis results 84 Cell nucleus area emphasis image 89(89a,89b) nodes 98 Recording Media 99 Network 100 Vendor side equipment 100A Deep Learning Device 100B Integrated Image Analysis Device 101 Training data generation unit 102 Training data input section 103 Algorithm Update Unit 104 Window Size Database 105 Algorithm Database 200 User side device 200A Image Analysis Device 200B Integrated Image Analysis Instrument 200C Terminal Equipment 201 Analysis data generation unit 202 Analysis data input section 203 Analysis Department 204 Cell nucleus region detection unit 300,400 imaging device 301,401 image sensor 302,402 Fluorescence microscope 308,408 Sample tissue 309,409 stages W1 Window W2 window

Claims

1. A computer program for generating a deep learning algorithm of a neural network structure for analyzing images of tissues or cells, comprising: On the computer, a first acquisition step of acquiring first training data corresponding to first training images of tissues or cells; a second acquisition step of acquiring second training data corresponding to second training images showing regions of cell nuclei in the first training images; a learning step of causing the deep learning algorithm to learn a relationship between the first training data and the second training data; It is possible to execute the first training images are bright-field images of a specimen of a tissue sample collected from an individual or a specimen of a sample containing cells collected from an individual, captured under bright-field conditions; the second training image is a fluorescent image of a cell nucleus captured under fluorescent observation of a specimen corresponding to or identical to the specimen and prepared by applying fluorescent nuclear staining to the specimen, the second training data is binarized data to distinguish between the cell nucleus region and other regions, By causing the computer to execute the first acquisition step, the second acquisition step, and the learning step, the deep learning algorithm can generate data indicating the area of ​​a cell nucleus for the tissue or cell image. Computer program.

2. The computer program of claim 1 , wherein the second training data is generated by comparing a color density value of each pixel of the second training image with a predetermined threshold.

3. The computer program according to claim 1 , wherein the first training data is data indicating a color density value of each pixel of the first training image.

4. The computer, generating the first training data from the first training images before the first acquiring step; generating the second training data from the second training images before the second acquiring step; The computer program according to any one of claims 1 to 3, further capable of executing:

5. the computer is connected to an imaging device that images the specimen; the first training image and the second training image are images of the specimen captured by the imaging device, and are stored in a storage device of the computer; The computer program causes the computer to: inputting the first training images and the second training images from the storage device to a processing unit of the computer; The computer program of claim 4 , further capable of executing:

6. The learning step includes a step of having the deep learning algorithm learn a plurality of data obtained by cutting out the first training data and the second training data by a predetermined number of pixels. The computer program according to any one of claims 1 to 5.

7. The computer, generating analysis data from an image to be analyzed that includes tissue or cells to be analyzed; inputting the analysis data into the deep learning algorithm that has completed learning in the learning step; generating data indicating the area of ​​the cell nucleus in the image to be analyzed by the deep learning algorithm that has completed learning; The computer program according to any one of claims 1 to 6, further comprising:

8. The image to be analyzed is a bright-field image obtained by capturing a specimen of a tissue sample collected from an individual or a specimen of a sample containing cells collected from an individual under bright-field conditions.

8. A computer program according to claim 7.

9. The computer program product of any one of claims 1 to 8, wherein the neural network structure is a convolutional neural network.

10. The computer program according to any one of claims 1 to 9, wherein the first training images and the second training images are images for tissue diagnosis.

11. The computer program according to any one of claims 1 to 9, wherein the first training images and the second training images are images for cytological diagnosis.

12. An apparatus for generating deep learning algorithms of neural network structures for analyzing images of tissues or cells, comprising: the device comprises a processing unit; The processing unit acquiring first training data corresponding to first training images of tissues or cells; obtaining second training data corresponding to second training images showing regions of cell nuclei in the first training images; causing the deep learning algorithm to learn a relationship between the first training data and the second training data; the first training images are bright-field images of a specimen of a tissue sample collected from an individual or a specimen of a sample containing cells collected from an individual, captured under bright-field conditions; the second training image is a fluorescent image of a cell nucleus captured under fluorescent observation of a specimen corresponding to or identical to the specimen and prepared by applying fluorescent nuclear staining to the specimen, the second training data is binarized data to distinguish between the cell nucleus region and other regions, the processing unit acquires the first training data and the second training data, and causes the deep learning algorithm to learn the relationship between the first training data and the second training data, thereby enabling the deep learning algorithm to generate data indicating the area of ​​a cell nucleus for the tissue or cell image. Device.

13. 1. A method for generating a deep learning algorithm of a neural network structure for analyzing tissue or cell images, comprising: acquiring first training data corresponding to first training images of tissues or cells; obtaining second training data corresponding to second training images showing regions of cell nuclei in the first training images; causing the deep learning algorithm to learn a relationship between the first training data and the second training data; Including, the first training images are bright-field images of a specimen of a tissue sample collected from an individual or a specimen of a sample containing cells collected from an individual, captured under bright-field conditions; the second training image is a fluorescent image of a cell nucleus captured under fluorescent observation of a specimen corresponding to or identical to the specimen and prepared by applying fluorescent nuclear staining to the specimen, the second training data is binarized data to distinguish between the cell nucleus region and other regions, By acquiring the first training data and the second training data and having the deep learning algorithm learn the relationship between the first training data and the second training data, the deep learning algorithm is able to generate data indicating the area of ​​a cell nucleus for the tissue or cell image. method.

Images